Kernel panic when freeing a Tempesta message #56
Comments
|
This issue is very convoluted and difficult to comprehend. Here's my analysis. All data SKBs that come to Tempesta are organazied into messages that make up either an HTTP request message, or an HTTP response message. Incoming HTTP message is parsed, processed and then passed over to the other end - a backend server if it's a request, or a client if it's a response. It's important to note that the messages are passed in a zero-copy fashion in the same SKBs they have come to Tempesta. The SKBs that are not longer needed in Tempesta need to be deleted and returned to the kernel's memory pool. The issue is where and how that needs to be done. Few exceptions aside, Tempesta doesn't create or own SKBs it keeps in the messages. These SKBs come from the Linux kernel. Socket and TCP layers are the real owners. Moreover, these layers consider themselves the sole owners of these SKBs, and so they use a /* Don't free Tempesta owned socket buffers. */
if (unlikely(ss_skb_passed(skb)))
return;It's important to note that Synchronous Sockets that is part of Tempesta uses only In the current logic, Tempesta keeps a request message until the response comes, then uses part of the request with the response and caches the data. Before it caches the data it passes the response over to the client by passing the SKBs to the Linux stack. After the data is cached Tempesta deletes both request and response message with all associated data (read - SKBs). A crucial piece of knowledge to mention here is that the Linux stack deletes SKBs after it has sent them out (it's not that simple, the stack keeps some for possible retransmissions, etc). That means that the SKBs in the request message had been most probably freed by the stack and from the stack's point of view. They had not been actually freed due to the special case in
All in all, generally, we don't know whether the Linux stack considers the SKBs freed or not. So we don't know if Tempesta can remove a particular SKB. If we assume that the Linux stack frees all SKBs when they have been sent out, then a solution could be to free request's SKBs and do not free response SKBs when Tempesta destroys the request/response messages. We can strengthen the case by caching the response before it's passed over to the Linux stack. Then under the assumption the response message's SKBs are freed by the stack. Again, the solution within current logic would be to destroy messages differently in Tempesta. Free request's SKBs as they had been freed by the Linux stack when they were sent out. Do not free the response SKBs are the Linux stack still processes them. I tried this solution, and it works on simple tests. However, considering the above note that might not always be true. We're still not sure about the request SKBs. To "guarantee" freeing the request's SKBs under the same assumption, we need to store the needed request data in cache right after we have processed the request and before we pass it over to a backend server. Then we can be sure that the request's SKBs are eventually freed by the Linux stack. |
|
After further analysis the following has became clear. The major issue here is whether an SKB is in Tempesta's full possession or not. And it is from the moment an SKB is taken out of a socket's receive queue, and until that SKB is sent out (that is, put into a socket's send queue). While an SKB is in Temepsta's full possession, it's Tempesta's responsibility to free an SKB should it decide not to send it out for any reason. As soon as an SKB is sent out the Linux networking stack takes possession of that SKBs, and it will take care of freeing it later. |
This is a solution to #56. The key is knowing whether an SKB is in Tempesta's full possession or not. It is from the moment it's been taken out of a socket's receive queue, and until it's sent out, i.e. put in a socket's send queue. If a message is freed during that time frame, Tempesta is responsible of freeing SKBs that make the message. Otherwise, the Linux networking stack will take care of that when it finishes sending those SKBs out.
|
The proposed patch above is only half of the solution. Yes, while an SKB is completely in Tempesta's possession, Tempesta can and must delete it when it deletes a message that contains it. So, the above patch is an easy and cheap solution that covers the described case. The problem, however, is what to do when Tempesta sends a message out, either to a backend server, or to a client. In many cases Tempesta deletes the message immediately after it sends it out. Does it need to delete the SKBs as well? Well, the kernel deletes SKBs after it has finished with sending. So, generally speaking, the case we have here is that both Tempesta and the kernel attempt to free SKBs that make the message at nearly the same time. If the kernel is faster, then it's okay, an SKB is still in Tempesta's message queue, and so it doesn't get freed, Tempesta will free it in a moment. If Tempesta is faster, though, then an attempt by the kernel will lead to a crash. There's also a request message that Tempesta keeps around until it receives the response to it. Even though it may appear that the kernel must have attempted to free the request's SKBs by that time, that's not a guarantee. So, essentially, it's still the same case. It appears that the solution requires a reference counter in SKB, like the one that |
|
The concept behind skb management is very simple:
The concept must be obeyed, so we wouldn't need extra reference counting or other complex things. |
This part of a series of changes to implement #56. The kernel is modified to not free SKBs that are still in Tempesta queues. That gives Tempesta control over SKBs. Request SKBs are kept until a corresponging response it received. However. when response is sent to a client, its SKBs are no longer needed. The new argument tells ss_send() if it needs to remove SKBs from a Tempesta queue.
|
Fixed with 6ea9bfb and krizhanovsky/linux-3.10.10-sync_sockets@06f1b61 |
This appears to be a case of SKB's double-free - freeing an SKB that had been freed already.
The text was updated successfully, but these errors were encountered: