Network issues when using pasta instead of slirp4netns in podman 5.0.2 #22593
Comments
sbrivio-rh
added
network
|
A packet capture would be nice to have, you can ask pasta to save one with e.g. You can drop packets you don't want to share using |
|
Running a packet capture seems to only pick up a few
|
Ah, no, my bad: pasta implements a tap bypass path for local connections that splices TCP sockets between container and host directly, to improve throughput by avoiding Layer-2 / Layer-4 translations where not needed, see the appendage in this diagram. Packets on that path are not captured (because they're not really packets -- we just move the Layer-4 payload around). See also #22575 (comment) (that issue might be related, but we're not sure there's an issue in pasta yet) and the following comment. To capture traffic in this case, |
|
Here's the tshark output of a pcap ran for a few minutes on the container using You can see around 2.7s in (near the end of the file) all tcp traffic stops until around 121s in. The migration never finishes, and I stop things manually, although I haven't let it go for more than about 15 min. But usually the hundreds of migrations (when using slirp4netns) take less than 2 min total, whereas this one migration gets stuck for over 10 min using pasta with seemingly nothing actually happening. |
Thanks a lot for the capture, this seems to be compatible with the current findings from #22575 -- I'm trying to reproduce that at the moment. |
|
@jadonclegg, assuming that #22575 is in fact the same issue, I just posted a patch to fix this at https://archives.passt.top/passt-dev/20240508090338.2735208-1-sbrivio@redhat.com/ (see also #22575 (comment)). It would be great if you could test it by building from source (something on the lines of |
|
@sbrivio-rh your patch appears to have fixed the issue. After compiling and installing it, I ran the migration three times in a row and it worked every time. I then reverted to the commit before yours, installed again, and the migration got stuck at the same place it was happening before installing your patched version. |
|
This is now fixed in passt version 2024_05_10.7288448 and the corresponding Fedora 40 update. |
In tcp_splice_sock_handler(), if we get EAGAIN on the second splice(),
from pipe to receiving socket, that doesn't necessarily mean that the
pipe is empty: the receiver buffer might be full instead.
Hence, we can't use the 'never_read' flag to decide that there's
nothing to wait for: even if we didn't read anything from the sending
side in a given iteration, we might still have data to send in the
pipe. Use read/written counters, instead.
This fixes an issue where large bulk transfers would occasionally
hang. From a corresponding strace:
0.000061 epoll_wait(4, [{events=EPOLLOUT, data={u32=29442, u64=12884931330}}], 8, 1000) = 1
0.005003 epoll_ctl(4, EPOLL_CTL_MOD, 211, {events=EPOLLIN|EPOLLRDHUP, data={u32=54018, u64=8589988610}}) = 0
0.000089 epoll_ctl(4, EPOLL_CTL_MOD, 115, {events=EPOLLIN|EPOLLRDHUP, data={u32=29442, u64=12884931330}}) = 0
0.000081 splice(211, NULL, 151, NULL, 1048576, SPLICE_F_MOVE|SPLICE_F_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable)
0.000073 splice(150, NULL, 115, NULL, 1048576, SPLICE_F_MOVE|SPLICE_F_NONBLOCK) = 1048576
0.000087 splice(211, NULL, 151, NULL, 1048576, SPLICE_F_MOVE|SPLICE_F_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable)
0.000045 splice(150, NULL, 115, NULL, 1048576, SPLICE_F_MOVE|SPLICE_F_NONBLOCK) = 520415
0.000060 splice(211, NULL, 151, NULL, 1048576, SPLICE_F_MOVE|SPLICE_F_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable)
0.000044 splice(150, NULL, 115, NULL, 1048576, SPLICE_F_MOVE|SPLICE_F_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable)
0.000044 epoll_wait(4, [], 8, 1000) = 0
we're reading from socket 211 into to the pipe end numbered 151,
which connects to pipe end 150, and from there we're writing into
socket 115.
We initially drop EPOLLOUT from the set of monitored flags for socket
115, because it already signaled it's ready for output. Then we read
nothing from socket 211 (the sender had nothing to send), and we keep
emptying the pipe into socket 115 (first 1048576 bytes, then 520415
bytes).
This call of tcp_splice_sock_handler() ends with EAGAIN on the writing
side, and we just exit this function without setting the OUT_WAIT_1
flag (and, in turn, EPOLLOUT for socket 115). However, it turns out,
the pipe wasn't actually emptied, and while socket 211 had nothing
more to send, we should have waited on socket 115 to be ready for
output again.
As a further step, we could consider not clearing EPOLLOUT at all,
unless the read/written counters match, but I'm first trying to fix
this ugly issue with a minimal patch.
Link: containers/podman#22575
Link: containers/podman#22593
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
Issue Description
Somewhat of a complex scenario, but I have a java spring web app using flyway for db migration. I have my db (postgres:14) running in podman using the official docker image. When running a larger migration (400k lines) that uses
COPY ... FROM STDIN;(can't give a lot of details, enterprise app) it consistently freezes and never completes the migration. Switching podman to--network hostor--network slirp4netnsfixes the issue and the migration runs basically instantly.I'm not sure where to find any useful logs for more information, running
podman logs -f postgresdoesn't give any useful information, and there's no useful information coming from the tomcat logs either. Not sure where to get logs for the inner workings of podman.Steps to reproduce the issue
Steps to reproduce the issue
COPY ... FROM STDINin flyway on a db running postgres:14 in podman 5.0.2 using pasta as the network backend (rootless)Describe the results you received
database migration hangs. I'm assuming the network just hangs or something similar, but not sure how to get more information on this. Waited 10+ minutes with no change.
Describe the results you expected
network shouldn't hang
podman info output
Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
Yes
Additional environment details
Fedora 40, all packages up to date as of May 3
Additional information
No response
The text was updated successfully, but these errors were encountered: