Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

vsock: unexpected EWOULDBLOCK while reading from backing stream #3497

Closed
ghost opened this issue Dec 23, 2021 · 6 comments · Fixed by #3517
Closed

vsock: unexpected EWOULDBLOCK while reading from backing stream #3497

ghost opened this issue Dec 23, 2021 · 6 comments · Fixed by #3517

Comments

@ghost
Copy link

ghost commented Dec 23, 2021

cloud-hypervisor v20.1.0

--vsock cid=3,socket=/path/v.sock

Reproduced with streaming uncompressed video/assets over waypipe (wayland over sockets)
Frequency of occurrence depends on size of window (and therefore bandwidth)

On host:
waypipe --socket /path/v.sock_5000 client &

On guest:
socat UNIX-LISTEN:/tmp/waypipe-server.sock,reuseaddr,fork VSOCK-CONNECT:2:5000,forever &
waypipe --no-gpu --socket /tmp/waypipe-server.sock --control /tmp/waypipe-ctrl.pipe --display /tmp/wayland-0 server -- sleep inf &
WAYLAND_DISPLAY=/tmp/wayland-0 mpv /path/to/4k_video.mp4

cloud-hypervisor: 1155.939756072s: <_vsock2> WARN:virtio-devices/src/vsock/csm/connection.rs:239 -- vsock: unexpected EWOULDBLOCK while reading from backing stream: lp=2000, pp=630240254, err=Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }
cloud-hypervisor: 1156.142021665s: <_vsock2> WARN:virtio-devices/src/vsock/csm/connection.rs:239 -- vsock: unexpected EWOULDBLOCK while reading from backing stream: lp=2000, pp=630240254, err=Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }
cloud-hypervisor: 1156.1425842s: <_vsock2> WARN:virtio-devices/src/vsock/csm/connection.rs:239 -- vsock: unexpected EWOULDBLOCK while reading from backing stream: lp=2000, pp=630240254, err=Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }
cloud-hypervisor: 1156.143808813s: <_vsock2> WARN:virtio-devices/src/vsock/csm/connection.rs:239 -- vsock: unexpected EWOULDBLOCK while reading from backing stream: lp=2000, pp=630240254, err=Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }
cloud-hypervisor: 1156.189278481s: <_vsock2> WARN:virtio-devices/src/vsock/csm/connection.rs:239 -- vsock: unexpected EWOULDBLOCK while reading from backing stream: lp=2000, pp=630240254, err=Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }
cloud-hypervisor: 1156.412921305s: <_vsock2> WARN:virtio-devices/src/vsock/csm/connection.rs:239 -- vsock: unexpected EWOULDBLOCK while reading from backing stream: lp=2000, pp=630240254, err=Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }
cloud-hypervisor: 1156.563153059s: <_vsock2> WARN:virtio-devices/src/vsock/csm/connection.rs:239 -- vsock: unexpected EWOULDBLOCK while reading from backing stream: lp=2000, pp=630240254, err=Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }
cloud-hypervisor: 1156.564757199s: <_vsock2> WARN:virtio-devices/src/vsock/csm/connection.rs:239 -- vsock: unexpected EWOULDBLOCK while reading from backing stream: lp=2000, pp=630240254, err=Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }

This times 1000.
@ghost ghost changed the title sock: unexpected EWOULDBLOCK while reading from backing stream vsock: unexpected EWOULDBLOCK while reading from backing stream Dec 24, 2021
@sboeuf
Copy link
Member

sboeuf commented Jan 3, 2022

Hi @jackloomen
Have you tried this with firecracker? I'm asking because the vsock device has been ported from the firecracker repository and this might be a common issue that would be better to fix there and then backport to Cloud Hypervisor.
That being said, if the issue is specific to Cloud Hypervisor, I'll be happy to look into it.

@andreeaflorescu
Copy link

We fixed a few months ago a problem with the vsock (not sure if it's the same one), but you can check this commit: firecracker-microvm/firecracker@38bc78b.

Btw, we are now looking at upstreaming the vsock building blocks as part of rust-vmm. We will soon publish a design document. @sboeuf do you think you could also take a look?

@sboeuf
Copy link
Member

sboeuf commented Jan 3, 2022

Btw, we are now looking at upstreaming the vsock building blocks as part of rust-vmm. We will soon publish a design document. @sboeuf do you think you could also take a look?

Definitely! Sounds great!

@sboeuf
Copy link
Member

sboeuf commented Jan 3, 2022

@jackloomen could you have a quick try with #3517 and let me know if that fixes the issue for you?

@ghost
Copy link
Author

ghost commented Jan 3, 2022

@jackloomen could you have a quick try with #3517 and let me know if that fixes the issue for you?

After applying the patch to 20.1 the issue could not be reproduced.

@sboeuf sboeuf mentioned this issue Jan 4, 2022
Merged
sboeuf pushed a commit to sboeuf/cloud-hypervisor-1 that referenced this issue Jan 4, 2022
virtio-devices: Forward correct evset in vsock muxer
ac00e78 
When forwarding an epoll event from the unix muxer to the
targeted connection event handler, the eventset the connection
registered is forwarded instead of the actual epoll
operation (IN/OUT).

For example, if the connection was registered for EPOLLIN,
and receives an EPOLLOUT, the connection will actually handle
an EPOLLOUT.

This is the root cause of previous bug, which caused the
introduction of some workarounds (i.e: handling ewouldblock
when reading after receiving EPOLLIN, which should never happen).

When matching the connection, we retrieve and use the evset of
the connection instead of the one passed as a parameter.
The compiler does not complain for an unused variable because
it was first logged in a debug! statement.

This is an unfortunate naming mistake that caused a lot of problems.

Fixes cloud-hypervisor#3497

Signed-off-by: Eduard Kyvenko <eduard.kyvenko@gmail.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
sboeuf pushed a commit to sboeuf/cloud-hypervisor-1 that referenced this issue Jan 4, 2022
virtio-devices: Forward correct evset in vsock muxer
fcde213 
When forwarding an epoll event from the unix muxer to the
targeted connection event handler, the eventset the connection
registered is forwarded instead of the actual epoll
operation (IN/OUT).

For example, if the connection was registered for EPOLLIN,
and receives an EPOLLOUT, the connection will actually handle
an EPOLLOUT.

This is the root cause of previous bug, which caused the
introduction of some workarounds (i.e: handling ewouldblock
when reading after receiving EPOLLIN, which should never happen).

When matching the connection, we retrieve and use the evset of
the connection instead of the one passed as a parameter.
The compiler does not complain for an unused variable because
it was first logged in a debug! statement.

This is an unfortunate naming mistake that caused a lot of problems.

Fixes cloud-hypervisor#3497

Signed-off-by: Eduard Kyvenko <eduard.kyvenko@gmail.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
rbradford pushed a commit that referenced this issue Jan 4, 2022
@rbradford
virtio-devices: Forward correct evset in vsock muxer
c47ab55 
When forwarding an epoll event from the unix muxer to the
targeted connection event handler, the eventset the connection
registered is forwarded instead of the actual epoll
operation (IN/OUT).

For example, if the connection was registered for EPOLLIN,
and receives an EPOLLOUT, the connection will actually handle
an EPOLLOUT.

This is the root cause of previous bug, which caused the
introduction of some workarounds (i.e: handling ewouldblock
when reading after receiving EPOLLIN, which should never happen).

When matching the connection, we retrieve and use the evset of
the connection instead of the one passed as a parameter.
The compiler does not complain for an unused variable because
it was first logged in a debug! statement.

This is an unfortunate naming mistake that caused a lot of problems.

Fixes #3497

Signed-off-by: Eduard Kyvenko <eduard.kyvenko@gmail.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
rbradford pushed a commit to rbradford/cloud-hypervisor that referenced this issue Jan 4, 2022
@rbradford
virtio-devices: Forward correct evset in vsock muxer
6e25e13 
When forwarding an epoll event from the unix muxer to the
targeted connection event handler, the eventset the connection
registered is forwarded instead of the actual epoll
operation (IN/OUT).

For example, if the connection was registered for EPOLLIN,
and receives an EPOLLOUT, the connection will actually handle
an EPOLLOUT.

This is the root cause of previous bug, which caused the
introduction of some workarounds (i.e: handling ewouldblock
when reading after receiving EPOLLIN, which should never happen).

When matching the connection, we retrieve and use the evset of
the connection instead of the one passed as a parameter.
The compiler does not complain for an unused variable because
it was first logged in a debug! statement.

This is an unfortunate naming mistake that caused a lot of problems.

Fixes cloud-hypervisor#3497

Signed-off-by: Eduard Kyvenko <eduard.kyvenko@gmail.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
(cherry picked from commit c47ab55)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
rbradford pushed a commit that referenced this issue Jan 4, 2022
@rbradford
virtio-devices: Forward correct evset in vsock muxer
868e566 
When forwarding an epoll event from the unix muxer to the
targeted connection event handler, the eventset the connection
registered is forwarded instead of the actual epoll
operation (IN/OUT).

For example, if the connection was registered for EPOLLIN,
and receives an EPOLLOUT, the connection will actually handle
an EPOLLOUT.

This is the root cause of previous bug, which caused the
introduction of some workarounds (i.e: handling ewouldblock
when reading after receiving EPOLLIN, which should never happen).

When matching the connection, we retrieve and use the evset of
the connection instead of the one passed as a parameter.
The compiler does not complain for an unused variable because
it was first logged in a debug! statement.

This is an unfortunate naming mistake that caused a lot of problems.

Fixes #3497

Signed-off-by: Eduard Kyvenko <eduard.kyvenko@gmail.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
(cherry picked from commit c47ab55)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
@lauralt
Copy link

lauralt commented Jan 10, 2022

Btw, we are now looking at upstreaming the vsock building blocks as part of rust-vmm. We will soon publish a design document. @sboeuf do you think you could also take a look?

Definitely! Sounds great!

Hi @sboeuf; I just opened an issue with a proposal for the vsock packet abstraction. Can you also have a look? Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

virtio-devices: Forward correct evset in vsock muxer sboeuf/cloud-hypervisor-1
3 participants
@andreeaflorescu @sboeuf @lauralt