Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

clh: virtiofsd: vfio: VFIO test us failing #862

Closed
devimc opened this issue Oct 5, 2020 · 6 comments · Fixed by #876
Closed

clh: virtiofsd: vfio: VFIO test us failing #862

devimc opened this issue Oct 5, 2020 · 6 comments · Fixed by #876
Labels
bug Incorrect behaviour needs-review Needs to be assessed by the team.
Projects
Issue backlog Cloud hypervisor integration

Comments

@devimc
Copy link

devimc commented Oct 5, 2020

Looks like virtiofs is crashing when a VFIO device is hotplugged - I can work around this issue by disabling DAX in the configuration file

ct 05 20:15:53 test virtiofsd[30057]: [ID: 00000001] virtio_loop: Unexpected poll revents 11
Oct 05 20:15:53 test virtiofsd[30057]: [ID: 00000001] virtio_loop: Stopping queue 0 thread
Oct 05 20:15:53 test virtiofsd[30057]: [ID: 00000003] fv_queue_thread: kill event on queue 0 - quitting
Oct 05 20:15:53 test virtiofsd[30057]: [ID: 00000001] virtio_loop: Stopping queue 1 thread
Oct 05 20:15:53 test virtiofsd[30057]: [ID: 00000016] fv_queue_thread: kill event on queue 1 - quitting
Oct 05 20:15:53 test virtiofsd[30057]: [ID: 00000001] virtio_loop: Exit
Oct 05 20:15:53 test virtiofsd[30057]: [ID: 00000001] fv_panic: libvhost-user: Error while writing: Broken pipe

cc @likebreath
@devimc devimc added bug Incorrect behaviour needs-review Needs to be assessed by the team. labels Oct 5, 2020
@jodh-intel jodh-intel added this to To do in Issue backlog Oct 5, 2020
@likebreath
Copy link
Contributor

Thanks for reporting this issue and the work-around.

@jcvenegas Given your extended experience on virtiofsd, do you have any inputs? What's our plan for the DAX option?

@likebreath likebreath moved this from To do to In progress in Cloud hypervisor integration Oct 6, 2020
@likebreath likebreath mentioned this issue Oct 6, 2020
Merged
@likebreath
Copy link
Contributor

The PR to disable DAX option for virtio-fs is pending in kata-runtime (kata-containers/runtime#2997), and will be forward ported to kata 2.0. I am assuming that would bring our VFIO CI on CLH back in kata 2.0, right? @devimc

@likebreath likebreath mentioned this issue Oct 6, 2020
Closed
5 tasks
@devimc
Copy link
Author

devimc commented Oct 7, 2020

@likebreath you are right

@likebreath likebreath moved this from In progress to Known issues/bugs in Cloud hypervisor integration Oct 7, 2020
@likebreath likebreath moved this from Known issues/bugs to In progress in Cloud hypervisor integration Oct 7, 2020
@jcvenegas
Copy link
Member

@likebreath @likebreath I will send the 2.0 Port to by tomorrow

@likebreath likebreath mentioned this issue Oct 7, 2020
Closed
5 tasks
@jcvenegas jcvenegas mentioned this issue Oct 8, 2020
Merged
Cloud hypervisor integration automation moved this from In progress to Done Oct 9, 2020
Issue backlog automation moved this from To do to Done Oct 9, 2020
bergwolf pushed a commit that referenced this issue Oct 9, 2020
@jcvenegas
virtiofs: Disable DAX
c447248 
virtiofs DAX support is not stable today, there are
a few corner cases to make it default.

Fixes: #862
Fixes: #875

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
@jodh-intel jodh-intel mentioned this issue Oct 9, 2020
Merged
bergwolf pushed a commit to bergwolf/kata-containers that referenced this issue Oct 10, 2020
@jcvenegas @bergwolf
virtiofs: Disable DAX
0a2478b 
virtiofs DAX support is not stable today, there are
a few corner cases to make it default.

Fixes: kata-containers#862
Fixes: kata-containers#875

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
@jodh-intel jodh-intel moved this from Done to In progress in Issue backlog Oct 17, 2020
bergwolf pushed a commit to bergwolf/kata-containers that referenced this issue Oct 17, 2020
@jcvenegas @bergwolf
virtiofs: Disable DAX
ef01856 
virtiofs DAX support is not stable today, there are
a few corner cases to make it default.

Fixes: kata-containers#862
Fixes: kata-containers#875

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
bergwolf pushed a commit to bergwolf/kata-containers that referenced this issue Oct 17, 2020
@jcvenegas @bergwolf
virtiofs: Disable DAX
4f1d23b 
virtiofs DAX support is not stable today, there are
a few corner cases to make it default.

Fixes: kata-containers#862
Fixes: kata-containers#875

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
@hejianet
Copy link

Looks like virtiofs is crashing when a VFIO device is hotplugged - I can work around this issue by disabling DAX in the configuration file

ct 05 20:15:53 test virtiofsd[30057]: [ID: 00000001] virtio_loop: Unexpected poll revents 11
Oct 05 20:15:53 test virtiofsd[30057]: [ID: 00000001] virtio_loop: Stopping queue 0 thread
Oct 05 20:15:53 test virtiofsd[30057]: [ID: 00000003] fv_queue_thread: kill event on queue 0 - quitting
Oct 05 20:15:53 test virtiofsd[30057]: [ID: 00000001] virtio_loop: Stopping queue 1 thread
Oct 05 20:15:53 test virtiofsd[30057]: [ID: 00000016] fv_queue_thread: kill event on queue 1 - quitting
Oct 05 20:15:53 test virtiofsd[30057]: [ID: 00000001] virtio_loop: Exit
Oct 05 20:15:53 test virtiofsd[30057]: [ID: 00000001] fv_panic: libvhost-user: Error while writing: Broken pipe

@devimc Hi，does this crashing also happened with qemu vfio hotplugging?
And in this case, what is the vfio device? disk/NIC ? thanks for the details.
I tried to reproduce it on arm64 with qemu (no Kata image/container rootfs involved) but failed to reproduce.
I started the virtiofsd and mount a fs in the qemu guest with dax mode. And then hot-adding a nvme disk. After that, everything
seems to be ok. Am I missing anything?

@devimc
Copy link
Author

devimc commented Nov 11, 2020

@hejianet this error is reproducible hot-adding a virtio nic into qemu + virtiofs with dax enabled, for more details see https://github.com/kata-containers/tests/blob/2.0-dev/integration/kubernetes/vfio.sh

c3d pushed a commit that referenced this issue Apr 12, 2021
@jcvenegas @c3d
virtiofs: Disable DAX
71cabc4 
virtiofs DAX support is not stable today, there are
a few corner cases to make it default.

Fixes: #862
Fixes: #875

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
c3d pushed a commit to c3d/kata-containers that referenced this issue Apr 12, 2021
@jcvenegas @c3d
virtiofs: Disable DAX
196fdb1 
virtiofs DAX support is not stable today, there are
a few corner cases to make it default.

Fixes: kata-containers#862
Fixes: kata-containers#875

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Incorrect behaviour needs-review Needs to be assessed by the team.
Projects
Issue backlog
  
In progress
Cloud hypervisor integration
  
Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

virtiofs: Disable DAX jcvenegas/kata-containers
4 participants
@hejianet @likebreath @devimc @jcvenegas