msg: add new async event driver based on poll()

[rafalop]

This is a new event driver for async IO based on poll(), intended to overcome the file descriptor limitations experienced by the select() based driver used by Windows clients.

In Windows, select() can only manage 64*async_op_threads file descriptors, limiting cluster OSD counts to around 200 with default config - see linked bug. Although there are better native event based IO methods for windows, using poll() allows us to keep an unmodified posix stack so it integrates easily.

This new poll() driver can also be used on other niche/legacy systems that provide poll and do not have anything better than select at the moment.

Fixes: https://tracker.ceph.com/issues/55840

[ionutbalutoiu]

Hello everyone,

I tried to benchmark the performance impact of this change, using the benchmarking methodology described in this Cloudbase blog post.

I ran 5 consecutive benchmarks (with 1000 RBD volumes / each benchmark) for each use case.

The are the results for each use case tested:

• READ operation:

  • Ceph main branch: https://paste.samfira.com/public/p/fBdXuEiRt0w60Ped7p7SyCRK
  • Ceph main branch with this PR applied: https://paste.samfira.com/public/p/XBjfC2WHvJxjzyYqJBJBoMba

• WRITE operation:

  • Ceph main branch: https://paste.samfira.com/public/p/966oQWNIWNuSbU6mQOi59p0P
  • Ceph main branch with this PR applied: https://paste.samfira.com/public/p/gWfAGNJ7QXk1EZSjVdEdb7k9

We notice that:

• For the READ operation we have aprox. 1385 MB/sec median bandwidth in both cases (clean main branch vs main branch with this change applied)
• For the WRITE operation we have aprox. 115 MB/sec median bandwidth in both cases (clean main branch vs main branch with this change applied)

Therefore, I didn't find any performance impact of this change in my testing efforts.

[djgalloway]

@idryomov Might you be the right person to review this? I'm not really sure which team/lead would be best.

[idryomov]

@djgalloway The labeler is never wrong -- this belongs to Core.

[petrutlucian94]

@tchaikov thanks for pushing the recent Windows related fixes. could you please review this one as well?

[djgalloway]

@neha-ojha @jdurgin Can either of you please do a code review or delegate someone? Would love to get this merged. Thanks!

[tchaikov]

@tchaikov thanks for pushing the recent Windows related fixes. could you please review this one as well?

@petrutlucian94 sorry, i don't have enough bandwidth reviewing this change. just skimmed through it, though. there are quite a few formatting issues.

[petrutlucian94]

jenkins test make check arm64

[petrutlucian94]

jenkins test windows

[rafalop]

@petrutlucian94 are you familiar with the jenkins tests? I don't think the 'ceph windows tests' failure is an issue with the code based on console output.

[petrutlucian94]

@petrutlucian94 are you familiar with the jenkins tests? I don't think the 'ceph windows tests' failure is an issue with the code based on console output.

indeed, it's unrelated. The job is spinning up a Windows vm using libvirt and apparently fails when trying to retrieve the IP address:

++ sudo virsh domifaddr --source agent --interface Ethernet --full ceph-win-ltsc2019-ceph-windows-pull-requests-11980
++ grep ipv4
++ awk '{print $4}'
++ cut -d / -f1
+ VM_IP=
+ echo 'Retrying in 10 seconds'

All recent jobs seem to have failed because of the same issue. @ionutbalutoiu any thoughts on this?

[ionutbalutoiu]

@petrutlucian94 are you familiar with the jenkins tests? I don't think the 'ceph windows tests' failure is an issue with the code based on console output.

indeed, it's unrelated. The job is spinning up a Windows vm using libvirt and apparently fails when trying to retrieve the IP address:

++ sudo virsh domifaddr --source agent --interface Ethernet --full ceph-win-ltsc2019-ceph-windows-pull-requests-11980
++ grep ipv4
++ awk '{print $4}'
++ cut -d / -f1
+ VM_IP=
+ echo 'Retrying in 10 seconds'

All recent jobs seem to have failed because of the same issue. @ionutbalutoiu any thoughts on this?

I see that starting from the end of last week, this error appeared constantly on all the windows jobs.

I'm trying to see what's wrong.

[ionutbalutoiu]

@petrutlucian94 @rafalop

It seems that the Jenkins job failed to properly get the VM IP address on some of the libvirt hosts from the Jenkins infra.
I submitted this PR to improve the VM IP detection: ceph/ceph-build#2045

Once that gets fixed, I'll retry the tests here.

[ionutbalutoiu]

jenkins test windows

[ionutbalutoiu]

jenkins test windows

[ionutbalutoiu]

jenkins test windows

[ionutbalutoiu]

jenkins test windows

[djgalloway]

[2022-08-10T20:06:32.000Z] [googletest] unittest_crush.exe failed. Error: Command returned non-zero code(1): "cmd /c 'C:\ceph\unittest_crush.exe --gtest_output=xml:C:\workspace\test_results\unittest_crush_results.xml  > C:\workspace\test_results\unittest_crush_results.log 2>&1'".
[2022-08-10T20:06:32.000Z] [googletest] unittest_crush_wrapper.exe failed. Error: Command returned non-zero code(3): "cmd /c 'C:\ceph\unittest_crush_wrapper.exe --gtest_output=xml:C:\workspace\test_results\unittest_crush_wrapper_results.xml  > C:\workspace\test_results\unittest_crush_wrapper_results.log 2>&1'".

Are these unrelated?

[ionutbalutoiu]

[2022-08-10T20:06:32.000Z] [googletest] unittest_crush.exe failed. Error: Command returned non-zero code(1): "cmd /c 'C:\ceph\unittest_crush.exe --gtest_output=xml:C:\workspace\test_results\unittest_crush_results.xml  > C:\workspace\test_results\unittest_crush_results.log 2>&1'".
[2022-08-10T20:06:32.000Z] [googletest] unittest_crush_wrapper.exe failed. Error: Command returned non-zero code(3): "cmd /c 'C:\ceph\unittest_crush_wrapper.exe --gtest_output=xml:C:\workspace\test_results\unittest_crush_wrapper_results.xml  > C:\workspace\test_results\unittest_crush_wrapper_results.log 2>&1'".

Are these unrelated?

I'm not sure.

@rafalop @petrutlucian94 - Are these failures related to the code changes from here ?

[petrutlucian94]

@rafalop @petrutlucian94 - Are these failures related to the code changes from here ?

those are unrelated, I've just submitted a fix: #47818

[petrutlucian94]

jenkins test windows

[ionutbalutoiu]

jenkins test windows

[petrutlucian94]

The windows job was failing because of some leftover apt sources:

E: Failed to fetch https://4.chacra.ceph.com/r/cortx-motr/master/39f89fa1c6945040433a913f2687c4b4e6cbeb3f/ubuntu/jammy/flavors/default/dists/jammy/main/binary-amd64/Packages  404  Not Found [IP: 158.69.64.56 443]

It's a bit unfortunate that CI jobs aren't isolated in containers or vms, thus subsequent jobs can be impacted if certain files are leaked by the previous jobs.

In the meantime, my colleague @ionutbalutoiu has manually cleaned up some of the Jenkins slaves.

[tchaikov]

the cortx-motr repo hosted on chacra was wiped out somehow.

[djgalloway]

the cortx-motr repo hosted on chacra was wiped out somehow.

If you don't want them wiped after two weeks, the packages need to be pushed to chacra.ceph.com instead. This is what we do for libboost, for example.

[petrutlucian94]

jenkins test windows

[petrutlucian94]

The code looks good, the formatting issues have been addressed. The CI tests are passing and we thoroughly tested the patch as well.

This change only affects Windows, the HAVE_POLL definition is currently surrounded by a Windows platform check. That being considered, we should merge this fix as soon as possible.

[neha-ojha]

@rzarzynski It will be great if you could take a look at this PR before we merge it. There is no teuthology testing needed on it.