[CMake] add liburing build option #5919

mxxo · 2020-06-23T22:49:03Z

This PR allows us to use liburing, the "application-level" interface to io_uring, the new Linux kernel IO interface.
liburing takes care of many of the low-level details (e.g. mmap calls, barriers) required to correctly use io_uring.

io_uring is supported on Linux kernels 5.1 and up, but it's up to the user to download and install liburing themselves, either from source (make && make install) or through a package manager.

The rationale for including liburing in ROOT is to experiment with io_uring's parallel, async IO features. This may drive performance improvements in low-level IO.

I based the CMake changes on how jemalloc is handled. I am not very experienced with CMake and would be happy to find out that I've done something wrong here. We need to able to include (at least) two liburing header files and link against the shared library, namely

#include "liburing.h"
#include "liburing/io_uring.h"
liburing.so

phsft-bot · 2020-06-23T22:49:13Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos7-multicore/default, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac1015/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2020-06-23T23:53:13Z

Build failed on mac1014/python3.
Running on macitois21.cern.ch:/build/jenkins/workspace/root-pullrequests-build
See cdash.
See console output.

Failing tests:

phsft-bot · 2020-06-24T02:49:34Z

Build failed on windows10/cxx14.
Running on null:C:\build\workspace\root-pullrequests-build
See cdash.
See console output.

jblomer · 2020-06-25T11:33:16Z

Looks good to me but I'd like to pass approval to someone with more experience on the build system, @oshadura or @pcanal?

oshadura

Nice job! I have other suggestions how to improve a bit more build system, but I think it will be in next iteration of review :)

oshadura · 2020-06-26T10:02:21Z

cmake/modules/Findliburing.cmake

@@ -0,0 +1,23 @@
+# Copyright (C) 1995-2020, Rene Brun and Fons Rademakers.


I found other nice example https://github.com/tkowalcz/io_uring_tutorial/blob/master/FindLibUring.cmake

cmake/modules/SearchInstalledSoftware.cmake

cmake/modules/RootBuildOptions.cmake

phsft-bot · 2020-06-26T16:30:54Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos7-multicore/default, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac1015/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2020-06-30T13:58:27Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos7-multicore/default, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac1015/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2020-06-30T13:58:41Z

Build failed on windows10/cxx14.
See cdash.
See console output.

phsft-bot · 2020-06-30T14:18:01Z

Build failed on ROOT-performance-centos7-multicore/default.
Running on olhswep22.cern.ch:/data/sftnight/workspace/root-pullrequests-build
See cdash.
See console output.

Failing tests:

phsft-bot · 2020-06-30T14:19:39Z

Build failed on ROOT-debian10-i386/cxx14.
Running on pcepsft10.dyndns.cern.ch:/build/workspace/root-pullrequests-build
See cdash.
See console output.

Failing tests:

projectroot.roottest.root.io.withoutDictionaries.roottest_root_io_withoutDictionaries_IOWithoutDictionaries

phsft-bot · 2020-06-30T15:22:19Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos7-multicore/default, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac1015/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2020-06-30T15:22:30Z

Build failed on windows10/cxx14.
See cdash.
See console output.

phsft-bot · 2020-06-30T15:46:01Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos7-multicore/default, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac1015/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2020-06-30T15:46:17Z

Build failed on windows10/cxx14.
See cdash.
See console output.

phsft-bot · 2020-06-30T16:06:08Z

Build failed on mac1015/cxx17.
Running on macphsft18.dyndns.cern.ch:/Users/sftnight/build/jenkins/workspace/root-pullrequests-build
See cdash.
See console output.

Failing tests:

projectroot.roottest.root.io.withoutDictionaries.roottest_root_io_withoutDictionaries_IOWithoutDictionaries

phsft-bot · 2020-06-30T16:06:54Z

Build failed on ROOT-debian10-i386/cxx14.
Running on pcepsft10.dyndns.cern.ch:/build/workspace/root-pullrequests-build
See cdash.
See console output.

Failing tests:

projectroot.roottest.root.io.withoutDictionaries.roottest_root_io_withoutDictionaries_IOWithoutDictionaries

phsft-bot · 2020-06-30T16:11:48Z

Build failed on mac1014/python3.
Running on macphsft17.dyndns.cern.ch:/build/jenkins/workspace/root-pullrequests-build
See cdash.
See console output.

Failing tests:

projectroot.roottest.root.io.withoutDictionaries.roottest_root_io_withoutDictionaries_IOWithoutDictionaries

phsft-bot · 2020-06-30T16:12:34Z

Build failed on ROOT-performance-centos7-multicore/default.
Running on olhswep22.cern.ch:/data/sftnight/workspace/root-pullrequests-build
See cdash.
See console output.

Failing tests:

phsft-bot · 2020-06-30T16:29:23Z

Build failed on ROOT-fedora31/noimt.
Running on root-fedora-31-2.cern.ch:/home/sftnight/build/workspace/root-pullrequests-build
See cdash.
See console output.

Failing tests:

projectroot.bindings.pyroot.pythonizations.test.pyunittests_pyroot_numbadeclare

phsft-bot · 2020-06-30T16:39:07Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos7-multicore/default, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac1015/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2020-06-30T16:39:19Z

Build failed on windows10/cxx14.
See cdash.
See console output.

phsft-bot · 2020-06-30T16:47:26Z

Build failed on ROOT-fedora31/noimt.
Running on root-fedora-31-2.cern.ch:/home/sftnight/build/workspace/root-pullrequests-build
See cdash.
See console output.

Failing tests:

projectroot.bindings.pyroot.pythonizations.test.pyunittests_pyroot_numbadeclare

phsft-bot · 2020-06-30T16:58:11Z

Build failed on ROOT-performance-centos7-multicore/default.
Running on olhswep22.cern.ch:/data/sftnight/workspace/root-pullrequests-build
See cdash.
See console output.

Failing tests:

phsft-bot · 2020-06-30T17:06:45Z

Build failed on mac1015/cxx17.
Running on macphsft18.dyndns.cern.ch:/Users/sftnight/build/jenkins/workspace/root-pullrequests-build
See cdash.
See console output.

Failing tests:

projectroot.roottest.root.io.withoutDictionaries.roottest_root_io_withoutDictionaries_IOWithoutDictionaries

phsft-bot · 2020-06-30T17:28:18Z

Build failed on ROOT-ubuntu16/nortcxxmod.
Running on sft-ubuntu-1604-4.cern.ch:/build/workspace/root-pullrequests-build
See cdash.
See console output.

Failing tests:

And 10 more

phsft-bot · 2020-06-30T17:47:23Z

Build failed on mac1014/python3.
Running on macitois21.cern.ch:/build/jenkins/workspace/root-pullrequests-build
See cdash.
See console output.

Failing tests:

phsft-bot · 2020-08-04T18:38:38Z

Build failed on ROOT-fedora30/cxx14.
Running on root-fedora30-1.cern.ch:/build/workspace/root-pullrequests-build
See cdash.
See console output.

Failing tests:

projectroot.bindings.pyroot.pythonizations.test.pyunittests_pyroot_numbadeclare

phsft-bot · 2020-08-04T18:44:28Z

Build failed on ROOT-ubuntu16/nortcxxmod.
Running on sft-ubuntu-1604-4.cern.ch:/build/workspace/root-pullrequests-build
See cdash.
See console output.

Failing tests:

And 37 more

phsft-bot · 2020-08-04T19:21:05Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos7-multicore/default, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac1015/cxx17, windows10/cxx14
How to customize builds

Axel-Naumann · 2020-08-04T19:51:12Z

All errors seem unrelated. @etejedor will hopefully look at the Python errors when he's back. And I haven't seen the cling symbol errors in my PR - which is why I merged it, and apparently I broke cling :-( I'll have to debug tomorrow...

phsft-bot · 2020-08-04T20:10:33Z

Build failed on ROOT-ubuntu16/nortcxxmod.
Running on sft-ubuntu-1604-4.cern.ch:/build/workspace/root-pullrequests-build
See cdash.
See console output.

Failing tests:

And 37 more

phsft-bot · 2020-08-04T20:17:31Z

Build failed on mac1014/python3.
Running on macitois21.cern.ch:/build/jenkins/workspace/root-pullrequests-build
See cdash.
See console output.

Failing tests:

projectroot.roottest.root.io.clones.roottest_root_io_clones_make

Axel-Naumann · 2020-08-05T11:33:07Z

io/io/test/CMakeLists.txt

@@ -9,3 +9,6 @@ ROOT_ADD_GTEST(TFile TFileTests.cxx LIBRARIES RIO)
 ROOT_ADD_GTEST(TBufferMerger TBufferMerger.cxx LIBRARIES RIO Imt Tree)
 ROOT_ADD_GTEST(TFileMerger TFileMergerTests.cxx LIBRARIES RIO Tree)
 ROOT_ADD_GTEST(TROMemFile TROMemFileTests.cxx LIBRARIES RIO Tree)
+if(uring AND NOT DEFINED ENV{ROOTTEST_IGNORE_URING})
+  ROOT_ADD_GTEST(RIoUring RIoUring.cxx LIBRARIES RIO uring)


Isn't uring inherited through RIO? I'd hope that you don't need to spell out LIBRARIES uring here:

Suggested change

ROOT_ADD_GTEST(RIoUring RIoUring.cxx LIBRARIES RIO uring)

ROOT_ADD_GTEST(RIoUring RIoUring.cxx LIBRARIES RIO)

The missing piece here is that I didn't make target_link_libraries PUBLIC instead of PRIVATE. I think it is fixed in 23d2a9f

phsft-bot · 2020-08-05T12:36:21Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos7-multicore/default, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac1015/cxx17, windows10/cxx14
How to customize builds

mxxo · 2020-08-05T12:41:25Z

@phsft-bot build also on ROOT-fedora32/default with flags -During=ON

phsft-bot · 2020-08-05T12:41:33Z

Starting build on ROOT-fedora32/default, ROOT-debian10-i386/cxx14, ROOT-performance-centos7-multicore/default, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac1015/cxx17, windows10/cxx14 with flags -During=ON
How to customize builds

phsft-bot · 2020-08-05T12:41:57Z

Build failed on ROOT-fedora30/cxx14.
Running on root-fedora30-1.cern.ch:/build/workspace/root-pullrequests-build
See cdash.
See console output.

Errors:

[2020-08-05T12:41:56.690Z] CMake Error at cmake/modules/SearchInstalledSoftware.cmake:1143 (message):
[2020-08-05T12:41:56.690Z] CMake Error at /build/workspace/root-pullrequests-build/rootspi/jenkins/root-build.cmake:1048 (message):

phsft-bot · 2020-08-05T12:42:00Z

Build failed on ROOT-fedora31/noimt.
Running on root-fedora-31-2.cern.ch:/home/sftnight/build/workspace/root-pullrequests-build
See cdash.
See console output.

Errors:

[2020-08-05T12:41:59.944Z] CMake Error at cmake/modules/SearchInstalledSoftware.cmake:1143 (message):
[2020-08-05T12:41:59.944Z] CMake Error at /home/sftnight/build/workspace/root-pullrequests-build/rootspi/jenkins/root-build.cmake:1048 (message):

mxxo · 2020-08-05T12:43:36Z

build also on ROOT-fedora32/default with flags -During=ON

I thought this would only build fedora32 with uring, sorry about that

Axel-Naumann · 2020-08-05T13:02:42Z

NP! FYI that's just inplace of also. But it's good you tried to build everywhere, to check the config failure!

phsft-bot · 2020-08-05T13:11:06Z

Build failed on ROOT-fedora32/default.
Running on root-fedora32-1.cern.ch:/home/sftnight/build/workspace/root-pullrequests-build
See cdash.
See console output.

Warnings:

[2020-08-05T12:43:36.840Z] include/Vc/scalar/vector.h:218:55: warning: ‘V2 Vc_1::Vector<T, Vc_1::VectorAbi::Scalar>::reinterpretCast() const’ is deprecated: use reinterpret_components_cast instead [-Wdeprecated-declarations]
[2020-08-05T12:43:37.114Z] include/Vc/scalar/vector.h:218:55: warning: ‘V2 Vc_1::Vector<T, Vc_1::VectorAbi::Scalar>::reinterpretCast() const’ is deprecated: use reinterpret_components_cast instead [-Wdeprecated-declarations]
[2020-08-05T12:47:22.990Z] include/Vc/scalar/vector.h:218:55: warning: ‘V2 Vc_1::Vector<T, Vc_1::VectorAbi::Scalar>::reinterpretCast() const’ is deprecated: use reinterpret_components_cast instead [-Wdeprecated-declarations]
[2020-08-05T12:48:03.868Z] math/unuran/unuran-1.8.0-root/src/methods/mvtdr_init.ch:886:17: warning: argument 1 value ‘18446744073709551608’ exceeds maximum object size 9223372036854775807 [-Walloc-size-larger-than=]

Failing tests:

phsft-bot · 2020-08-05T13:13:28Z

Build failed on ROOT-debian10-i386/cxx14.
Running on pcepsft10.dyndns.cern.ch:/build/workspace/root-pullrequests-build
See cdash.
See console output.

Errors:

[2020-08-05T13:13:27.086Z] CMake Error at cmake/modules/SearchInstalledSoftware.cmake:1143 (message):
[2020-08-05T13:13:27.086Z] CMake Error at /home/sftnight/build/workspace/root-pullrequests-build/rootspi/jenkins/root-build.cmake:1048 (message):

phsft-bot · 2020-08-05T13:15:02Z

Build failed on ROOT-performance-centos7-multicore/default.
Running on olhswep22.cern.ch:/data/sftnight/workspace/root-pullrequests-build
See cdash.
See console output.

Errors:

[2020-08-05T13:15:01.332Z] CMake Error at cmake/modules/SearchInstalledSoftware.cmake:1143 (message):
[2020-08-05T13:15:01.332Z] CMake Error at /data/sftnight/workspace/root-pullrequests-build/rootspi/jenkins/root-build.cmake:1048 (message):

phsft-bot · 2020-08-05T14:05:36Z

Build failed on ROOT-ubuntu16/nortcxxmod.
Running on sft-ubuntu-1604-4.cern.ch:/build/workspace/root-pullrequests-build
See cdash.
See console output.

Errors:

[2020-08-05T14:05:34.695Z] CMake Error at cmake/modules/SearchInstalledSoftware.cmake:1143 (message):
[2020-08-05T14:05:34.695Z] CMake Error at /mnt/build/workspace/root-pullrequests-build/rootspi/jenkins/root-build.cmake:1048 (message):

mxxo · 2020-08-06T13:47:20Z

@Axel-Naumann the tests on fedora32 look good, and all the other linux nodes "failed successfully" during the configuration search :)

Axel-Naumann · 2020-08-06T20:33:10Z

Thanks, Max. Do you want me to merge or do you prefer to wait for Jakob?

mxxo · 2020-08-06T21:35:45Z

I am definitely OK waiting for Jakob. I believe all the outstanding issues are fixed up, but waiting one more day doesn't block anything (I can keep going on my local branch and make a new PR for that).

jblomer

Looks good to me! Can't wait to benchmark SSD performance with and without async I/O.

jblomer · 2020-08-10T12:37:30Z

@Axel-Naumann I think all is addressed, please approve if it looks good to you

mxxo requested review from oshadura and pcanal as code owners June 23, 2020 22:49

couet assigned oshadura Jun 25, 2020

oshadura suggested changes Jun 26, 2020

View reviewed changes

mxxo requested a review from oshadura June 26, 2020 16:34

[io] (NFC) add todo for checking submission queue is empty in dtor

63eb63e

Axel-Naumann approved these changes Aug 5, 2020

View reviewed changes

Axel-Naumann requested changes Aug 5, 2020

View reviewed changes

[CMake] reexport liburing symbols through RIO library at CMake level

23d2a9f

mxxo requested a review from Axel-Naumann August 5, 2020 12:44

jblomer approved these changes Aug 10, 2020

View reviewed changes

Axel-Naumann approved these changes Aug 10, 2020

View reviewed changes

jblomer merged commit 1dea761 into root-project:master Aug 11, 2020

mxxo mentioned this pull request Aug 21, 2020

[io] io_uring ReadV implementation #6162

Merged

		@@ -0,0 +1,23 @@
		# Copyright (C) 1995-2020, Rene Brun and Fons Rademakers.

	ROOT_ADD_GTEST(RIoUring RIoUring.cxx LIBRARIES RIO uring)
	ROOT_ADD_GTEST(RIoUring RIoUring.cxx LIBRARIES RIO)

[CMake] add liburing build option #5919

[CMake] add liburing build option #5919

Conversation

mxxo commented Jun 23, 2020

phsft-bot commented Jun 23, 2020

phsft-bot commented Jun 23, 2020

Failing tests:

phsft-bot commented Jun 24, 2020

jblomer commented Jun 25, 2020

oshadura left a comment

Choose a reason for hiding this comment

oshadura Jun 26, 2020

Choose a reason for hiding this comment

phsft-bot commented Jun 26, 2020

phsft-bot commented Jun 30, 2020

phsft-bot commented Jun 30, 2020

phsft-bot commented Jun 30, 2020

Failing tests:

phsft-bot commented Jun 30, 2020

Failing tests:

phsft-bot commented Jun 30, 2020

phsft-bot commented Jun 30, 2020

phsft-bot commented Jun 30, 2020

phsft-bot commented Jun 30, 2020

phsft-bot commented Jun 30, 2020

Failing tests:

phsft-bot commented Jun 30, 2020

Failing tests:

phsft-bot commented Jun 30, 2020

Failing tests:

phsft-bot commented Jun 30, 2020

Failing tests:

phsft-bot commented Jun 30, 2020

Failing tests:

phsft-bot commented Jun 30, 2020

phsft-bot commented Jun 30, 2020

phsft-bot commented Jun 30, 2020

Failing tests:

phsft-bot commented Jun 30, 2020

Failing tests:

phsft-bot commented Jun 30, 2020

Failing tests:

phsft-bot commented Jun 30, 2020

Failing tests:

phsft-bot commented Jun 30, 2020

Failing tests:

phsft-bot commented Aug 4, 2020

Failing tests:

phsft-bot commented Aug 4, 2020

Failing tests:

phsft-bot commented Aug 4, 2020

Axel-Naumann commented Aug 4, 2020

phsft-bot commented Aug 4, 2020

Failing tests:

phsft-bot commented Aug 4, 2020

Failing tests:

Axel-Naumann Aug 5, 2020

Choose a reason for hiding this comment

mxxo Aug 5, 2020

Choose a reason for hiding this comment

phsft-bot commented Aug 5, 2020

mxxo commented Aug 5, 2020

phsft-bot commented Aug 5, 2020

phsft-bot commented Aug 5, 2020

Errors:

phsft-bot commented Aug 5, 2020

Errors:

mxxo commented Aug 5, 2020

Axel-Naumann commented Aug 5, 2020 • edited

phsft-bot commented Aug 5, 2020

Warnings:

Failing tests:

phsft-bot commented Aug 5, 2020

Errors:

phsft-bot commented Aug 5, 2020

Errors:

phsft-bot commented Aug 5, 2020

Errors:

mxxo commented Aug 6, 2020 • edited

Axel-Naumann commented Aug 6, 2020

Axel-Naumann commented Aug 5, 2020 •

edited

mxxo commented Aug 6, 2020 •

edited