Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(core): io_uring prototype #2298

Merged
merged 9 commits into from Jul 11, 2022
Merged

feat(core): io_uring prototype #2298

merged 9 commits into from Jul 11, 2022

Conversation

puzpuzpuz
Copy link
Contributor

@puzpuzpuz puzpuzpuz commented Jul 6, 2022

Refs #1867

WIP, as early as it can be

TODOs

  • Add more operations
  • Consider getting rid of liburing or at least avoid linking
  • Think of a more convenient API
  • Make sure that we check ulimit -l on startup and suggest the user to increase the limit

Sample

try (IOUring ring = new IOUring(rf, 32)) {
    long id = ring.enqueueRead(fd, 0, bufPtr, bufLen);
    if (id == -1) {
        // ring is full
    }
    // enqueue more operations ...
    ring.submit();
    
    // start waiting for them
    while (!ring.nextCqe()) {
        Os.pause();
    }
    long completedId = ring.getCqeId();
    int completedRes = ring.getCqeRes();
    if (completedRes < 0) {
        // operation failed
    }
    // do something with the buffer 
}

@puzpuzpuz puzpuzpuz added the Enhancement Enhance existing functionality label Jul 6, 2022
@puzpuzpuz puzpuzpuz self-assigned this Jul 6, 2022
@puzpuzpuz puzpuzpuz added the Performance Performance improvements label Jul 6, 2022
@bluestreak01
Copy link
Member

Looking very good! Ideally we'd need windows and OSX abstractions too, so that Java impls don't have to suffer.

Additionally, core/src/main/resources/io/questdb/bin/linux/libasm.a does not need to be checked in, common mistake

@puzpuzpuz
Copy link
Contributor Author

puzpuzpuz commented Jul 6, 2022

liburing will very likely go away entirely.

As for having a single abstraction for all async file APIs, it's tricky. The usual approach is to implement it via a thread pool.

We could do that or try to implement different behavior on Linux only. I'd propose to finalize io_uring API on its own, merge it and then try to come up with a common abstraction separately.

@puzpuzpuz
Copy link
Contributor Author

@eugenels thanks for dealing with cmake and static linking!

@bluestreak01 bluestreak01 marked this pull request as ready for review July 11, 2022 13:11
@ideoma
Copy link
Collaborator

ideoma commented Jul 11, 2022

[PR Coverage check]

😍 pass : 102 / 116 (87.93%)

file detail

path covered line new line coverage
🔵 io/questdb/std/IOUringFacadeImpl.java 23 29 79.31%
🔵 io/questdb/std/IOUring.java 55 62 88.71%
🔵 io/questdb/std/IOUringAccessor.java 24 25 96.00%

@bluestreak01 bluestreak01 merged commit 3f15130 into master Jul 11, 2022
@bluestreak01 bluestreak01 deleted the io_uring_prototype branch July 11, 2022 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement Enhance existing functionality Performance Performance improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants