Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

internal/poll: transparently support new linux io_uring interface #31908

Open
johanbrandhorst opened this issue May 8, 2019 · 6 comments

Comments

@johanbrandhorst
Copy link
Member

@johanbrandhorst johanbrandhorst commented May 8, 2019

A document on the latest linux IO syscall has made the rounds of discussion forums on the internet:

http://kernel.dk/io_uring.pdf

I wanted to open a discussion on whether we could (and should) add transparent support for this on supported linux kernels.

LWN article: https://lwn.net/Articles/776703/

@ianlancetaylor ianlancetaylor changed the title syscall: transparently support new linux io_uring interface internal/poll: transparently support new linux io_uring interface May 8, 2019
@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented May 8, 2019

It should be feasible to fit this approach into our current netpoll framework. For some programs I think it would reduce the number of threads doing file I/O. It should potentially reduce the number of system calls required to read from the network.

I'm concerned about coordinating access to the ring. The approach seems designed for high performance communication between the application and the kernel, but it seems easiest to use for an application that uses a single thread for I/O, or in which each I/O thread uses its own ring. In Go of course each goroutine is acting independently, and it seems infeasible for each thread to have a separate ring. so that means that goroutines will need to coordinate their access to the I/O ring. That's fine but on high GOMAXPROCS systems I would worry about contention for the ring, a contention that I think doesn't exist in the current epoll framework.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented May 8, 2019

Thinking about this further, it's not clear that it makes sense to use this new interface for network I/O. It seems that it can only handle a fixed number of concurrent I/O requests, and it's going to be quite hard to make that work transparently and efficiently in Go programs. Without knowing what the Go program plans to do, we can't allocate the ring appropriately.

If that is true, we would only use it for file I/O, where we can reasonably delay I/O operations when the ring is full. In that case it would not fit into the netpoll system. Instead, we might in effect send all file I/O requests to a single goroutine which would add them to the ring, while a second goroutine would sleep until events were ready and wake up the goroutine waiting for them. That should limit the number of threads we need for file I/O and reduce the number of system calls.

@CAFxX

This comment has been minimized.

Copy link
Contributor

@CAFxX CAFxX commented May 8, 2019

Without knowing what the Go program plans to do, we can't allocate the ring appropriately

I was wondering if we could use the map approach: when it becomes full we allocate a new, bigger one, and start submitting requests to the new one. Once the old one doesn't have pending requests anymore we deallocate it.

it seems easiest to use for an application that uses a single thread for I/O, or in which each I/O thread uses its own ring. In Go of course each goroutine is acting independently, and it seems infeasible for each thread to have a separate ring

Can you elaborate on the "infeasible" part? Assuming having multiple rings is feasible, wouldn't having per-P rings work (with the appropriate fallback slow paths in case the per-P ring is full)? I'm not so familiar with the poller, is the problem that the model mismatch is too big?

@coder543

This comment has been minimized.

Copy link

@coder543 coder543 commented May 9, 2019

If that is true, we would only use it for file I/O

File I/O is the main reason I'm interested in io_uring. My understanding is that the current Linux kernel abstractions for async networking work just fine, but making file I/O appear to be interruptible so that goroutines don't block entire OS threads requires using a pool of I/O OS threads. If io_uring could remove the need for a pool of OS threads for file I/O, that seems to make it worthwhile.

Whether having a ring per P (as @CAFxX suggests) or just having a single thread dedicated to managing a ring... either solution seems fine. Unless there's some measurable advantage to io_uring for networking, I don't think it would be that important to switch the networking code at this point.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented May 9, 2019

Using a ring per P seems possible, but it means that when a P is idle some M will have to be sleeping in a io_uring getevent call. And then there will have to be a way to wake up that M if we need it, and some way to avoid confusion if the P gets assigned to a different M. It may be doable but it seems pretty complicated.

@JimChengLin

This comment has been minimized.

Copy link

@JimChengLin JimChengLin commented May 26, 2019

I think it is a killer feature of Linux 5.1. libaio(a wrapper of Linux kernel native async file io api) is broken or at least flawed, which cannot do async buffered file io and could block unexpectedly. The author of io_uring claims all problems that libaio has are all solved. It would be great if Go can utilize the new feature seamlessly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.