New thread is spawned for every I/O request #731
Comments
|
Thanks for report. |
|
@k-nasa Here's a sample code to reproduce the problem use async_std::prelude::*;
fn main() {
async_std::task::block_on(async {
let mut f1 = async_std::fs::File::open("/dev/zero")
.await
.expect("File::open");
let mut f2 = async_std::fs::File::create("/dev/null")
.await
.expect("File::create");
let mut buf = [0u8; 1];
for _ in 0..10000 {
f1.read(&mut buf).await.expect("read");
f2.write(&buf).await.expect("write");
}
()
});
}I'm using latest It read/write about 10k times (as expected), and around 19k threads are created/destroyed during the process. |
|
Edit: I may have misread the code to perform a lot more Here's what I think is happening:
Putting this together with the code sample:
I'm not sure what the best way to resolve this is. Plausible that some |
|
I don't know how the internals are working, but shouldn't there be a policy that allows to spawn a predefined number of threads say N, which are waiting for tasks to be available. Then when the tasks are available, schedule those M tasks across the N available threads? |
|
I've looked back at the code again and noticed that it performs 1
For the particular
|
|
Ok, so I've dug a bit further: It appears that we're hitting the There's also one in So my previous incorrect comment is still somewhat informative here. |
Correct me if I am wrong, but this should hold true for regular files on all *nix systems, since it is assumed that a local file is always accessible and disk reads/writes are not slow enough to block. But the same should not be true for "special" character special devices such as /dev/zero and /dev/null. There is no reason for kernel to block reads and writes to these files, even on linux. On side note since you guys are using mio, my guess would be it is using epoll internally on linux, in which case it is important to note that epoll doesn't work on regular files. I also highly doubt that it would work on files like /dev/zero, which always has something to read. |
|
@adityashah1212 sure, that sounds right. Internally, As far as I can tell, there is no special casing for non-normal (ie: device) files. |
|
@jmesmon then wouldn't it better to have two sets of API's in two different modules, one performing blocking and other performing non-blocking File IO operations. Let the developer choose which ones s/he wants to use? Edit: This one is important for case where we are interacting with some custom device node under /dev which might block until data is available, like say a Linux UIO driver. |
|
Yes, there are non-normal-file use cases where one would want to wait on the fd instead of spawning a separate thread for each operation. I don't know what async-std's thoughts on where those should live are. As far as blocking operations, I believe one would just use the |
When trying to run a sequential I/O with
async-std, it seems that new thread is spawned for every I/O request.Here's strace log for a test workload, which shows that a thread is spawned for every request.
The workload is recorded with
and filtered with
grep -vP '(futex|sched|map|sig|prote|prctl|robust_|madvise|exited)' strace.logThe text was updated successfully, but these errors were encountered: