Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
[engine] Rust IO #4265
Porting the scheduling of tasks to rust helped to minimize its overhead, but we were still left with a huge number of python objects and the single-core bottleneck of python for the filesystem leaf nodes in the execution graph (the majority).
#4221 and #4261 laid the groundwork for executing concurrent IO on the rust side, and this patch enables that.
One significant difference though, is that the IO API in the engine has raised away from low level filesystem operations toward a higher level abstraction. The most fundamental operation exposed to users is now "get me a
We're able to get multiple cores involved while executing
lgtm w/ a handful of misc comments. thanks Stu!
lgtm. The future loop_fn is really neat. I made a few comments.