Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tune ThreadedRunner buffer size for better performance #7190

Closed
mih opened this issue Nov 18, 2022 · 3 comments · Fixed by #7250
Closed

Tune ThreadedRunner buffer size for better performance #7190

mih opened this issue Nov 18, 2022 · 3 comments · Fixed by #7250
Labels
performance Improve performance of an existing feature team-runner sub-process execution and IO (https://github.com/datalad/datalad/issues/6365)

Comments

@mih
Copy link
Member

mih commented Nov 18, 2022

Here is the patch: datalad/datalad-next#135

When reading from SSH with the runner I can boost the performance from 30-50MB/s to ~650MB/s with the new default behavior.
@mih mih added performance Improve performance of an existing feature team-runner sub-process execution and IO (https://github.com/datalad/datalad/issues/6365) labels Nov 18, 2022
@yarikoptic
Copy link
Member

Cool! Why not just a PR here?

@mih
Copy link
Member Author

mih commented Nov 18, 2022

We'll soon know how that works out in practice.

bpoldrack added a commit to bpoldrack/datalad that referenced this issue Jan 13, 2023
@bpoldrack
PERF: Buffer size optimization from datalad-next
f034840 
This is a trivial patch from datalad-next, that significantly improves
performance of save,status, etc. b/c of a much too small buffer when
the runner is reading output.

Closes datalad#7190
@bpoldrack bpoldrack mentioned this issue Jan 13, 2023
Merged
bpoldrack added a commit to bpoldrack/datalad that referenced this issue Jan 15, 2023
@bpoldrack
PERF: Buffer size optimization from datalad-next
e889b0d 
This is a trivial patch from datalad-next, that significantly improves
performance of save,status, etc. b/c of a much too small buffer when
the runner is reading output.

Closes datalad#7190
@yarikoptic-gitmate
Copy link
Collaborator

Issue fixed in 0.18.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
performance Improve performance of an existing feature team-runner sub-process execution and IO (https://github.com/datalad/datalad/issues/6365)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

PERF: Buffer size optimization from datalad-next bpoldrack/datalad
3 participants
@yarikoptic @mih @yarikoptic-gitmate