New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Writing output to $HOME/tool/temp puts load on Tool Labs NFS server #127

Open
bd808 opened this Issue Apr 13, 2017 · 3 comments

Comments

Projects
None yet
3 participants
@bd808

bd808 commented Apr 13, 2017

It would be nicer to write temporary files to the actual /tmp directory rather than to a directory in the tool's $HOME. You may need to write to $HOME eventually for delivery of results via the http service, but doing all the work to the NFS filesystem is causing high I/O load spikes. See https://phabricator.wikimedia.org/T161898#3180464 for some examination.

Ideally the web interface would use some queuing mechanism to limit the number of parallel conversion jobs that are used to avoid work spikes that overload I/O and processing power on the shared job grid. This could be accomplished using the Redis server as a queue and a small number of dedicated jobs that polled Redis for work to do. I'd actually suggest starting with just a single worker and monitoring the queue depth for a week or two to determine if wait times actually warrant using more than a single worker.

@chasemp

This comment has been minimized.

chasemp commented Apr 13, 2017

'/tmp' would be ideal and is provided on all the Tools exec nodes

@chasemp

This comment has been minimized.

chasemp commented Apr 18, 2017

Without remediation here we may be forced to hot patch this tool or take it down. I'm not sure yet exactly why the IO consumption has increased but it is impacting other tenants within the Tools ecosystem.

@Tpt

This comment has been minimized.

Member

Tpt commented Apr 18, 2017

Hi! Thank you for coming here. Sadly I have not a lot of free time currently so not time to implement a clean Redis queue.

I'll try to work on a change to use /tmp as much as possible in the next few days.

@samwilson Do you know if you could give some time to this tool as part of Community Tech team?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment