New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Writing output to $HOME/tool/temp puts load on Tool Labs NFS server #127

bd808 opened this Issue Apr 13, 2017 · 3 comments


None yet
3 participants

bd808 commented Apr 13, 2017

It would be nicer to write temporary files to the actual /tmp directory rather than to a directory in the tool's $HOME. You may need to write to $HOME eventually for delivery of results via the http service, but doing all the work to the NFS filesystem is causing high I/O load spikes. See for some examination.

Ideally the web interface would use some queuing mechanism to limit the number of parallel conversion jobs that are used to avoid work spikes that overload I/O and processing power on the shared job grid. This could be accomplished using the Redis server as a queue and a small number of dedicated jobs that polled Redis for work to do. I'd actually suggest starting with just a single worker and monitoring the queue depth for a week or two to determine if wait times actually warrant using more than a single worker.


This comment has been minimized.

chasemp commented Apr 13, 2017

'/tmp' would be ideal and is provided on all the Tools exec nodes


This comment has been minimized.

chasemp commented Apr 18, 2017

Without remediation here we may be forced to hot patch this tool or take it down. I'm not sure yet exactly why the IO consumption has increased but it is impacting other tenants within the Tools ecosystem.


This comment has been minimized.


Tpt commented Apr 18, 2017

Hi! Thank you for coming here. Sadly I have not a lot of free time currently so not time to implement a clean Redis queue.

I'll try to work on a change to use /tmp as much as possible in the next few days.

@samwilson Do you know if you could give some time to this tool as part of Community Tech team?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment