Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using up inode limit #32

Closed
SaladFork opened this issue Dec 12, 2016 · 6 comments
Closed

Using up inode limit #32

SaladFork opened this issue Dec 12, 2016 · 6 comments

Comments

@SaladFork
Copy link

We are building a medium-sized Ember application in a CI environment running Jenkins on an AWS EC2 instance. We are finding that /tmp/if-you-need-to-delete-this-open-an-issue-async-disk-cache never gets cleaned out and continues to grow until we run out of inodes. Clearing it today brought our inode usage on the machine down from 95% to 30%. This is causing all our other builds to fail as Jenkins is no longer able to create new files.

@stefanpenner
Copy link
Owner

stefanpenner commented Dec 12, 2016

@SaladFork interesting, many OS's clean this out periodically. Usually based on last access.

If yours does not, maybe periodic cleaning is required?

Alternatively, AsyncDiskCache is configurable, meaning you can choose where its cache resides. For a CI server, this may then be configured, so that as part of the regular CI task run, a cache for any given build can be manually cleaned up

@SaladFork
Copy link
Author

Yep, we're setting up a cron job to clean it out automatically. Given the folder name we thought it best to file an issue too.

@stefanpenner
Copy link
Owner

stefanpenner commented Dec 12, 2016

@SaladFork yup, good call opening this issue. As maybe we can do something better here.

My understanding, was that most linux did something similar to OSX. For example I believe Ubuntu uses tmpreaper for this task.

Out of curiousity which OS is on your CI environment?

@SaladFork
Copy link
Author

SaladFork commented Dec 12, 2016

We are using Ubuntu 14.04.4 LTS; odd that tmpreaper (or such) isn't cleaning up here.

Oh...

The program 'tmpreaper' is currently not installed. You can install it by typing:
sudo apt-get install tmpreaper

@stefanpenner
Copy link
Owner

stefanpenner commented Dec 12, 2016

@SaladFork askubuntu claims Ubuntu 14.04 uses tmpreaper, which is invoked from etc/cron.daily http://askubuntu.com/a/759048

Prior to 14.04 http://askubuntu.com/a/20831 claims /etc/init/mounted-tmp.conf which runs each time tmp is mounted, which in practice is typically on restart.

Obviously the restart variant is not desirable for your usecase. I believe success lies in figuring out whats-up with tmpreaper being missing..

@SaraaraS
Copy link

SaraaraS commented Jan 6, 2017

Ran into the same issue. Jenkins on aws ec2 medium instance running Ubuntu 14.04.4 LTS, inodes got eaten up, logging in and installing tmpreaper and adding it to the daily cron fixed the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants