Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix CI tests #4561

Merged
merged 2 commits into from
Apr 21, 2020
Merged

Fix CI tests #4561

merged 2 commits into from
Apr 21, 2020

Conversation

chrisburr
Copy link
Member

The CI started failing as elastic search stops accepting more data once disk usage is at 95%. For some reason the amount of available disk space has been decreasing recently causing this to now happen (actions/runner-images#709).

I've also added a stage for showing the logs from elasticsearch to make problems easier to debug in future.

@matthewfeickert
Copy link

@chrisburr Just wanted to mention that the pyhf dev team has found it is enough to run apt-get clean before installing things to have a hack fix.

@chrisburr
Copy link
Member Author

Hey @matthewfeickert! Thanks that will probably be useful in the other repo where I had the same issue and where I actually need the extra space.

I think this PR is still the right fix for the tests here as the CI doesn't need to care what percentage of disk space is remaining. The tests don't produce that much data in elasticsearch and we already reduce the memory reservation for the same reason.

@fstagni
Copy link
Contributor

fstagni commented Apr 21, 2020

@chrisburr nothing prevents to add in here also apt-get clean as suggested (https://github.com/scikit-hep/pyhf/pull/819/files#diff-e9f950f17198d3d5e3122a44230a09b9R29)

@chrisburr
Copy link
Member Author

I thought it would be better not to as it's a hack around a temporary issue with the runners. By the same logic we should remove other large objects that we don't use (boost/65 versions of the .NET SDK/44 versions of the Android SDK/...). It's worth it if we need the extra space but not if we don't.

The changes in this PR will always make sense as GitHub only promises that 14GB of disk will be available. As the runners provide a reasonably complete environment it's plausible that ~100GB will already be used making it very easy to trigger the 95% threshold despite having plenty of disk space available.

@fstagni
Copy link
Contributor

fstagni commented Apr 21, 2020

apt-get clean is innocuous but OK let's leave it out.

@fstagni fstagni merged commit 7782e1c into DIRACGrid:rel-v7r0 Apr 21, 2020
@chrisburr chrisburr deleted the raise-es-disk-thresholds branch April 21, 2020 09:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants