Ubuntu images are running out of disk space #709
Comments
|
Hello @jneira , it is expected behavior. Based on official documentation, hosted agents provide at least 10 GB of storage for your source and build outputs. As a possible workarounds, you can remove the part of pre-installed software in runtime. For example, this command will release 5+ of free space: |
maxim-lobanov
added
question
|
@maxim-lobanov thanks for clarification, i didnt know about that limit, but i have to admit it is reasonable. haskell builds can take a lot of space, more than usual for other langs. I can workaround the issue using the preinstalled Tnaks again for the quick response. |
|
@maxim-lobanov thanks a lot for posting this workaround. EDIT: this is the change: apache/flink@cb8ae28 |
|
So I'm pretty sure this is an actual new bug. I started running into it on Ubuntu 18.04 just a few days ago. If you look a the output of df, you will notice that the build isn't using any disk space at all on /mnt, which is where most of the disk space is available. I am guessing that /home/runner/work used to be located on /mnt, but now is being left on /, and so we have gone from having 14GB for source + build outputs and a few gigabytes for extra software we install to having only a few gigabytes total for the entire build operation. I can manually move my builds to try to use space on /mnt, but it seems like this really did change this week, I guess first for Ubuntu 16.04 and soon thereafter for Ubuntu 18.04, and that it would be better if the runner was automatically placed on /mnt. (I wish I could re-open this bug, but sadly I can't; if no one sees this comment I guess I'll open a new bug and reference this one? @maxim-lobanov) |
|
@alepauly, Does it make sense to extend disk space on Ubuntu 16.04/18.04 from 84gb to at least 128gb? Ubuntu 16.04: Ubuntu 18.04: |
@saurik Maybe i did something wrong but i tried to use |
We saw the same thing for pyhf. We did a hack fix in pyhf PR 819 by running |
|
FWIW the same thing is happening on 18.04. I have a Docker build that has been working for months that stopped working last week. |
|
@jneira I'd totally be willing to believe /mnt isn't directly usable; like, I wasn't saying you could definitely solve the problem by using /mnt: what I was even saying is that I purposefully didn't try, as this was previously working and I expect that any temporary workaround to manually try to move something to /mnt, assuming it worked, would break later. Like, maybe it would be less confusing if I just talked about /dev/sdb1 instead of /mnt, as that partition could potentially have been mounted to /home or something before: I have no clue what the previous configuration of this system was where we had lots and lots more space was; but I've seen lots of people talk about these machines having 14GB of secondary disk available and GitHub saying we have at least 10 GB of storage for code and build outputs, yet for that all to be true we would have to be working on that secondary disk, not the primary disk we are currently defaulting to. |
@alcheb - Unfortunately we can't easily do that at the time. Thanks everyone(@saurik, @Chuxel, @matthewfeickert, @jneira, @rmetzger) for the input and apologies for the pain this is causing! Seems to me the simplest workaround is to We're looking for mitigations we can apply quickly and safely but in the meantime, adding the workaround manually is your best bet to unblock your workflows. We'll post an update here as soon as we get something rolling. |
|
I can agree with all you :) Our build was running smooth for couple weeks / months. We were using /mnt as docker root. We are running reallly simple minikube on runnners and now we are not able to even initialize first minikube log. Same disk usage about 91%. /mnt disk is for some reason not usable now. |
@obabec, we'll look into this. |
Thanks very much for the reply and status update @alepauly. This is useful to know how we can proceed for the time being (for the pyhf dev team we're already doing this). 👍 I'll add that as a naive user to all the intricacies of what is actually happening, it was very strange that this is happening to only the |
|
@matthewfeickert fwiw, I would also view it as a bug - because you're not exceeding the space we make available, and because things all of a sudden changed and broke you. We try hard for this not to happen but it happens sometimes. We'll figure out a solution as soon as possible (probably will take a day or two to replicate everywhere) so you can remove the workaround. |
|
We have the same issue with As app developers, we don't want to manage any VMs (i.e. self-hosted runners). Instead, we would prefer trading a few more build minutes for more disk space.💰💰💰 |
Sounds great. Thanks very much for responding quickly on this and for all the hard work! |
Jobs are failing due to "no space left on device" when building the container image. Adding some df output and trying if cleaning up some stuff before the build helps. See: actions/runner-images#709 Signed-off-by: Michael Adam <obnox@redhat.com>
Jobs are failing due to "no space left on device" when building the container image. Adding some df output and trying if cleaning up some stuff before the build helps. See: actions/runner-images#709 Signed-off-by: Michael Adam <obnox@redhat.com>
Jobs are failing due to "no space left on device" when building the container image. Adding some df output. See: actions/runner-images#709 Signed-off-by: Michael Adam <obnox@redhat.com>
Jobs are failing due to "no space left on device" when building the container image. Adding some df output. See: actions/runner-images#709 Signed-off-by: Michael Adam <obnox@redhat.com>
Jobs are failing due to "no space left on device" when building the container image. Adding some df output. See: actions/runner-images#709 Signed-off-by: Michael Adam <obnox@redhat.com>
…cebook#23) Summary: As per actions/runner-images#709 there started to be some issies with Ubuntu envs running out of space. This should fix it. Also our Cargo builds use a lot of space, changing them to be non-incremental and removing debug symbols keeps the build fast, but greatly reduces the disk space usage leaving us enough space on GitHub Actions virtual machines. Pull Request resolved: facebook/sapling#23 Reviewed By: farnz Differential Revision: D22160020 Pulled By: lukaspiatkowski fbshipit-source-id: c23393e310c15ebf5a18b80f0bb5f1f894d24849
…n 3.1.23
David Hotham (1):
Fix typing of Head.create_head
Dominic (79):
Import typevar in util.py
flake8 fix
fix typo
another typo
Add files via upload
Add files via upload
Update test-requirements.txt
Update test_pytest.yml
Update and rename test_pytest.yml to Future.yml
Delete Future.yml
Update pythonpackage.yml
Update dev-requirements.txt
Update tox.ini
Update setup.py
Rename dev-requirements.txt to requirements-dev.txt
Update pythonpackage.yml
Update pythonpackage.yml
Update test-requirements.txt
Update pythonpackage.yml
Add pytests args
Update pythonpackage.yml
Update pythonpackage.yml
rmv tox from test-requirements.txt
Delete tox.ini
Delete mypy.ini
Delete .coveragerc
Create pyproject.toml
Create .flake8
Delete .codeclimate.yml
Delete .deepsource.toml
Update pshinx versions in docs/reqs.txt
Update conf.py
Update requirements-dev.txt
Update pythonpackage.yml
Update .flake8
Update pyproject.toml
Add mypy to test-requirements.txt
Add mypy to test-requirements.txt
Add sphinx-autodoc-typehints
Update README.md
Update README.md
Update README.md
update typing-extensions to 3.10.0
re-add package data for py.typed
fix find_packages()
Rmv EZ_setup from setup.py
Add build system to pyproject.toml
Update requirements.txt
Update util.py
Update util.py
Update base.py
Update test_refs.py
Update test_refs.py
Update config.py
Update tag.py
Update head.py
Update head.py
Update head.py
Update reference.py
update types commit.py
update types submodule/base.py
update types submodule/root.py
update types sybmbolic.py
rvrt symbolic.py types
Update symbolic.py
Update symbolic.py
Rvrt types of symbolic.py that were breaking pytest
Add overload to get_value()
Update pythonpackage.yml
Update pythonpackage.yml
Update setup.py
Update pythonpackage.yml
Update pythonpackage.yml
Update pythonpackage.yml
Update setup.py
Update setup.py
Update setup.py
Update changes.rst
Update changes.rst
Eric Wieser (8):
Do not call get_user_id if it is not needed
Fix mypy
Update config.py
Fix trailing whitespace and incorrect overload
Use get instead of get_value
Fix test
Try a better test
Update test_util.py
Igor Lakhtenkov (2):
Added clone multi_options to Submodule
Added support of spaces for clone multi_options
Kai Mueller (1):
Add flake8-typing-imports
Max Fan (2):
Fix exsit typo
Add missing article to error message
Michael Käufl (1):
Fix link to latest changelog
Michael Mulich (1):
Use the Git class type definition within Repo classmethods
Sebastian Thiel (16):
Run actions on main branch
remove duplication
Remove accidental file and assure they don't come back
Remove docker and appveyor configuration files
put badges (also) upfront
Make development status clearer
prepare patch release
prepare next patch release
fix docs
More specific version check for ordered dict type
Revert "More specific version check for ordered dict type"
Don't fail on import if the working dir isn't valid (#1319)
overhaul CONTIRIBUTING.md
Assure CWD is readable after acquiring it
bump version
Fix version discrepancy
Yobmod (225):
Update typing-extensions version in requirements.txt
Update typing-extensions version in requirements.txt
Add types to tree.Tree
Make IterableList generic and update throughout
Rename Iterable due to typing.Iterable. Add deprecation warning
Make Iterable deprecation warning on subclassing
fix an import
fix indent
update docstring
update some TBDs to configparser
Add typedDict
Add T_Tre_cache TypeVar
forward ref Gitconfigparser
Import TypeGuard to replace casts
Update typing-extensions dependancy to <py3.10 for typeguard
Add asserts and casts for T_Tree_cache
Add is_flatLiteral() Typeguard[] to remote.py
fix assert
Add '?' to controlcharacter literal
replace cast()s with asserts in remote.py
replace cast()s with asserts in fun.py
trigger checks to rurun
Add type to submodule to trigger checks to rurun
move py.typed from setup.py to MANIFEST.INI
Type Traversable.traverse() better, start types of submodule
rmv github squashed commit file
Type Tree.traverse() better
Type Traversable/list_traverse() better, make IterablleObj a protocol
Fix forward ref
Put protocol behind py version check
Flake 8 fix
Add missed types to Commit, uncomment to_native_path_linux()
Make Commit.message bytes | str
Improve typing of config_levels, add assert_never()
Fix IndexFile forwardref
put typing_extensions.get_types() behind python version guard
fix is_config_level for < 3.8
Rmv is_config_level() and get_args(), not worth the trouble
Add Literal_config_levels.__args__
Improve BlameEntry.commit typing
Add types to submodule.util.py
Make bytesIO forwardref
Add types to submodule.root.py
Add types to submodule.update()
Improve types of @unbare_repo and @git_working_dir decorators
Improve types of diff.py
Improve types of diff.py
Add cast(Repo, mrepo) in try block
Fix Literal Typeguards
Fix for mrepo
Fix for mrepo2
Rmv runtime_checkable < py3.8
Rmv runtime_checkable < py3.8 pt2
Rmv root.py types
Rmv base.py types
Rmv submodule types
Rmv submodule types2
Rmv diff typeguard
Re-add submodule.util.py types
Fix submodule.util.py types
Readd typeguard to Diff.py
Readd submodule.base.py types
Make subodule a forward ref in Index.base
Make subodule a forward ref in Index.base2
Make Repo a forward ref in Submodule.base
Make subodule a forward ref in Index.base3
UnMake subodule a forward ref in Index.base
UnMake subodule a forward ref in Index.base2
Type index _items_to_rela_paths()
Check change_levels (should fail)
Add 'U' to change_levels (should pass)
Make diff.DiffIndex generic List['Diff']
Increase mypy strictness (no_implicit_optional & warn_redundant_casts) and fix errors
Finish initial typing of index folder
Mak GitCmdObjectDB a froward ref
Mak EntryTup a froward ref
Mmmmm
Mmmmmm
Fix traverse_trees_recursive()
Fix traverse_trees_recursive() again
Use Tuple not tuple
fix base,ours,theirs typing
Change List to MutableSequence in fun.py _find_by_name()
tests TraversableIterableObj typeguard
improve TraversableIterableObj typeguard
Rmv typeguard from list_traverse(), was wrong
Change type of list_traverse() again.
Add list_traverse() to Tree and TraversableIterableObj.
Rmv circular import, create Has_id_attribute Protocol instead
Fix list_traverse() docstring for Autodoc
Make has_repo protocol runtime checkable and use in Diffable
Flatten list_traverse()
Move TraverseNT to global, cos mypy complained on testing
Rmv submodule.base Repo assert
Try to distinguation git.diff module from diff.Diff.diff and diff.Daffable.diff()
Improve Diffable method typing
Rmv Diffable assert, add Remoote.url property
Add remote.url type
Remove defsult_index decorator from diff() and do check within function. Breaks typechecking for some reason
Make IndexFile and Diffable .diff() types agree
Improve IndexFile_process_diff_args() to get checks to rerun
Fix typing of index.fun.write_tree_from_cache()
Use TreeCacheTup type alias throughout
Make TreeCacheTup forward ref
update types in types.py
Add types to refs/tag.py
Add types to refs/symbolic.py
Add types to refs/remote.py
Add types to refs/reference.py
Add types to refs/log.py
Add types to refs/head.py
Make types in refs compatible with previous
Make types in refs compatible with repo
Make traversable and serilizable into protocols
Make types in refs compatible with objects
rmv python 3.5 checks from tests
rmv redundant IOerror except
Add flake8-bugbear and -comprehensions to test-requirements.txt
Change remaining type comments to py3.6+ types
update types in types.py
Make pathlike a forward ref
Add a cast to git.cmd _version_info
Rmv old py2.7 __future__ imports
Replace all Typeguard with cast, revert update to typing-extensions==3.10.0
Use BaseIndexEntry named access in index/fun.py
Rmv with_metaclass shim, make section constraint generic wrt its configparser type
Readd with_metaclass shim
change ordereddict to typing.ordereddict
put ordereddict behind py version guard
import ordereddict from types
fix rdereddict, cannot subclass typing-extensiosn version
Rmv with_metclass() shim again
change ordereddict guard, add type: ignore
Fix SymbolicReference reference typing
Add another type ignore for Ordereddict
Rmv py 3.10 check for typing.Ordereddict - its deprecated then, but will still work until 3.12
Fix some SymbolicReference types
Fix forwardref
Fix more missing types in Symbolic.py
Fix more missing types in Symbolic.py
Fix more missing types in Symbolic.py.
Fix more missing types in Symbolic.py, cos GuthubActions pytest stuck
Rmv test file
Fix more missing types in Symbolic.py, cos GuthubActions pytest stuck
Try downgrading pip
its not pip...
try actions/runner-images#709 workaround
Add type to symbolicreference.name()
Add type to symbolicreference.iter_items()
Add type to symbolicreference.rename()
Add type to symbolicreference.__repr__()
Add type to symbolicreference._get_ref_info()
Add type to symbolicreference._get_packed_refs_path()
Add type to symbolicreference.dereference_recursive()
Add type to symbolicreference.dereference_recursive()
Add type to symbolicreference()
Add type to symbolicreference.get_()
Add type to symbolicreference.is_remote()
Add type to symbolicreference._create()
Add type to symbolicreference.delete()
Add type to symbolicreference.log_append()
Add type to symbolicreference.is_valid()
Add type to symbolicreference.set_reference()
Add type to symbolicreference.set_object()
cleanup
Add type to symbolicreference.reference()
Add type to symbolicreference.references()
Add type to repo.base._to_full_tag_path
Add type to refs.head.delete()
Add type to refs.log._read_from_file()
Add type to objects.base.new()
Add final types to config.py
Add set_dirty_and_flush_changes() types to config.py
Add final types to commit.py
Add final types to tree.py
Add final types to submodule.py
Add final types to cmd.py
Add final types to index/fun.py
Add final types to symbolic.py
Add final final types to symbolic.py
replace some TBDs wiht runtime types
replace more TBDs wiht runtime types
increase mypy strictness (warn unused ignored)
increase mypy strictness (warn unused ignored and warn unreachable)
rmv 3.6 from CI matrix
rmv 3.6 from setup.py
rmv 3.6 README
Add __future__.annotations to cmd.py
Add __future__.annotations to cmd.py2
Fix parse_date typing
Fix parse_date typing 2
Fix parse_date typing 3
Fix parse_date typing 4
Fix parse_date typing 5
Fix parse_date typing 6
Fix parse_date typing 7
Add __future__.annotations to repo/base.py
Minor type fixes
Test new union syntax (Pep604)
Test trailing comma in args (>py3.6?)
Test Dataclass in repo.base.blame()
Test Dataclass in repo.base.blame() 2
Test Dataclass in repo.base.blame() 3
Test TypedDict in repo.base.blame() 2
Test TypedDict in repo.base.blame() 1
Test Dataclass in repo.base.blame() 4
Test Dataclass in repo.base.blame() 5
Test Dataclass in repo.base.blame() 6
Choose TypedDict!
Improve type of repo.blame_incremental()
Improve type of repo.currently_rebasing_on()
Add Typing :: Typed to setup.py
add type check to conf_encoding (in thoery could be bool or int)
type fix
type fixo
add pypy to test matrix
update 3.10 to rc1 in test matrix
try fix for Protocol buy in 3.10
try fix for Protocol buy in 3.10 2
try fix for Protocol buy in 3.10 3
upgrade sphinx for 3.10 compat
Update changelog and version
Change CI to 3.10.0-beta.4, to get docs to pass
Update changes.rst
Change CI python 3.10 to rc1 again. Spinx broken either way
f100024 (1):
Add encoding to utf-8 for fetch_info_lines;
yobmod (1):
fix setup.py classifiers, improvefnmatchprocess handler types
…s Maxim Lobanov
Workflow "Build" has started failing with out of disk space errors.
Expand the existing efforts to free up space (removal of some .NET,
Android, and GHC files). Use advice from
actions/runner-images#709 (comment)
and remove also C++ Boost library files and some GitHub's own Python
tools.
Describe the bug
Jobs running in Ubuntu-16.04 images are starting to throw error related with low disk space
Area for Triage: Servers
Question, Bug, or Feature?: Bug
Virtual environments affected
Expected behavior
The disk should have enough free space for custom user software needed for building.
Actual behavior
du -h) is:Just before the error due to not enough disk space:
The concrete error is:
But it is not important, whatever command that needs disk space will fail
The text was updated successfully, but these errors were encountered: