Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Torcx running out of diskspace #32

Closed
moritz31 opened this issue Mar 1, 2020 · 8 comments
Closed

Torcx running out of diskspace #32

moritz31 opened this issue Mar 1, 2020 · 8 comments
Labels
channel/edge Issues concerning the EDGE channel of Flatcar Container Linux. kind/bug Something isn't working

Comments

@moritz31
Copy link

moritz31 commented Mar 1, 2020

on the flatcare edge version torcx is running out of diskspacen on the google cloud

error msg:

Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="common configuration parsed" base_dir=/var/lib/torcx/ conf_dir=/etc/torcx/ run_dir=/run/torcx/ store_paths="[/usr/share/torcx/store /usr/share/oem/torcx/store/2411.99.0 /usr/share/oem/torcx/store >
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="profile found" name=docker-1.12-no path=/usr/share/torcx/profiles/docker-1.12-no.json
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="profile found" name=docker-1.12-yes path=/usr/share/torcx/profiles/docker-1.12-yes.json
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="profile found" name=vendor path=/usr/share/torcx/profiles/vendor.json
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=info msg="no vendor profile selected by /etc/coreos/docker-1.12"
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="skipped missing lower profile" missing profile=oem
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=warning msg="no next profile: unable to read profile file: open /etc/torcx/next-profile: no such file or directory"
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="apply configuration parsed" lower profiles (vendor/oem)="[vendor]" upper profile (user)=
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="mounted tmpfs" target=/run/torcx/unpack
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="profile found" name=docker-1.12-no path=/usr/share/torcx/profiles/docker-1.12-no.json
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="profile found" name=docker-1.12-yes path=/usr/share/torcx/profiles/docker-1.12-yes.json
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="profile found" name=vendor path=/usr/share/torcx/profiles/vendor.json
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="new archive/reference added to cache" format=tgz name=docker path="/usr/share/torcx/store/docker:19.03.torcx.tgz" reference=19.03
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=debug msg="new archive/reference added to cache" format=tgz name=docker path="/usr/share/torcx/store/docker:com.coreos.cl.torcx.tgz" reference=com.coreos.cl
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=info msg="store skipped" err="open /usr/share/oem/torcx/store/2411.99.0: no such file or directory" path=/usr/share/oem/torcx/store/2411.99.0
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=info msg="store skipped" err="open /usr/share/oem/torcx/store: no such file or directory" path=/usr/share/oem/torcx/store
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=info msg="store skipped" err="open /var/lib/torcx/store/2411.99.0: no such file or directory" path=/var/lib/torcx/store/2411.99.0
Mar 01 21:29:00 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:00Z" level=info msg="store skipped" err="open /var/lib/torcx/store: no such file or directory" path=/var/lib/torcx/store
Mar 01 21:29:08 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:08Z" level=error msg="failed to unpack: unpacking "/usr/share/torcx/store/docker:com.coreos.cl.torcx.tgz": error extracting tar: write /bin/containerd: no space left on device" image=docker reference=com.coreos.cl
Mar 01 21:29:08 localhost /usr/lib64/systemd/system-generators/torcx-generator[600]: time="2020-03-01T21:29:08Z" level=error msg="apply failed: failed to install 1 images"

but on the root device is still enough space, only sum partitions seem to be limited

Filesystem Size Used Avail Use% Mounted on
devtmpfs 264M 0 264M 0% /dev
tmpfs 289M 0 289M 0% /dev/shm
tmpfs 289M 276K 289M 1% /run
tmpfs 289M 0 289M 0% /sys/fs/cgroup
/dev/sda9 27G 1.7G 24G 7% /
/dev/mapper/usr 985M 949M 0 100% /usr
none 289M 289M 0 100% /run/torcx/unpack
tmpfs 289M 0 289M 0% /media
tmpfs 289M 4.0K 289M 1% /tmp
/dev/sda6 108M 76M 24M 77% /usr/share/oem
/dev/sda1 127M 61M 66M 49% /boot
tmpfs 58M 0 58M 0% /run/user/0

@dongsupark
Copy link
Member

Thanks for the bug report.
Interesting. It would be great to know how to reproduce it.

@dongsupark dongsupark added kind/bug Something isn't working channel/edge Issues concerning the EDGE channel of Flatcar Container Linux. labels Mar 6, 2020
@moritz31
Copy link
Author

moritz31 commented Mar 7, 2020

So i just uploaded the image with the docker container to google cloud and started an f1.micro instance with 30gb of disk space. Just to make sure it is not an diskspace issue i tried it even with a 100gb instance but didn't make an difference

@anakaiti
Copy link

I'm having the same issue on AWS nano instances on version 2605.5.0. It runs normally on larger instances.

@alkar
Copy link

alkar commented Sep 24, 2020

I'm having the same issue on AWS nano instances on version 2605.5.0. It runs normally on larger instances.

We see the same issue in both AWS (t3a.nano) and GCP (f1-micro) running 2605.5.0
Neither reboot nor a re-deploy fixes this.

@george-angel
Copy link

The problem is default tmpfs size is 50% of your physical RAM:

The size parameter also accepts a suffix %
to limit this tmpfs instance to that percentage of your physical RAM:
the default, when neither size nor nr_blocks is specified, is size=50%

ref: https://www.kernel.org/doc/Documentation/filesystems/tmpfs.txt

And thats what torcx is using, default: https://github.com/flatcar-linux/torcx/blob/flatcar-master/internal/torcx/perform.go#L287

2512.3.0 - had unpacked size of ~200M - so it would run on any machine with 400M+ physical RAM.
2605.5.0 - has unpacked size of ~290M - and so now it will only run on machines with 580M+ RAM.

Perhaps flatcar/scripts#16 is relevant here to attribute the size increase, this part I'm not sure about.

@pothos
Copy link
Member

pothos commented Sep 24, 2020

Thanks for the detailed report. Yes, it looks like we need to specify the size of the tmpfs directly and not rely on the percentage.

@pothos
Copy link
Member

pothos commented Sep 24, 2020

I created a PR here flatcar/torcx#4

@pothos
Copy link
Member

pothos commented Sep 24, 2020

This will be addressed in the next Stable bugfix release (and any other releases).

@pothos pothos closed this as completed Oct 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
channel/edge Issues concerning the EDGE channel of Flatcar Container Linux. kind/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

6 participants