GitlabCI fails constantly once cache is made. Any help welcome

[tlaurion]

You can see CI errors when looking at commit logs on master.

One example of such failing build.

Output:

Skipping Git submodules setup
Restoring cache
Checking cache for heads-tlaurion-3...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/12675230/heads-tlaurion-3 
WARNING: packages/linux-4.14.62.tar.xz: write packages/linux-4.14.62.tar.xz: no space left on device (suppressing repeats) 
WARNING: build/musl-cross-38e52db8358c043ae82b346a2e6e66bc86a53bc1/build/local/i386-linux-musl/obj_sysroot/lib/ld-musl-i386.so.1: chmod build/musl-cross-38e52db8358c043ae82b346a2e6e66bc86a53bc1/build/local/i386-linux-musl/obj_sysroot/lib/ld-musl-i386.so.1: no such file or directory (suppressing repeats) 
Successfully extracted cache
Executing "step_script" stage of the job script
ERROR: Job failed (system failure): Error response from daemon: mkdir /var/lib/docker/overlay2/1f937de3cf9c657afd27cfc0ac932ec69a35817c81659c54e63ef9080f74a2ed-init/diff: no space left on device (docker.go:705:0s)

[tlaurion]

Additional steps to make the local caches specialized would be desired, just like attempted for CircleCI at #769

[tlaurion]

@szszszsz : were you able to go around that problem on your side? Do you know anybody that has more experience on that level? How much would it cost to fix this?

[tlaurion]

Really not understand why CircleCI cache is 5.1GiB:

Found a cache from build 487 at heads-modules-and-patches-k0hlzenNW+zlyb4e06TMH0oTBx3v6NyBX8OMU_WnhNQ=<no value>
Size: 5.1 GiB
Cached paths:
  * /root/project/packages
  * /root/project/crossgcc
  * /root/project/build

And why GitlabCI is 14GiB (for less boards..., same paths...):

$ echo "Calculate used space for cache"
Calculate used space for cache
$ du -shc packages crossgcc build
205M	packages
1.9G	crossgcc
12G	build
14G	total
Saving cache
15:44
Creating cache heads-tlaurion-2-32...
packages: found 75 matching files and directories  
crossgcc: found 5818 matching files and directories 
build: found 296003 matching files and directories 

Of course, deflating 14Gb on top of a docker image might break docker reserved space. The question is why such big discrepency between the two CI environements, where CircleCI cache ressembles the size of local builds, where GitlabCI cache size..... doesn't make any sense, like if files were archived multiple times.

Symlinks not followed and archived multiple times? Other hypothesis?

[tlaurion]

Note: GitlabCI now implements a caching system based on file hashes. So CircleCI equivalent can be made.

I reply to

And why GitlabCI is 14GiB (for less boards..., same paths...):

$ echo "Calculate used space for cache"
Calculate used space for cache
$ du -shc packages crossgcc build
205M packages
1.9G crossgcc
12G build
14G total
Saving cache
15:44
Creating cache heads-tlaurion-2-32...
packages: found 75 matching files and directories
crossgcc: found 5818 matching files and directories
build: found 296003 matching files and directories

The number of files matches what is observed locally. The problem here seems to be that the cache itself is not really compressed. It was not a problem before, but it seems that GitLabCI changed their internals, where cache.zip doesn't seem to imply strong compression/CircleCI implements it, which explains the difference of size of the cache, but not why it is incapable of decompressing without causing disk space issue.

Reasoning: If the cahce built from local build created files were able to be expending by compilation, there should not be a problem expending that cache locally on top of a docker image. So probably a problem of paths and misunderstanding on my side of the paths implied. Will do WIP based on that with trial and errors

[tlaurion]

@szszszsz @jan23 @MrChromebox, @daringer @alex-nitrokey : any help welcome, as always.

[tlaurion]

Update:
Docker Locally, stored under Docker volumes: /var/lib/docker/volumes/<volume-id>/_data/<user>/<project>/<cache-key>/cache.zip.

Is where the cache is extracted. Will try to figure out how the tasks can be reordered in BitlabCI conf, since:

1. We need to have git branch downloaded and forced resetted.
2. We then need to hash the modules and patches (as of CircleCI) in a temp file from the git downloaded path
3. We then need to take those files (hash of musl-cross-make module and patches, and all modules and all patches hashes) and download caches the same way, to be able to accelerate builds based on musl-cross-make cache if modules+patches cache is not found (new)
4. And only then resume normal building process

[tlaurion]

Since its two steps, I have now problems downloading cache of musl-cross when there is no matches for modules+patches.
The actual config checks for each steps for its cache, and doesnt use musl-cross cache to build modules.

[tlaurion]

Swtiched to CircleCI years ago