CCACHE_TEMPDIR="/tmp/ccache-tmp" if /tmp is mounted as tmpfs

[The-going]

Description

When the system temporary folder is mounted as tmpfs, it makes sense to mount
the temporary compiler cache folder in /tmp. This speeds up the compilation process
and saves the hard disk resource.
About the SSD resource, I think that it is clear to everyone.

How Has This Been Tested?

For my virtual machine 6 spu frequency 3.6G
Compile Time before: Runtime 46:43 min
Compile time after: Runtime 35:42 min

Checklist:

• Test build

[rpardini]

This might make sense, I've explored similar in armbian-next (but took other paths after consideration).

Other, possibly lunatic, ideas:

• the compression used by ccache reduces disk usage, but takes CPU, even if little, and CPU is what is needed during a build -- so might be worth investigating disabling it
• any changes to versions in the kernel/u-boot Makefile, LOCALVERSION, or anything related causes version.h to change, which then causes everything else to change. ccache does not help at all if that happens, since the new version causes a cache miss for every single file. in that scenario, ccache is slowing us down.
• scenarios ccache helps with are:
  • new build after a make clean or git clean -xfd, but with the same versions as before..
  • when only changing the contents of patches, but keeping the same version and rebuilding
  • (compensates a bit for) fact master does/did bizarre make clean in the tree (during packaging for cross build)
  • (compensates a bit for) fact master does not keep proper modification time of patched files / drivers / .config etc
  • when building same version across families of the same architecture (armhf/arm64) - but is counterproductive if building across arches

So it's a tradeoff with ccache already, trying to optimize it by tmpfs I don't think makes much difference.
I wouldn't do it by default since I think having /tmp in tmpfs is done by systemd tmp.mount, (so everywhere) and then any reboot causes lost cache, which might be surprising.
Maybe just use PRIVATE_CCACHE for pointing to your tmpfs instead.

EDIT: I had missed it's about CCACHE_TEMPDIR not CCACHE_DIR

[rpardini]

trying to optimize it by tmpfs I don't think makes much difference.

Scratch that. I didn't pay attention (or you edited the original post?) to the below.

Compile Time before: Runtime 46:43 min
Compile time after: Runtime 35:42 min

I will shut up and go test your idea.

[rpardini]

For my virtual machine 6 spu frequency 3.6G

Please also tell us the storage configuration?
Also what kind of virtualization technology.

[iav]

What do you think about to add into config-docker.conf something like DOCKER_FLAGS+=(-v=$(mktemp -d)/:/tmp/)?

[iav]

I wouldn't do it by default since I think having /tmp in tmpfs is done by systemd tmp.mount, (so everywhere) and then any reboot causes lost cache, which might be surprising.

It's not ccache data itself.

[The-going]

@rpardini Do you understand well how the compiler cache works?
Explanations:

CCACHE_DIR - The folder in which the result of the compilation process is recorded with the process log for each file.

CCACHE_TEMPDIR - Temporary files are constantly being written and destroyed in this folder in the process of work. when the result is achieved, it is transferred to the CCACHE_DIR folder. This is a very large read write traffic. And a faster option when it happens in RAM.

My desktop machine has an i7 CPU of 8 threads. The RAM is 32 Gigs.
Virtual machine QEMU 6 cpu 8 gigs.

[The-going]

What do you think about to add into config-docker.conf

Igor, I don't use this technology and I can't say anything.

[rpardini]

It's not ccache data itself.

CCACHE_TEMPDIR

Oh damn, I missed the TEMP part of it. Thanks for pointing it out.
That should definitely be in tmpfs!

but CCACHE_TEMPDIR defaults to something like /run/user/0 which should already be tmpfs.
(We might be missing that because of sudo and/or lack of XDG_RUNTIME_DIR)

[The-going]

We might be missing that because of sudo

That's right.
verify:

ccache -p
sudo su
ccache -p

[rpardini]

What do you think about to add into config-docker.conf something like DOCKER_FLAGS+=(-v=$(mktemp -d)/:/tmp/)?

Or, using Docker Volumes, --mount type=tmpfs,destination=/tmp which works across hosts/VMs too.

[iav]

@The-going 177 min for full image with your proposal, about 230 min without.
ODroid-N2, emmc disk.
Nice!

[hzyitc]

Maybe point to .tmp/ccache and mount it as tmpfs in the script? So it can work with all environments.

[iav]

Maybe point to .tmp/ccache and mount it as tmpfs in the script? So it can work with all environments.

then ccache build database will be lost on reboot

[hzyitc]

then ccache build database will be lost on reboot

I mean CCACHE_TEMPDIR not CCACHE_DIR. Just make this PR work everywhere. Mounting /tmp as tmpfs isn't a default behavior on Ubuntu or Debian.

[hzyitc]

What's more, I check the manual of ccache.

temporary_dir (CCACHE_TEMPDIR)
  This option specifies where ccache will put temporary files. The default is /run/user/<UID>/ccache-tmp if /run/user/<UID> exists, otherwise <cache_dir>/tmp.

And /run/user/<UID> is mounted as tmpfs by default.

[rpardini]

What's more, I check the manual of ccache.

That's outdated manual, new version states XDG_RUNTIME_DIR -- which amounts to the same thing.

And /run/user/<UID> is mounted as tmpfs by default.

On sane systems, yes. Using sane sudo enchantment, yes. Otherwise no -- which is the case with master.

Mounting /tmp as tmpfs isn't a default behavior on Ubuntu or Debian.

That's why he is checking to see it if /tmp is a tmpfs before doing it.

[hzyitc]

That's why he is checking to see it if /tmp is a tmpfs before doing it.

I mean let's do this mount in our script. So it can work for everyone since most users or developers didn't known mounting /tmp can speed up the completion.

[rpardini]

I mean let's do this mount in our script. So it can work for everyone since most users or developers didn't known mounting /tmp can speed up the completion.

True. I've been doing this in different spots in armbian-next -- where the trap manager helps a lot.

tmpfs helps a lot when building on really slow I/O subsystems like mmc or spinning rust -- and not only for ccache.

Either way thanks @The-going for sparking this initiative