Exploring squashfs

[mattgodbolt]

Based on various conversations about NFS performance, filing this as a place to put info:

Idea is:

• build image(s) per compiler. Either direct ext4 or squashfs
• place these on network attached storage (/opt/images/...)
• mount these using loopback or squashfs on each node (maybe at /img/*/)
• unionfs magic to make this "override" /opt/compiler-explorer/* for matches

Theory:

• 99.99% of our files are read only and never changed. NFS doesn't know this
• Most of our network performance is "has this changed?" or re-fetching data and metadata that hasn't changed
• By using loopback IO etc we "launder" the file check and do caching of (uncompressed) blocks read from NFS without always going back to see "is this still valid?"

Or something.

[mattgodbolt]

Test setup:

• gcc 10.1.0 packed with mksquashfs /opt/compiler-explorer/gcc-10.1.0 /opt/squash-images/gcc-10.1.0.img -all-root -progress -comp zstd on admin node
• idle staging node, mounted /opt/... on /mnt

Metadata tests:

ubuntu@ip-172-30-4-41:~$ ls -lR /opt/compiler-explorer/gcc-10.1.0/ | wc -l
9721
ubuntu@ip-172-30-4-41:~$ ls -lR /mnt | wc -l
9721

$ time ls -lR /mnt > /dev/null

real    0m0.041s
user    0m0.020s
sys     0m0.020s
$ time ls -lR /opt/compiler-explorer/gcc-10.1.0/ > /dev/null

real    0m11.557s
user    0m0.019s
sys     0m0.177s

Data tests, from mostly cold (some metadata had been looked at before, but no file contents).

$ time find /opt/compiler-explorer/gcc-10.1.0/ -type f | xargs md5sum > /dev/null

real    2m7.206s
user    0m1.440s
sys     0m0.692s

ubuntu@ip-172-30-4-41:~$ time find /mnt/ -type f | xargs md5sum > /dev/null

real    0m6.092s
user    0m1.290s
sys     0m2.205s

output later confirmed to be "the same". Because it seems so unlikely that it could be that much faster...

[mattgodbolt]

After a # sync; echo 3 > /proc/sys/vm/drop_caches:

root@ip-172-30-4-41:/home/ubuntu# sync;
root@ip-172-30-4-41:/home/ubuntu#  echo 3 > /proc/sys/vm/drop_caches 
root@ip-172-30-4-41:/home/ubuntu# exit
ubuntu@ip-172-30-4-41:~$ time find /mnt/ -type f | xargs md5sum > /tmp/mnt

real    0m6.107s
user    0m1.336s
sys     0m2.299s

[mattgodbolt]

Mounted with options:

/efs/squash-images/gcc-10.1.0.img on /mnt type squashfs (ro,nodev,relatime)

[mattgodbolt]

Looks like we can "just" mount over the top:

ubuntu@ip-172-30-4-41:~$ sudo mount -t squashfs /efs/squash-images/gcc-10.1.0.img /opt/compiler-explorer/gcc-10.1.0 -o ro,nodev,relatime
ubuntu@ip-172-30-4-41:~$ time find /opt/compiler-explorer/gcc-10.1.0/ -type f | xargs md5sum > /tmp/mounted-over-nfs

real    0m3.680s
user    0m1.374s
sys     0m2.209s

[mattgodbolt]

https://github.com/AgentD/squashfs-tools-ng/ worth a look to directly transform our tarballs into squashies

[mattgodbolt]

With just NFS on staging instance, no other traffic:

• 1800ms (first), 523, 626, 670,511,572

Mounting squashfs:

• 409ms (first), 298ms, 311ms,381ms,355ms,518,294

[mattgodbolt]

Ok! I think we're done here! There's no special casing for boost any more, and libraries and compiler will "auto squash". Instances will use NFS immediately, but upon restart will notice and mount squashed images.

I deliberately don't squash "trunk" builds, but we can consider doing something similar for them with more interactive remounting on the instances (or a forced "update" process to get them to mount new directories and unmount old)

There's many other ways to skin this too; including using autofs instead and some clever overlayfs stuff. but for now, I think we're good!