Roadmap of upcoming major features

[jrosdahl]

What's this?

This issue tracks major features that I (@jrosdahl) would like to see for ccache in the future. The tasks are roughly in a logical implementation order, but many can be worked on in parallel.

I will split the tasks to separate issues when time allows.

Roadmap

• 1. Introduce aggregated result files (Introduce aggregated result files #411)
• 2. Switch from MD4 to BLAKE2b (Switch from MD4 to BLAKE2b #412)
• 3. Switch from zlib to zstd ( Switch from zlib to zstd #413)
• 4. Design and implement a backend API and plugin framework (Design and implement a backend API and plugin framework #414) (See https://github.com/ccache/ccache/projects/1.)
• 5. Refactor the current file system backend code to use the backend AP. (Won't be done.)
• 6. Implement a memcached backend. (Probably won't be done.)
• 7. Improve cache cleanup mechnism (Improve cache cleanup mechanism #417) (May be done but not high priority.)

[afbjorklund]

I am obviously in favor of 1, 2 and 3, since I suggested them. There are other checksums and other compressions, but those two are the best ones for the purpose IMHO. Should also mention that just like gzip, lz4f also has a "HC" option (like --best) that allows for recompressing for smaller size. The decompression speed is the same, it is only the compression speed that differs between the "levels".

[afbjorklund]

The main purpose of the memcached backend is to be able to share cache over network between different computers. If you just want to put your cache in memory, you can use /dev/shm (ramdisk) already today. The choice of memcached for this is because it has been around for long and because it is fastest, but it does have some drawbacks like a 1M maxmum value size limit (worked around by splitting up data...)

[jrosdahl]

I am obviously in favor of 1, 2 and 3, since I suggested them.

Great!

Just for the record, I have been considering single-result files for a long time. Bullet 2 and 3 are indeed inspired by your work and ideas, thanks!

There are other checksums and other compressions, but those two are the best ones for the purpose IMHO

Yes, I have made a similar evaluation and came to the same conclusion (using LZ4 for a ccache-like tool at work).

The main purpose of the memcached backend is to be able to share cache over network between different computers.

Of course.

[afbjorklund]

Implementations:

1. Aggregated .result: Add aggregated file storing all the results #408
2. BLAKE2b checksum: Allow using BLAKE2b checksum instead of bundled MD4 #406
3. LZ4f compression: Add support for more compression types #118

[afbjorklund]

For stats, see #168 (and I'm sure the whole "stats" system could use a similar overhaul to the .result)
@jrosdahl mentions that he has some secret special plans for improving cleanup, so maybe stats too ?

One long outstanding feature is to have stats per-build in addition to per-cache, mentioned in #262
So having multiple stats would be a good idea to plan, if designing a "stats" file replacement / addition.

[jonesmz]

Another candidate for distribute object caching, in addition to memcached, would be the Ceph rados object storage api

[afbjorklund]

Another candidate for distribute object caching, in addition to memcached, would be the Ceph radios object storage api

The workaround for now would be to use CephFS, but it could be interesting to compare that with using librados instead

[jonesmz]

I expect that the largest difference would come from avoiding the FUSE filesystem layer, with the second largest savings coming from skipping the metadata server.

[afbjorklund]

I expect that the largest difference would come from avoiding the FUSE filesystem layer, with the second largest savings coming from skipping the metadata server.

Please open a separate feature request about this, so that the discussion can be moved there...

[iveqy]

If I understand it correctly memcached is limited by RAM size while redis support cluster of redis instances. Even if memcached is faster/more mature it seems as it is easier to implement redis from a ccache perspective for large workloads. Or do I miss something?

[afbjorklund]

If I understand it correctly memcached is limited by RAM size while redis support cluster of redis instances. Even if memcached is faster/more mature it seems as it is easier to implement redis from a ccache perspective for large workloads. Or do I miss something?

Please move discussion to a separate issue, thanks. (it was easier, but the libmemcached implementation had optimizations)

[jrosdahl]

I'm closing this tracking issue now since it's obsolete. See also discussion in #414 and the Storage backends project.