[MNG-7389] Incremental .m2 cache cleanup for CI

[jira-importer]

Thomas Skjølberg opened MNG-7389 and commented

One or more popular continous integration are unable to properly manage the .m2 repository cache, resulting in wasted resources in the form of increased CI runtime and bandwidth consumption.

CircleCI cache behaviour:

• immutable cache entries
• default behaviour is to wipe the cache each time a pom file is modified (i.e. using pom hash as a cache key)
• cache entries TTL > weeks

So CircleCI always has a cache containing only the necessary artifacts, but has to download all dependencies every time the pom file changes.

Github Actions cache behaviour

• (effectively) mutable cache entries
• incremental cache (if it gets too big, it is wiped).
• cache entries TTL 1 week

So Github actions work well if the cache entries expire from time to time, otherwise the cache keeps growing.

Summary
Perhaps this does not look so bad at first glance, but for a project under active development, with a lot of artifacts, the pom file changes often. For example we have apps with 100 dependencies and automatic dependency bumping via Renovate, in addition to an hierarchy of libraries.

Key takeaways; time is wasted

• saving caches in CI
• loading cache in CI
• loading artifacts from external artifact store

This happens quite a lot. From the artifact store perspective, this probably multiplies the load by a factor of 10.

Possible solution: A way to define a "transaction" for artifact use, i.e.

1. run command to mark start of transaction
2. run one or more maven commands
3. run command to mark end of transaction, deleting artifacts not in use.

For reference, Gradle has the same problem.

Proof of concept:

• CircleCI : https://github.com/entur/maven-orb
• Github actions: https://github.com/skjolber/tidy-cache-github-action

The implementation uses instrumentation to record artifact access, then delete the artifacts not recorded. 

Alternatives:
I did try the last-accessed file timestamp first, turns out most CI filesystems are mounted without that option. However it should also be possible to update the modified timestamp and/or add read access to some existing metadata file. 

---

No further details from MNG-7389

[jira-importer]

Michael Osipov commented

I don't understand why you are explicitly requesting from us.

[jira-importer]

Thomas Skjølberg commented

Michael Osipov please elaborate. Is the dependencies component not the right home for managing the cache?

[jira-importer]

Michael Osipov commented

I still don't understand why we should clean up something for CI if it should know better? Maven only knows about MavenSession.

[jira-importer]

Thomas Skjølberg commented

The CI knows when to clean the cache, but has no good way to actually perform the operation it desires. What is lacking is some goal like dependency:purge-local-repository, but which deletes only unused dependencies. And "unused dependencies" as in the dependencies used by more than one Maven invocation. Surely a solution must involve the file system in some way.

A very simple approach would be to update the last-modified timestamp when an artifact is accessed during a build. Then a script or plugin goal could delete all artifacts with an old last-modified timestamp.

The PoC actually prints artifact access to a text file, and then scans through the cache, deleting files not in the text file. I am guessing that there is some better way of doing this.

[jira-importer]

Michael Osipov commented

Using atime is stupid because using atime has a very severe performance penalty, that is why most use atime=off.

the easiest way I see is to extend Maven Resolver by intercepting artifact access after its physical resolution on disk, write a properties file, e.g., {gav}.cachestatus which will contain all information you need. Walk the repo cache tree, read those file, delete state artifacts. Done. Consider that since the repo cache might be subject to access during your cleanup you will require multiprocess (MP) based locking which makes it even harder. Thus, you will need to write Java code which uses Resolver's SyncContextFactory.

Don't expect anyone of us to provide this anytime soon especially because diskspace is cheap.

Tamas Cservenak

[jira-importer]

Thomas Skjølberg commented

Thanks for the input, your suggested solution seems straigthforward. 

Just to be clear, while disc space is cheap, the limiting factor is typically the network capacity and to some extent CPU (compress / decompress). Usually CI docker images are downloaded and executed, then Maven cache is restored on top of them as part of CI workflow logic. So the opportunity here is to reduce cost of CI, reduce wait time for developers and speed up deploy pipelines. 

I do not have any expectations of implementation timeline or participants, I am just attempting to point out a relativly low-hanging fruit which will save time and money every day, albeit more for some systems than others. Using our own (CircleCI) systems as an example, building an app takes perhaps 10 minutes. Something like 1 in 5 commits update the pom file, resulting in the build taking about 1 minute longer (to transfer artifacts). So that is (very approximately) a 2% improvement; as an optimization this is not great, but fair enough.

[jira-importer]

Michael Osipov commented

Ideally, you would solve this with a Nexus repo manager physically close your your CI system. This will massively speed up your requests will reduce outbound network traffic.
I fail to understand how an eviction policy will reduce network traffic? Yes, you have transfer less, you mount that repo volume with the cache, but you also increase the probability of a cache miss which will result in additional network traffic.

Therefore, I recommend to properly describe the problem you have and not what you ideally see as a solution. Maybe a completely different solution will solve a problem better. I know from a user who's using a shared repo cache on CI nodes with Maven Resolver + Redis-based locks (developed by me) with massive speedups.

[jira-importer]

Thomas Skjølberg commented

Another way to explain this is a multi-level cache. I agree that ideally all artifact stores are located in the same datacenter, but for many, they are not, for various practical reasons. We buy CI and artifact store as a service (artifactory) in the cloud, something which is still cost-effective even if they are not offered in the same datacenter. Some shops do not even have an artifact store at all, and rely on the global artifact store. Our CI is priced per minute, the artifact store per GB. CI cache storage capacity is 'free' (cost is only the load/save time).

So for a practial example, for an incremental cache, if we have the levels

 1. CI internal cache
   * 50 MB/s read
   * 30 MB/s write
 2. Nexus style repo manager (i.e. for us Artifactory)
   * 5 MB/s read
 3. Global artifact store (i.e. Maven central)
   * 1 MB/s read
   
So lets do a new build. First git commit results in:

 1. cache key is generated
 2. CI cache cannot be loaded (it does not exist)
 3. build populates .m2 repository directory during build
 4. .m2 repository directory is saved to the CI cache

The second commit does not modify the pom:

 1. cache key is generated
 2. CI cache is loaded
 3. build (all artifacts are present)
 4. .m2 respository directory is not saved (it did not have any changes)
 
The third commit modifies the pom, bumping an artifact version from 2.0.0 to 2.0.1

 1. cache key is generated
 2. CI cache is loaded
 3. the build populates the .m2 respository with artifact version 2.0.1 - adding a minor delay
 4. .m2 respository is saved (containing both {}2.0.0 and 2.0.1{})

The forth commit does not modify the pom:

 1. cache key is generated
 2. CI cache is loaded, containing both 2.0.0 and 2.0.1
 3. build (all artifacts are present)
 4. .m2 respository directory is not saved (it did not have any changes)

The fifth commit modifies the pom, bumping an artifact version from 2.0.1 to 2.0.2

 1. cache key is generated
 2. CI cache is loaded, containing both 2.0.0 and 2.0.1
 3. the build populates the .m2 respository with the bumped artifact version - again adding a minor delay
 4. .m2 respository is saved (containing both {}2.0.0, 2.0.1 and 2.0.2{})

So now the CI cache contains more data than necessary. Loading and saving the cache takes longer than before. Developers see build times increase, possibly making them question their build and/or the health of the cache. In the following weeks, build times steadily increases at the hands of tools like Renovate, bumping at least a handful artifacts each day, eventually forcing an manual interaction (cache reset). 

In a nutshell, the eviction policy makes it easier to stay at cache level 1 - keeping build time low and stable ('within acceptable constraints'). This also mostly holds even if there is no Nexus style repo mananger. 

[jira-importer]

Michael Osipov commented

Yes, you need a multilayer cache which changes with the version of your project. Per-artifact notion of layer version and a version management for the project to purge the cache. Cloud is not a panacea. Sometimes a ProLiant for 5000 € is cheaper than fighting for CI minutes. I am now almost 100% certain that we will not deliver something like this because it is too specialized for the broad audience.

[jira-importer]

Thomas Skjølberg commented

The default Maven cache handling for two cloud CI providers (have not looked at others) is suboptimal, so this kind of improvement captures a broad audience. I've yet to see any project using the cache optimally, which is natural, because it is in fact quite hard with the current tooling.

And I am all for investing in sufficient processing power, but one could argue that it is rather the case that introducing more level 1 cache equivalents (i.e. in the form of a colocated artifact store) is for the specialized segment with the know-how and volume to justify the investment (servers cost more than their yearly lease). Also, better cache management makes doing so less important or even redundant from a performance standpoint.

Look, this is a real opportunity to reduce Maven "cost of ownership" and improve perceived stability, even get a leg up on Gradle, which also lacks this feature. The rise of free automatic dependency updates (via Renovate, Snyk, etc) has been heating up CI pipelines and artifact stores like there is no tomorrow; optimizing the cache use contributes towards keeping the cost of having up-to-date and safe apps down.