Concepts of g3 cache
Version: G3v1, G3v2
Updated: 2022-11-05
The custom g3-cache has been introduced in the second version (G3v2) of the project.
The local g3-cache is an additional cache used by the building pipeline and it should not be confused with the Docker builder cache maintained by the Docker Build itself.
The g3-cache stores the selected pre-downloaded packages used by the Dockerfiles, that would be otherwise repeatedly downloaded from the external sources by each build.
It results in a significantly higher performance by building sets of images or by repeated builds.
The are two parts of the g3-cache:
- local
g3-cache - shared
g3-cache
They are both stored in different places and serve different purposes.
You can learn more about the implementation on the Wiki page "How g3-cache works".
The local g3-cache is actually used by the Dockerfiles and the Docker builder.
Therefore it must be always placed inside the Docker build context, because the Docker builder has no access to the files outside the context.
In practice in means, that the local g3-cache must be a subfolder of the Docker build context root folder (e.g. docker/), containing all the resources for the particular build.
However, there could be other projects, that also use some of the packages contained in the local g3-cache. Because of the builder context limitation they must themselves download the same packages to their own local g3-cache, placed inside their own Docker build context.
To avoid repeated downloads of the same packages, there is a concept of the shared g3-cache.
The shared g3-cache is not used by the Dockerfiles nor the Docker builder.
The cache is always placed outside the Docker build context and it is used for faster initialization of the local g3-caches of multiple projects.
It stores all the packages of all the participating projects, that can copy the required packages to their own local g3-caches instead of downloading them from the external source.
The shared g3-cache is especially useful on the building stages that natively support CI/CD, e.g. the GitLab, where the Docker build context is destroyed after completing each CI/CD job.
The g3-caches can contain several versions of the same conceptual package and each conceptual package can consist of several files.
To keep the packages organized, the resources of each conceptual package are stored in its own cache section.
A cache section is essentially a cache subfolder, named after the conceptual name of the package, e.g. tigervnc or chromium.
Refreshing the g3-cache should be part of the building pipeline.
Each time a particular package has been downloaded from the external sources, both g3-caches should be updated.
Obsolete packages should be periodically removed from the caches.
However, there should be different rules and schedules for the local and shared g3-caches.
On some stages, e.g. the GitLab, the local g3-cache will be automatically destroyed after each CI/CD job is completed.
By the shared g3-cache it should be considered, that it contains the packages used by multiple projects.
The Dockerfiles should preferably use the packages from the local g3-cache.
However, they should fall back to the ad-hoc downloading, if the local g3-cache or the particular package are not available.