Reorganize caching to be an interface to allow for different implementations.

[coollog]

This will help:

• Allow for layer up-to-date checking to be different strategies
• Have all layers be independent so no cache data is lost during concurrent executions

The cache implementation will consist of a storage engine and a query system.

Tasks:

• Add design proposal for new cache mechanism. Proposal RFC: New cache mechanism #878
• Create new cache interfaces under a new package ncache that will eventually replace the existing cache package. Create new cache interfaces under a new package ncache. #869
• Add default implementations of the new cache interfaces. Adds default implementations for CacheWriteEntry and CacheReadEntry. #874, Fixes CacheStorage interface and renames CacheReadEntry/CacheWriteEntry to CacheEntry/CacheWrite. #896, Adds DefaultCacheStorage#write. #879, jib-maven-plugin Typo #989, Adds DefaultCacheStorage#retrieve. #901
  • DefaultCacheWriteEntry Adds default implementations for CacheWriteEntry and CacheReadEntry. #874, Fixes CacheStorage interface and renames CacheReadEntry/CacheWriteEntry to CacheEntry/CacheWrite. #896
  • DefaultCacheReadEntry Adds default implementations for CacheWriteEntry and CacheReadEntry. #874, Fixes CacheStorage interface and renames CacheReadEntry/CacheWriteEntry to CacheEntry/CacheWrite. #896
  • DefaultCacheStorage Adds DefaultCacheStorage#write. #879, Adds DefaultCacheStorage#listDigests. #898, Adds DefaultCacheStorage#retrieve. #901
• Add a new ncache.Cache that provides the higher-level operations used by the builder steps. Adds Cache with initial method for fetching a cache entry by layer entries. #975, Finishes implementing new Cache. #1022, Has Cache use last modified time as metadata. #1031, Stores temporary cache files under cache directory. #1037, Changes CacheWrite into UncompressedCacheWrite and adds ability to write compressed layer blob. #1046
• Have the builder steps use the new ncache.Cache and compare performance with old cache.Cache (performance is basically the same after checking with 100,000+ files)
• Change the builder steps to use the new ncache.Cache. Changes the builder steps to use the new ncache.Cache. #1047
• Remove old cache package Changes the builder steps to use the new ncache.Cache. #1047
• Move ncache to cache. Renames ncache to cache. #1056
• Consider adding a cache pruner that deletes all cache entries without a selector.
• Make sure all issues in Multiple jib tasks cannot run in parallel due to cache directory conflict #922 are fixed.
• Explore detections of potential cache corruptions - see Explore detections of potential cache corruptions. #980