This repository has been archived by the owner on Apr 25, 2019. It is now read-only.
Incremental rebuilds #17
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This changeset captures a ton of work related to getting Katamari set up with a new build product cache structure and a sane (at least for the default extensions) implementation of `katamari.roll.extensions/rule-id` which computes the ID for a given rule based on the inputs (and their IDs), and the sorted set of content hashes of all the files used by the target. This allows Katamari to do at worst coarse grained change detection based on changes in content hashes. The change detection machinery could be improved to leverage the same fundamental first date check then content check logic which the Rollfile parser uses. The `rule-id` machinery is used to implement lazy cache hit based rebuilding - products are only built if they can't be fetched from the cache. Unfortunately because the check and hit algorithm uses full build products and is baked into the implementation of `roll`, there isn't (currently) a way to determine what target(s) will be rebuilt without just doing the build. Also included in this patch - a new TTL based cache cleaning task. This also allows for cache busting - `cache-clean 0` to wipe it all away. There is currently no automatic cache cleaning say after builds.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This changeset implements the machinery required to achieve incremental content hash based rebuilding atop dependency order builds.
The fundamental strategy here is to use the
rule-idextension point to compute a content address for every single rule to be built. If that content address exists in a trivial filesystem cache, then the previously built product in the cache is deserialized and used.The
rule-buildcontract has been extended to define rules to occur within a chroot addressed byme.raynes.fs/*cwd*. This allows therollmachinery to preserve and potentially restore previously rolled build products.The product records have been extended with
:id, being the content address of the product.The task
clean-cache [ttl-age-ms]has been added, allowing users to clean cached build products beyond some age or to bust the cache entirely usingclean-cache 0to remove all cached products.Fixes #5