Work in progress, moving to dbuild 0.10 #221
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 19 commits
October 26, 2018 11:17
The BuildGraph calculation was performed twice, once after extraction, and one in case of a repeated build when deserializing the previously serialized RepeatableDBuildConfig, to make sure it matches. That was not supposed to be the case: the "graph" value inside the RepeatableDBuildConfig is actually a lazy val. However, an inappropriately placed "checkCycles()" would always trigger the graph creation. This commit moves checkCycles() to the end of the BuildGraph creation, so that the graph is not immediately built upon deserialization. Also, this commit makes "topological" in graph a lazy val, for a further improvement. Note that "checkCycles()" already relied on "isCyclic", which is also a lazy val.
We will have resolve() determine the stable URI (for instance, converting a branch/tag into a commit hash), while prepare() will be in charge of preparing the source tree. The existing implementation of ProjectResolver currently do all the work in resolve(), while prepare() is empty; however, we will shortly convert the git resolver so that reference resolution and source tree setup are distinct. The reason is that in many cases an extracted dependency set will already be in the cache: we want to be able to look into the cache, but there is no point in preparing a source tree if we are not going to rebuild/reextract.
This commit reworks completely the Git resolver, in order to implement shallow reference fetching. Other operations are also optimized, and the --skipGitUpdates flag fully implemented. The supporting code for jgit, which was always inefficient with respect to the main git support implementation, has now been removed in favor of regular git.
The flag is still accepted, but setting it to true, or using the "jgit://" uri scheme, will display an error message.
Normally dbuild will automatically reclaim the space used by working dirs where projects are built or extracted, after a configurable age. The reclaiming takes place at the start of each dbuild run. If the configurable age is set to zero, all of the old data will be deleted at the beginning of the run; the data corresponding to the current run will however remain available at the end. This commit allows the age to be set to negative values. If that is the case, the temporary working data will be deleted immediately after each extraction or building of each project, saving disk space. It is still possible, for instance, to specify that we want the temporary data of the projects that succeeded to be discarded immediately, while the data of the failed projects can be kept in order to diagnose the failure.
Rather than multiple clones, there is now only one central clone per project. The directories for extraction and building now receive their files directly from the central clone, by moving around the working tree. Also, the resolved shas are now taken directly from the main clone, and if an extraction or build are cached, no files are checked out at all.
Removed the short-lived "shallow-allowed" flag. Deleted all references to jgit Added project flag "git-full-clone" TODO: adjust docs
This was referenced Nov 18, 2018
added 5 commits
November 18, 2018 15:30
If the ref that we are resolving looks like a commit hash, then look if it is already present in the cached clone before performing a full fetch, which would unshallow the central clone. That also prevents the build stage (in which all refs are hashes) from unshallowing all of the clones as it progresses. A possible further improvement would be heuristically deciding that refs that look like hashes and are exactly 40 characters long most probably are indeed hashes, therefore skip the lookup for a matching branch or tag.
We heuristically decide that refs that look like hashes and are exactly 40 characters long most probably are indeed hashes, therefore skip the lookup for a matching branch or tag. We could possibly check, in case that didn't work, if it is a bizarre 40-character branch name or tag name that looks exactly like a hash, but that is improbable enough; at present we just abort.
|
I tried it in the Scala 2.11 community build, but a number of projects failed to extract; details at scala/community-build#1063 (comment) |
|
also scala/community-build#970 (comment) mentions some perhaps spurious new warnings I'm seeing |
|
unlikely to progress :-/ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.