submodule add when adding subdatasets
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
This works now. The only test failure is:
it is unclear to me how it can get slower (I have not yet observed a scenario where things got slower), but there are some optimization that can still be done.
This was referenced
Aug 31, 2020
Instead go manual, avoiding various checks, because we already know inside `save` that a particular directory is considered untracked by Git. This approach is substantially faster. Already with 50 subdatasets the speed-up is 2x. Quick'n'dirty benchmark: ``` % rm -rf * .git* .datalad; datalad create . && \ ( for i in $(seq 50); do datalad create $i; done ) && \ time datalad save && datalad status Current master ``` datalad save 3.70s user 1.50s system 103% cpu 4.998 total ``` This PR ``` datalad save 1.68s user 1.02s system 108% cpu 2.499 total ``` It may well be that it needs more safety checks that would eat up that margin. But in the context of datalad#2880 (comment) (save thousands of new subdatasets) even much smaller benefits would still translate to substantial absolute speed-ups.
When saving many new subdatasets this is substantially faster than incremental changes via normal manipulators. In particular the straight unchecked addition of text to the config files, rather than a variable-wise manuipulation makes a big difference in the case of a large number of registered subdatasets.
@@ Coverage Diff @@ ## master #4793 +/- ## ========================================== - Coverage 89.69% 89.69% -0.01% ========================================== Files 289 289 Lines 40480 40527 +47 ========================================== + Hits 36310 36352 +42 - Misses 4170 4175 +5
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.