Publish the libraries a few at a time - #351
Merged
Merged
Conversation
Once the tests are done, the run publishes one library at a time, and what that costs is not the copying but waiting for the far side: each library asks rsync four times over ssh, and the one that matters lists a remote directory of tens of thousands of files to honour `--delete-excluded`. Three things were paid for per library and are now paid for once: | What | Before | After | | ---------------------------------- | ------------ | ------------ | | git log of this repository | once per lib | once | | git log of the reference files | once per lib | once per dir | | rsync connections to make the dirs | 3 | 1 | The directories are made in one connection by telling rsync -R where the relative part of the source path starts, so `emptydir/./<lib>/files` creates the branch, the library and its files directory at once instead of one rsync per level. The libraries are then published four at a time. Not more: each one is an ssh connection of its own, and sshd allows ten unauthenticated connections by default. The loop body became a function, which also stops it shadowing the `stats` dictionary of the run with a list of its own. Assisted-by: Claude Opus 5
sjoelund
enabled auto-merge (squash)
September 4, 2026 18:47
sjoelund
disabled auto-merge
September 4, 2026 19:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Once the tests are done, the run publishes one library at a time, and what that costs is not the copying but waiting for the far side: each library asks rsync four times over ssh, and the one that matters lists a remote directory of tens of thousands of files to honour
--delete-excluded.Three things were paid for per library and are now paid for once:
The directories are made in one connection by telling rsync -R where the relative part of the source path starts, so
emptydir/./<lib>/filescreates the branch, the library and its files directory at once instead of one rsync per level.The libraries are then published four at a time. Not more: each one is an ssh connection of its own, and sshd allows ten unauthenticated connections by default.
The loop body became a function, which also stops it shadowing the
statsdictionary of the run with a list of its own.Assisted-by: Claude Opus 5