Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rework checkout mechanism again, again #9

Closed
capnfabs opened this issue Jan 14, 2020 · 5 comments
Closed

Rework checkout mechanism again, again #9

capnfabs opened this issue Jan 14, 2020 · 5 comments
Labels
internal-cleanup Internal Cleanups / Refactors
Milestone
0.2

Comments

@capnfabs
Copy link
Owner

capnfabs commented Jan 14, 2020 •
edited
Loading

Ok, this time I think I've finally got something pretty performant with minimal hacks.

Here's a series of bash commands I used to do this on a repo with nested submodules (https://github.com/capnfabs/grouse/blob/master/test-fixtures/nested-submodules-missing-submodules.zip)

~# git clone /app/nested-submodules-missing-submodules clone
Cloning into 'clone'...
done.
~# cd clone/
~/clone# git submodule init themes/bare/
Submodule 'themes/bare' (https://github.com/orf/bare-hugo-theme.git) registered for path 'themes/bare'
~/clone# export REPO=/app/nested-submodules-missing-submodules
~/clone# git -c submodule."themes/bare".url=$REPO/themes/bare submodule update themes/bare
Cloning into 'themes/bare'...
done.
Submodule path 'themes/bare': checked out '6f2f95a1ce89a6e642491072e848eba0b23fac2d'
~/clone# cat .git/modules/themes/bare/config
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
	worktree = ../../../../themes/bare
[remote "origin"]
	url = /app/nested-submodules-missing-submodules/themes/bare
	fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
	remote = origin
	merge = refs/heads/master
~/clone# git submodule update themes/bare
~/clone# # i feel like the remote is supposed to point at the original submodule URL, not this one
~/clone# cd themes/bare/
~/clone/themes/bare# git remote -v
origin	/app/nested-submodules-missing-submodules/themes/bare (fetch)
origin	/app/nested-submodules-missing-submodules/themes/bare (push)
~/clone/themes/bare# git remote remove origin
~/clone/themes/bare# git submodule init assets/css/
bulma/     site.sass
~/clone/themes/bare# git submodule init assets/css/bulma
Submodule 'assets/css/bulma' (https://github.com/jgthms/bulma.git) registered for path 'assets/css/bulma'
~/clone/themes/bare# git -c submodule."assets/css/bulma".url=$REPO/themes/bare/assets/css/bulma submodule update assets/css/bulma
fatal: repository '/app/nested-submodules-missing-submodules/themes/bare/assets/css/bulma' does not exist
fatal: clone of '/app/nested-submodules-missing-submodules/themes/bare/assets/css/bulma' into submodule path 'assets/css/bulma' failed
~/clone/themes/bare# echo $?
128
~/clone/themes/bare# git submodule update --depth 1 assets/css/bulma
Cloning into 'assets/css/bulma'...
remote: Enumerating objects: 898, done.
remote: Counting objects: 100% (898/898), done.
remote: Compressing objects: 100% (870/870), done.
remote: Total 898 (delta 58), reused 545 (delta 22), pack-reused 0
Receiving objects: 100% (898/898), 63.89 MiB | 269.00 KiB/s, done.
Resolving deltas: 100% (58/58), done.
Checking connectivity... done.
fatal: reference is not a tree: 617febbfbf5b95e827d20654a16748c66bb5d647
Unable to checkout '617febbfbf5b95e827d20654a16748c66bb5d647' in submodule path 'assets/css/bulma'
root@0778ade78b9c:~/clone/themes/bare# cd assets/css/bulma/
root@0778ade78b9c:~/clone/themes/bare/assets/css/bulma# git fetch --depth=1000000
remote: Enumerating objects: 14301, done.
remote: Counting objects: 100% (14301/14301), done.
remote: Compressing objects: 100% (3631/3631), done.
remote: Total 13879 (delta 10016), reused 13404 (delta 9542), pack-reused 0Receiving objects:  99% (1

Receiving objects: 100% (13879/13879), 16.74 MiB | 186.00 KiB/s, done.
Resolving deltas: 100% (10016/10016), completed with 273 local objects.
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
From https://github.com/jgthms/bulma
 * [new tag]         0.0.1      -> 0.0.1
 * [new tag]         0.0.10     -> 0.0.10
 * [new tag]         0.0.11     -> 0.0.11
 * [new tag]         0.0.12     -> 0.0.12
 * [new tag]         0.0.13     -> 0.0.13
 * [new tag]         0.0.14     -> 0.0.14
 * [new tag]         0.0.15     -> 0.0.15
 * [new tag]         0.0.16     -> 0.0.16
 * [new tag]         0.0.17     -> 0.0.17
 * [new tag]         0.0.18     -> 0.0.18
 * [new tag]         0.0.19     -> 0.0.19
 * [new tag]         0.0.2      -> 0.0.2
 * [new tag]         0.0.20     -> 0.0.20
 * [new tag]         0.0.21     -> 0.0.21
 * [new tag]         0.0.22     -> 0.0.22
 * [new tag]         0.0.23     -> 0.0.23
 * [new tag]         0.0.24     -> 0.0.24
 * [new tag]         0.0.25     -> 0.0.25
 * [new tag]         0.0.26     -> 0.0.26
 * [new tag]         0.0.27     -> 0.0.27
 * [new tag]         0.0.28     -> 0.0.28
 * [new tag]         0.0.3      -> 0.0.3
 * [new tag]         0.0.4      -> 0.0.4
 * [new tag]         0.0.5      -> 0.0.5
 * [new tag]         0.0.7      -> 0.0.7
 * [new tag]         0.0.8      -> 0.0.8
 * [new tag]         0.1.0      -> 0.1.0
 * [new tag]         0.1.1      -> 0.1.1
 * [new tag]         0.1.2      -> 0.1.2
 * [new tag]         0.2.0      -> 0.2.0
 * [new tag]         0.2.1      -> 0.2.1
 * [new tag]         0.3.0      -> 0.3.0
 * [new tag]         0.3.1      -> 0.3.1
 * [new tag]         0.3.2      -> 0.3.2
 * [new tag]         0.4.0      -> 0.4.0
 * [new tag]         0.4.1      -> 0.4.1
 * [new tag]         0.4.2      -> 0.4.2
 * [new tag]         0.4.4      -> 0.4.4
 * [new tag]         0.5.0      -> 0.5.0
 * [new tag]         0.5.1      -> 0.5.1
 * [new tag]         0.5.2      -> 0.5.2
 * [new tag]         0.5.3      -> 0.5.3
 * [new tag]         0.6.0      -> 0.6.0
 * [new tag]         0.6.1      -> 0.6.1
 * [new tag]         0.6.2      -> 0.6.2
 * [new tag]         0.7.0      -> 0.7.0
 * [new tag]         0.7.1      -> 0.7.1
 * [new tag]         0.7.2      -> 0.7.2
 * [new tag]         0.7.3      -> 0.7.3
 * [new tag]         0.7.4      -> 0.7.4
 * [new tag]         0.7.5      -> 0.7.5
 * [new tag]         0.8.0      -> 0.8.0
root@0778ade78b9c:~/clone/themes/bare/assets/css/bulma# cd -
/root/clone/themes/bare
root@0778ade78b9c:~/clone/themes/bare# git submodule update assets/css/bulma
Submodule path 'assets/css/bulma': checked out '617febbfbf5b95e827d20654a16748c66bb5d647'

So basically, the idea is:

  • Clone from the current working tree
  • Init each submodule, attempt to clone from the filesystem
  • Then, we should probably modify the remote such that 'origin' points at the real remote again
  • If cloning from filesystem doesn't work, just use a regular git submodule update.

Something that I don't love about this is -- you can't use git submodule update --depth 1 for nested submodules, because under some circumstances, they point at commits (which you can't fetch directly from remotes). This is just a git problem in general though.

I tested this all on git 2.7.4, which is the current version running on Ubuntu LTS Xenial. I think it would be good to try on git 2.1.4 (Debian Jessie) as well -- but that's probably the oldest stable version we need to support by a long way.

Quick note to performance -- apparently if the cloned repo is on the same disk, then git uses hardlinks so this is mega fast (see Local Protocols in Git On the Server)
@capnfabs capnfabs added the internal-cleanup Internal Cleanups / Refactors label Jan 14, 2020
@jasikpark jasikpark mentioned this issue Mar 13, 2020
Closed
@capnfabs
Copy link
Owner Author

Ok, picking this up again. Rather than rely upon the hardlink behaviour, I'm going to do git clone --shared:

When the repository to clone is on the local machine, instead of using hard links, automatically setup .git/objects/info/alternates to share the objects with the source repository. The resulting repository starts out without any object of its own.

This works regardless of whether the temporary directory is on the same mount or not, which is a major advantage of relying upon hardlinks ✨

@capnfabs
Copy link
Owner Author

capnfabs commented Jun 10, 2020 •
edited
Loading

Wow, this is working really well and going much faster than I'd expected. I think having five months of noodling around in Rust code has taught me some pretty good habits, and a lot of the skills were transferable.

@capnfabs
Copy link
Owner Author

Still todo:

  • Add the hack to swap back in the original remote on submodules after clone
  • Fix the tests, make the test suite more comprehensive and also not reliant upon 180MB of remote repos 🙃

@capnfabs
Copy link
Owner Author

capnfabs commented Jun 12, 2020 •
edited
Loading

Just some logging improvements left!

@capnfabs
Copy link
Owner Author

Merged.

@capnfabs capnfabs added this to the 0.2 milestone Jun 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
internal-cleanup Internal Cleanups / Refactors
Projects
None yet
Milestone
0.2
Development

No branches or pull requests
1 participant
@capnfabs