Git package install optimizations #38

bgagnon · 2019-07-31T03:05:32Z

This introduces three distinct optimizations for install operations on Git packages (the only type supported in jb at this point):

using GitHub tarball archive download whenever possible
using shallow clone whenever possible
use sparse checkouts whenever possible

GitHub archive download

The most efficient case. If the repo is from https://github.com, download a tarball at the requested revision. The ETag header contains the resolved commit SHA1. When unpacking the tarball, we can skip anything that isn't under the SubDir path since it won't be needed.

This does not require authentication with GitHub. At least, not for public repositories. Perhaps this needs some adjustment to avoid breaking private repo compatibility.

Shallow clones

If the server supports it, perform a shallow clone (actually a fetch) of --depth 1 so only the objects for the requested revision are downloaded. Not all servers support this, and some do it only if the request is reasonable (ie. it won't cause too much load on its side). For this reason, recover from the error and do a full fetch (all revisions). This has been tested to work with GitLab and GitHub (in some cases, but the archive download fast track covers those).

This change replaces the previous git clone step with distinct git init, git remote add and git fetch steps. This has the side effect of not creating an extra working copy before the git checkout step (ie. equivalent to git clone -n).

Sparse checkouts

For cases where a sub directory is specified, there is no need to create a full working copy; the subdir path can act as a filter. Remember that there is no need to end up with a functional git repository since it will all be deleted by the time the package installer is done.

Benchmarks

On kube-prometheus:

before:

$ time jb install
real    0m32.044s
user    0m19.452s
sys     0m7.984s

after:

$ time jb install
real    0m7.921s
user    0m1.575s
sys     0m0.622s

YMMV.

Will happily accept any criticism on my Go style as I am still learning this language and its standard library. For a future improvement, I would love to make the package installs concurrent (or at least, the IO-bound operations like downloads).

There's an explicit git checkout command issued moments later, so there's no need to create a working copy during the clone.

If a SubDir is configured for the package, everything but that directory will be thrown away after the package is installed.

If the server supports it, fetch a specific revision with --depth 1. Otherwise, fall back to the normal fetch. This replaces the previous "clone" operation. The bandwidth and time savings can be significant depending on the history of the repository (number of commits).

bgagnon · 2019-07-31T03:06:23Z

@davidovich please have a look when you have a chance :)

sh0rez

This looks very good to me so far! Some minor things to adress.

👍

sh0rez · 2019-07-31T07:49:56Z

pkg/git.go

+	cmd.Stdout = os.Stdout
+	cmd.Stderr = os.Stderr
+	cmd.Dir = dir
+	err = cmd.Run()


This err should be checked that it is actually a failed fetch and not something else.

pkg/git.go

davidovich · 2019-07-31T20:43:49Z

pkg/git.go

+		archiveFilepath := fmt.Sprintf("%s.tar.gz", dir)
+
+		defer os.Remove(archiveFilepath)
+		commitSha, err := downloadGitHubArchive(archiveFilepath, archiveUrl)


For future providers (gitlab which has a similar archive functionality) I would fold the archive URL computation inside a downloadArchive function. This one could return a struct containing sha or version, depending on the capabilities of the service. (I believe gitlab does not support sha1 Etag).

davidovich · 2019-07-31T20:52:16Z

pkg/git.go

+		return commitSha, nil
+	}
+
+	cmd := exec.CommandContext(ctx, "git", "init")


Why not use the git clone --depth 1 --branch [version] here instead of the git init dance ?

I could not find a way to combine shallow fetch and sparse checkout into a single git clone command, hence this init + fetch + checkout combo. Do you have an idea?

metalmatze · 2019-08-02T00:38:36Z

These are awesome improvements!!! 🎉
I'm currently on vacation, but last week I've basically finished #36. Only one test is not passing for some reason.
Could you make sure to base your work on that PR? Once that one is merged, git will probably tell your about a few conflicts.

brancz · 2019-09-09T07:22:23Z

Sorry for missing this. I'm happy to review this once the above comments are addressed and rebased.

Very awesome work, thank you! 🙂

olivierlemasle · 2019-09-25T14:08:33Z

This looks a great improvement! 🎉

@bgagnon Did you had time to address the comments and rebase?

# Conflicts: # pkg/git.go

bgagnon

merged with master and addressed some review comments

this will allow the optimization to work for both ssh and https

bgagnon · 2019-09-29T21:02:49Z

Hmm something went wrong in the merge; the new tmpDir is not handled properly for archives.

bgagnon · 2019-09-29T21:27:23Z

Alright, better now. I notice the unit tests do not do any assertions on the contents of the installed package inside the vendor directory. That would make a good addition.

brancz · 2019-09-30T11:47:08Z

That would make a good addition.

Agreed.

As for now, this is in good shape I think. Let's move ahead with it! Thanks a lot for taking the time to do these very awesome improvements! 🎉

metalmatze · 2019-09-30T11:59:36Z

jb is sooooo fast now. Thanks so much! 🚀

bgagnon added 6 commits July 24, 2019 22:16

Pass -n flag to git clone

c9a5b0a

There's an explicit git checkout command issued moments later, so there's no need to create a working copy during the clone.

Use git sparse checkout whenever possible

30d7929

If a SubDir is configured for the package, everything but that directory will be thrown away after the package is installed.

Download tarball archives from GitHub (wip)

ba93fb7

use Go-native untar and gunzip support to unpack github archives

134b21a

Resolve commit SHA1 from GitHub archive ETag header

5e742d5

bgagnon mentioned this pull request Jul 31, 2019

[WIP] Cache git repos to make updating faster. #9

Closed

sh0rez requested changes Jul 31, 2019

View reviewed changes

davidovich suggested changes Jul 31, 2019

View reviewed changes

bgagnon added 6 commits September 28, 2019 10:24

Merge remote-tracking branch 'origin/master' into git-clone-optim

9268307

# Conflicts: # pkg/git.go

fix "errors" pkg double import

1fe94dc

use filepath.Join instead of string concatenation

b597b16

motivate the explicit f.Close()

93c18a7

additional error checking

5b12e9c

honor header.Mode for directories found in tarballs

76d85b1

bgagnon commented Sep 28, 2019

View reviewed changes

bgagnon added 4 commits September 28, 2019 11:23

use regex instead of prefix match to detect github remotes

524c820

this will allow the optimization to work for both ssh and https

improve github archive download error reporting

2b48551

fall back to git if github archive download fails for any reason

741e7f3

undo typo in tarball url format

41278d8

move the extracted archive into place before exiting Install

50eac0e

brancz merged commit 10e24cb into jsonnet-bundler:master Sep 30, 2019

davidovich mentioned this pull request Oct 22, 2019

Github ETag doesn't seem to provide the sha1 anymore #46

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Git package install optimizations #38

Git package install optimizations #38

bgagnon commented Jul 31, 2019 •

edited

bgagnon commented Jul 31, 2019

sh0rez left a comment

sh0rez Jul 31, 2019

davidovich Jul 31, 2019

davidovich Jul 31, 2019

bgagnon Sep 28, 2019

metalmatze commented Aug 2, 2019

brancz commented Sep 9, 2019

olivierlemasle commented Sep 25, 2019

bgagnon left a comment •

edited

bgagnon commented Sep 29, 2019

bgagnon commented Sep 29, 2019

brancz commented Sep 30, 2019

metalmatze commented Sep 30, 2019

Git package install optimizations #38

Git package install optimizations #38

Conversation

bgagnon commented Jul 31, 2019 • edited

GitHub archive download

Shallow clones

Sparse checkouts

Benchmarks

bgagnon commented Jul 31, 2019

sh0rez left a comment

Choose a reason for hiding this comment

sh0rez Jul 31, 2019

Choose a reason for hiding this comment

davidovich Jul 31, 2019

Choose a reason for hiding this comment

davidovich Jul 31, 2019

Choose a reason for hiding this comment

bgagnon Sep 28, 2019

Choose a reason for hiding this comment

metalmatze commented Aug 2, 2019

brancz commented Sep 9, 2019

olivierlemasle commented Sep 25, 2019

bgagnon left a comment • edited

Choose a reason for hiding this comment

bgagnon commented Sep 29, 2019

bgagnon commented Sep 29, 2019

brancz commented Sep 30, 2019

metalmatze commented Sep 30, 2019

bgagnon commented Jul 31, 2019 •

edited

bgagnon left a comment •

edited