flatpak-builder: allow to shallow clone git repos

[alexlarsson]

From @alexlarsson on May 19, 2016 15:32

From @hbons on February 27, 2016 21:39

Git repos can be huge, and when building you may not be interested in the full history.
Maybe a property in the json file (or even changing the default) to allow shallow cloning would save a lot of time and bandwidth.

Copied from original issue: alexlarsson/xdg-app#112

Copied from original issue: flatpak/flatpak#15

[alexlarsson]

Hmm, I can see how this could be useful. But its also somewhat tricky. For the initial pull things are fine, but on an update the depth is stored, so the new pull will drop all local history other than the new head of the branch you pulled.

If you're only pulling a repo once this is not really a problem. However, with xdg-app you can have a single download cache shared by multiple json files, or the same json file can refer to the same git repo (either directly or via submodules) which could be problematic if those refered to different branches (or direct commit ids).

I wish there was a way to do an initial shallow clone, but then allow further pulls to just get all changes between your current commit and the new HEAD.

[alexlarsson]

From @hbons on February 29, 2016 13:21

I think recent Git releases fixed this. You can do a shallow clone and then pull whatever else you need later.

[alexlarsson]

There was some kind of fix, but I don't think it does quite what we want here anyway.

[alexlarsson]

From @hbons on February 29, 2016 15:35

Shallow clones are the default setting in SparkleShare and it works great, so it might be worth looking into. :)

[alexlarsson]

From @jcelerier on June 1, 2016 9:47

+1 for shallow clones

[alexlarsson]

From @albfan on March 11, 2017 10:39

Any little test that can be configured to implement this? Like a flatpak project with a dependency with more than one commit. Doing a shallow copy from dep and commiting another change to it should be enough to test if this will work.

I'm trying to install gnome-todo with flatpak and there's a huge repos gnome-online-accounts and evolution-data-server that has no sense to download entirely.

Moreover when I have them downloaded to use with jhbuild. is there too an option to reuse existing repo clones?

Another problem related is on a unsuccesful build (interrunped) flatpak cannot reuse clone_tmp dir and fails (maybe that can be solved too here or another issue is needed)

[alexlarsson]

From @albfan on March 11, 2017 10:57

Humm, shallow copies fails

telepathy-account-widgets is a submodule of gnome-online-accounts and it seek for a specific commit so after bare clone, depth 1 is not correct

Given that I have those repos cloned, is there a way to instruct flatpak to use existing clones (like ~/jhbuild/checkout)?

[albfan]

Shallow Clone a repo to a specific commit should work, but that's something git needs to provide (or any other VCS)

[adityashah1212]

@albfan Yes git is capable of that, look at this answer

[albfan]

Then

git clone -b <branch> git@git.example.com:repo --depth 1

is possible

Seems seing https://stackoverflow.com/a/19394911/848072

that you can do

git fetch --depth 1 <branch>

Implement that could be a disk space and time saver

[alexlarsson]

It is possible to make shallow clones of a branch, yes, but what if i want a particular commit. For instance, say the git source in the json manifest has only a git uri + a commit id. This is quite common, because we want reproducible builds (not just latest branch head). Additionally, it is hit every time we mirror git submodules, because submodules are always refered to by commit id, and never by branch name.

[albfan]

fetch says it can be refspec, so maybe

git fetch --depth 1 <refspec>

would work

I'm reluctant this will work correctly in all situations, is just I don't know where to start to create a use case and what to modify to test it

Feel free to close it

[alexlarsson]

I don't want to close this because I want this feature, but it just doesn't seem possible:

$ git init --bare gtk
Initialized empty Git repository in /var/tmp/gtk
$ cd gtk/
$ git remote add --mirror=fetch origin https://git.gnome.org/browse/gtk+
$ git fetch --depth 1 origin master
remote: Counting objects: 4328, done.
remote: Compressing objects: 100% (3864/3864), done.
remote: Total 4328 (delta 1296), reused 1940 (delta 327)
Receiving objects: 100% (4328/4328), 18.49 MiB | 2.75 MiB/s, done.
Resolving deltas: 100% (1296/1296), done.
From https://git.gnome.org/browse/gtk+
 * branch            master     -> FETCH_HEAD
 * [new branch]      master     -> master
$ git fetch --depth 1 origin 5b2cae6703d8a55e047ef2d830dbe80c392feac7
error: Server does not allow request for unadvertised object 5b2cae6703d8a55e047ef2d830dbe80c392feac7

[alexlarsson]

Hmm, in lastest upstream docs they say:

If the remote has enabled the options uploadpack.allowTipSHA1InWant, uploadpack.allowReachableSHA1InWant, or uploadpack.allowAnySHA1InWant, they may alternatively be 40-hex sha1s present on the remote.

So, maybe it will eventually be possible?

[albfan]

Adding those config now error changes:

git init --bare gtk
cd gtk/
git remote add --mirror=fetch origin https://git.gnome.org/browse/gtk+
git config uploadpack.allowTipSHA1InWant true
git config uploadpack.allowReachableSHA1InWant true
git fetch --depth 1 origin master
git fetch --depth 1 origin 5b2cae6703d8a55e047ef2d830dbe80c392feac7

error: no such remote ref 5b2cae6703d8a55e047ef2d830dbe80c392feac7

https://git.gnome.org/browse/gtk+/commit/?h=wip/matthiasc/pipeline-layouts&id=5b2cae6703d8a55e047ef2d830dbe80c392feac7

[alexlarsson]

The config has to be on the server side.

[albfan]

initial implementation on #47