Feature request: shallow clone for git repos #972

heitorPB · 2019-06-22T19:24:53Z

It would save some time to do a shallow clone of a git repo, instead of getting the full history:

git clone --depth 1 --recursive --shallow-submodules <repo>

The text was updated successfully, but these errors were encountered:

Morganamilo · 2019-06-22T19:28:05Z

It would save some time to do a shallow clone of a git repo, instead of getting the full history:

Prove it.

heitorPB · 2019-06-22T21:41:36Z

Prove it.

$ time git clone https://github.com/KiCad/kicad-source-mirror.git
Cloning into 'kicad-source-mirror'...
remote: Enumerating objects: 76, done.
remote: Counting objects: 100% (76/76), done.
remote: Compressing objects: 100% (59/59), done.
remote: Total 189715 (delta 34), reused 35 (delta 17), pack-reused 189639
Receiving objects: 100% (189715/189715), 356.21 MiB | 9.87 MiB/s, done.
Resolving deltas: 100% (150876/150876), done.

real	0m44.684s
user	0m43.036s
sys	0m9.057s

$ time git clone --depth 1 --recursive --shallow-submodules https://github.com/KiCad/kicad-source-mirror.git
Cloning into 'kicad-source-mirror'...
remote: Enumerating objects: 4407, done.
remote: Counting objects: 100% (4407/4407), done.
remote: Compressing objects: 100% (4054/4054), done.
remote: Total 4407 (delta 1250), reused 1015 (delta 321), pack-reused 0
Receiving objects: 100% (4407/4407), 20.78 MiB | 6.25 MiB/s, done.
Resolving deltas: 100% (1250/1250), done.

real	0m8.779s
user	0m2.518s
sys	0m0.800s

It is faster to get only the last commit of a branch than to get the full history of a repo. Enough for you?

Morganamilo · 2019-06-22T21:43:40Z

Those are upstream git repos and is makepkg's job to handle. Yay only downloads the AUR repos which are small enough that a shallow clone would have no impact.

And as far as makepkg goes, they refuse to support this: https://bugs.archlinux.org/task/52957?

Jguer · 2019-06-23T10:41:05Z

I think it's just a confusion of what yay actually git clones when you pull a PKGBUILD.
When yay says it's cloning a repo (and let's use kicad-git as the example you chose) it's cloning https://aur.archlinux.org/kicad-git.git that only has a the very small PKGBUILD file and not the place where the code actually lives https://git.launchpad.net/kicad.

% time git clone https://aur.archlinux.org/kicad-git.git                    /tmp
Cloning into 'kicad-git'...
remote: Enumerating objects: 122, done.
remote: Counting objects: 100% (122/122), done.
remote: Compressing objects: 100% (110/110), done.
remote: Total 122 (delta 12), reused 122 (delta 12)
Receiving objects: 100% (122/122), 36.69 KiB | 1.08 MiB/s, done.
Resolving deltas: 100% (12/12), done.
git clone https://aur.archlinux.org/kicad-git.git  0.05s user 0.03s system 11% cpu 0.628 total

% time git clone --depth 1 --recursive --shallow-submodules https://aur.archlinux.org/kicad-git.git
Cloning into 'kicad-git'...
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 5 (delta 0), reused 5 (delta 0)
Unpacking objects: 100% (5/5), done.
git clone --depth 1 --recursive --shallow-submodules   0.06s user 0.04s system 13% cpu 0.764 total

Although this data has no statistical relevance since it's not done in a controlled and measured environment, it relies only on one sample and it won't be possible to replicate specifically it should at least hint that there is no advantage to shallow cloning PKGBUILD repos (specially when they are so useful for diffing).

In relation to makepkg clones: not our job, @Morganamilo s link also shows they are aware. Maybe in alternative you can shallow git clone the repo and then run makepkg, I do think it should pick up the source.

Closing.

heitorPB · 2019-06-23T13:13:38Z

It makes sense. For some unknown reason, I thought it was yay's job to parse the PKGBUILD... Thanks for clearing things up.

…

On Sun, 23 Jun 2019, 07:41 J Guerreiro, ***@***.***> wrote: I think it's just a confusion of what yay actually git clones when you pull a PKGBUILD. When yay says it's cloning a repo (and let's use kicad-git as the example you chose) it's cloning https://aur.archlinux.org/kicad-git.git that only has a the very small PKGBUILD file and not the place where the code actually lives https://git.launchpad.net/kicad. % time git clone https://aur.archlinux.org/kicad-git.git /tmp Cloning into 'kicad-git'... remote: Enumerating objects: 122, done. remote: Counting objects: 100% (122/122), done. remote: Compressing objects: 100% (110/110), done. remote: Total 122 (delta 12), reused 122 (delta 12) Receiving objects: 100% (122/122), 36.69 KiB | 1.08 MiB/s, done. Resolving deltas: 100% (12/12), done. git clone https://aur.archlinux.org/kicad-git.git 0.05s user 0.03s system 11% cpu 0.628 total % time git clone --depth 1 --recursive --shallow-submodules https://aur.archlinux.org/kicad-git.git Cloning into 'kicad-git'... remote: Enumerating objects: 5, done. remote: Counting objects: 100% (5/5), done. remote: Compressing objects: 100% (5/5), done. remote: Total 5 (delta 0), reused 5 (delta 0) Unpacking objects: 100% (5/5), done. git clone --depth 1 --recursive --shallow-submodules 0.06s user 0.04s system 13% cpu 0.764 total Although this data has no statistical relevance since it's not done in a controlled and measured environment, it relies only on one sample and it won't be possible to replicate specifically it should at least hint that there is no advantage to shallow cloning PKGBUILD repos (specially when they are so useful for diffing). In relation to makepkg clones: not our job, @Morganamilo <https://github.com/Morganamilo> s link also shows they are aware. Maybe in alternative you can shallow git clone the repo and then run makepkg, I do think it should pick up the source. Closing. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#972?email_source=notifications&email_token=ADGWRRVDYHDH2RVJQTP7HVDP35HMJA5CNFSM4H2XNPY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYK3PRQ#issuecomment-504739782>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADGWRRXQNUX3DG77TQAJARTP35HMJANCNFSM4H2XNPYQ> .

soloturn · 2020-03-23T00:50:13Z

@Jguer , @Morganamilo you are right when you say it makes sense. but - otoh - makepkg is very modular and crealting a file "makepkg-shallow" with the following contents and call it instead of makepkg would do the trick.

#!/bin/bash 

git() {
  if [[ $# -gt 1 && $1 == 'clone' ]]; then
    /bin/git "$@" --depth=1
  else
    /bin/git "$@"
  fi
}

source /bin/makepkg "$@"

i know it woudl be not beautiful to add a script like this to the yay codebase. but it is so small that one might try it, switchable, in yay, and if it works suggst to add it upstream.

sources:

andrew gregory, make-shallow.sh https://gist.github.com/andrewgregory/770cbedfa2da3817d762483155a330b8
allan mcrae, saying if somebody comes up with a switch he'd consider adding it: https://bugs.archlinux.org/task/34677

Jguer · 2020-03-25T10:37:34Z

just those /bin/ entries will create issues for users

this is a makepkg/PKGBUILD specific issue, not a package fetching and dependency resolving issue.

I would prefer the switch to be added on makepkg as we support makepkg options

JakkuSakura · 2020-09-11T17:48:52Z

Sorry for my misunderstanding

ryuukk · 2022-03-06T15:23:12Z

Although this data has no statistical relevance since it's not done in a controlled and measured environment, it relies only on one sample and it won't be possible to replicate specifically it should at least hint that there is no advantage to shallow cloning PKGBUILD repos (specially when they are so useful for diffing).

Wrong

Look the improvements it makes ImperatorStorm/PKGBUILDs#4

grandchild · 2022-03-06T15:29:19Z

@ryuukk Again, this is the same misunderstanding explained in the comment that you quote. Your example is cloning the upstream source repo, not cloning the AUR repo containing mainly the PKGBUILD. These are two different git operations.

yay clones the AUR PKGBUILD repo, makepkg then clones the source repo. Since the possible speedups in the first case are super tiny (and this is the only thing in yay's responsibility!) this issue is rightfully closed. If anything it's a makepkg feature request.

HaleTom · 2023-08-23T07:26:10Z

For others interested in this, paru supports --makepkg $HOME/bin/makpkg and can be used with the wrapper solution above, however I suggest --filter=blob:none instead of --shallow.

Denzy7 · 2023-10-16T14:03:44Z

pacman-git and GITFLAGS="--depth=1" yay -S package works. You could also put GITFLAGS in /etc/makepkg.conf and ensure your ~/.config/yay/config.json has "makepkgconf": "/etc/makepkg.conf". GITFLAGS will however break packages that need a specific tagged commit version

nikelborm · 2024-01-24T20:58:23Z

implemented by #2348

HaleTom · 2024-02-26T09:13:50Z

The Arch Wiki now gives an example of how to use GITFLAGS="--filter=tree:0" makepkg

nikelborm · 2024-02-26T15:29:42Z

Now the question is how to make yay work with this env var

Jguer closed this as completed Jun 23, 2019

Jguer mentioned this issue Feb 15, 2022

Don't clone entire git repository when compiling a library from git #1713

Closed

Jguer added Status: Upstream Issue Type: Feature Request labels Feb 16, 2022

0pcom mentioned this issue Sep 14, 2022

git clone excessive bandwidth skycoin/skywire#1340

Closed

Jguer mentioned this issue Feb 23, 2023

Reduces time and internet data consumption when git cloning #1934

Closed

HaleTom mentioned this issue Aug 24, 2023

Support git blobless clones Morganamilo/paru#1027

Closed

Jguer mentioned this issue Jan 25, 2024

Refactor: speed up all clone operations by loading all commits compressed into 1 #2348

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature request: shallow clone for git repos #972

Feature request: shallow clone for git repos #972

heitorPB commented Jun 22, 2019

Morganamilo commented Jun 22, 2019

heitorPB commented Jun 22, 2019

Morganamilo commented Jun 22, 2019

Jguer commented Jun 23, 2019

heitorPB commented Jun 23, 2019 via email

soloturn commented Mar 23, 2020 •

edited

Jguer commented Mar 25, 2020

JakkuSakura commented Sep 11, 2020 •

edited

ryuukk commented Mar 6, 2022

grandchild commented Mar 6, 2022

HaleTom commented Aug 23, 2023 •

edited

Denzy7 commented Oct 16, 2023 •

edited

nikelborm commented Jan 24, 2024

HaleTom commented Feb 26, 2024

nikelborm commented Feb 26, 2024 •

edited

Feature request: shallow clone for git repos #972

Feature request: shallow clone for git repos #972

Comments

heitorPB commented Jun 22, 2019

Morganamilo commented Jun 22, 2019

heitorPB commented Jun 22, 2019

Morganamilo commented Jun 22, 2019

Jguer commented Jun 23, 2019

heitorPB commented Jun 23, 2019 via email

soloturn commented Mar 23, 2020 • edited

Jguer commented Mar 25, 2020

JakkuSakura commented Sep 11, 2020 • edited

ryuukk commented Mar 6, 2022

grandchild commented Mar 6, 2022

HaleTom commented Aug 23, 2023 • edited

Denzy7 commented Oct 16, 2023 • edited

nikelborm commented Jan 24, 2024

HaleTom commented Feb 26, 2024

nikelborm commented Feb 26, 2024 • edited

soloturn commented Mar 23, 2020 •

edited

JakkuSakura commented Sep 11, 2020 •

edited

HaleTom commented Aug 23, 2023 •

edited

Denzy7 commented Oct 16, 2023 •

edited

nikelborm commented Feb 26, 2024 •

edited