feat: add git sparse checkout #14272

yordis · 2023-06-29T15:57:03Z

No description provided.

yordis · 2023-08-25T05:13:46Z

yordis · 2023-08-25T06:19:46Z

I made it work using the git binary instead of the Git Go library ... downstream support is required unless we are OK doing everything using the git binary (which I do not see what the problem will be honestly)

yordis · 2023-08-25T06:36:17Z

Note: #11198

yordis · 2023-08-25T06:41:52Z

util/git/client.go

+
+	_, err = m.runCmd("init")
 	if err != nil {
 		return err
 	}
-	_, err = repo.CreateRemote(&config.RemoteConfig{
-		Name: git.DefaultRemoteName,
-		URLs: []string{m.repoURL},
-	})
-	return err
+
+	_, err = m.runCmd("remote", "add", git.DefaultRemoteName, m.repoURL)
+	if err != nil {
+		return err
+	}
+	return nil
+}


I HAD to do this to make things work! Whatever that package is doing, it doesn't work with sparse checkout

yordis · 2023-08-25T06:43:13Z

util/git/client.go

@@ -62,6 +62,8 @@ type gitRefCache interface {
 type Client interface {
 	Root() string
 	Init() error
+	SparseCheckout(patterns []string) error
+	Pull(revision string) error


I need this one over Fetch with sparse checkout

util/git/client.go

codecov · 2023-08-25T07:16:11Z

Codecov Report

Attention: 47 lines in your changes are missing coverage. Please review.

Comparison is base (2730170) 49.98% compared to head (594056a) 49.95%.
Report is 8 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #14272      +/-   ##
==========================================
- Coverage   49.98%   49.95%   -0.03%     
==========================================
  Files         266      266              
  Lines       46043    46110      +67     
==========================================
+ Hits        23016    23036      +20     
- Misses      20770    20805      +35     
- Partials     2257     2269      +12

Files	Coverage Δ
pkg/apis/application/v1alpha1/types.go	`58.44% <0.00%> (-0.14%)`	⬇️
reposerver/repository/repository.go	`59.51% <52.63%> (-0.34%)`	⬇️
util/git/client.go	`50.85% <42.22%> (-1.41%)`	⬇️

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

yordis · 2023-10-02T07:36:36Z

reposerver/repository/repository.go

-		err = gitClient.Fetch(revision)
-		if err != nil {
-			return status.Errorf(codes.Internal, "Failed to checkout revision %s: %v", revision, err)
+		// TODO: I am not sure about this Pull vs Fetch situation


Need help here

Have you tested sparse checkouts using fetch instead of pull? Seems like it would be ideal to limit the number of ways we retrieve repo contents.

I tried locally using a git bash script and didn't work for me. I used https://github.com/derrickstolee/sparse-checkout-example repo for it.

I am gonna try again, but I think it is require, I am not sure.

I leaved a note here to follow on it

Signed-off-by: Yordis Prieto <yordis.prieto@gmail.com> add notes run codegen asd asdsadasda run codegen Signed-off-by: Yordis Prieto <yordis.prieto@gmail.com>

crenshaw-dev · 2023-10-03T13:33:15Z

util/git/client.go

+		sb.WriteString(pattern)
+	}
+
+	if _, err := m.runCmd("sparse-checkout", "set", sb.String()); err != nil {


If I understand this correctly, the sparse-checkout setting "sticks." I think this means we need to prevent concurrent access to the repo for apps where the sparse checkout setting doesn't match (similar to what we do for Kustomize apps where some on-disk state is set by a config option).

It does stick to the configuration of a git repo. Gonna be working on that next.

Note:

git sparse-checkout disable

similar to what we do for Kustomize apps where some on-disk state is set by a config option

@crenshaw-dev I am gonna be digging into it, but in the meantime, could you share some links to where the Kustomize version is happening?

Trying to figure out how the codebase reasons about such a situation.

There's a semaphore mechanism to allow multiple read-only processes to use a repo checkout out at a particular commit at the same time.

I think this is the field which determines whether concurrency is allowed for a certain repo operation:

argo-cd/reposerver/repository/repository.go

Line 259 in c200d0c

allowConcurrent bool

sahotay · 2024-03-03T15:16:08Z

As the repo size is growing I am seeing error
failed to generate manifest for source 1 of 1: rpc error: code = Internal desc = Failed to fetch default: git fetch origin --tags --force --prunefailed exit status 128: fatal: fetch-pack: invalid index-pack output

I hope this issue will resolve this error, is this already released in any specific version?

yordis force-pushed the add-git-sparse-checkout branch 2 times, most recently from 9d2a848 to 10ab6f3 Compare August 25, 2023 04:49

yordis mentioned this pull request Aug 25, 2023

feature request: support sparse checkout go-git/go-git#90

Open

yordis force-pushed the add-git-sparse-checkout branch from 2e35678 to bcc409a Compare August 25, 2023 06:41

yordis commented Aug 25, 2023

View reviewed changes

yordis force-pushed the add-git-sparse-checkout branch from bcc409a to 6da09a9 Compare August 25, 2023 06:42

yordis commented Aug 25, 2023

View reviewed changes

yordis force-pushed the add-git-sparse-checkout branch from 6da09a9 to fbe696b Compare August 25, 2023 06:44

yordis commented Aug 25, 2023

View reviewed changes

util/git/client.go Show resolved Hide resolved

yordis force-pushed the add-git-sparse-checkout branch 5 times, most recently from 61748e1 to 888a97f Compare August 25, 2023 07:02

yordis marked this pull request as ready for review August 25, 2023 20:52

yordis marked this pull request as draft August 25, 2023 20:55

crenshaw-dev mentioned this pull request Sep 19, 2023

Improve support for monorepo by supporting git options such as depth, and sparse checkout #11198

Open

yordis force-pushed the add-git-sparse-checkout branch 3 times, most recently from 0622736 to f017698 Compare October 2, 2023 07:35

yordis commented Oct 2, 2023

View reviewed changes

yordis force-pushed the add-git-sparse-checkout branch 3 times, most recently from 271e714 to 6689cd5 Compare October 2, 2023 08:43

feat: add git sparse checkout to git client

594056a

Signed-off-by: Yordis Prieto <yordis.prieto@gmail.com> add notes run codegen asd asdsadasda run codegen Signed-off-by: Yordis Prieto <yordis.prieto@gmail.com>

yordis force-pushed the add-git-sparse-checkout branch from 6689cd5 to 594056a Compare October 2, 2023 20:15

crenshaw-dev reviewed Oct 3, 2023

View reviewed changes

blakepettersson mentioned this pull request Oct 4, 2023

Shallow clone option for reposerver? #5467

Open

blakepettersson mentioned this pull request Nov 2, 2023

feat: add depth flag #16064

Open

momilo mentioned this pull request Apr 8, 2024

Very slow performance of newest argocd versions - plugin + monorepo #17775

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add git sparse checkout #14272

feat: add git sparse checkout #14272

yordis commented Jun 29, 2023

yordis commented Aug 25, 2023

yordis commented Aug 25, 2023 •

edited

yordis commented Aug 25, 2023

yordis Aug 25, 2023

yordis Aug 25, 2023

codecov bot commented Aug 25, 2023 •

edited

yordis Oct 2, 2023

crenshaw-dev Oct 3, 2023

yordis Oct 3, 2023

crenshaw-dev Oct 3, 2023

yordis Oct 3, 2023

yordis Oct 5, 2023

yordis Oct 5, 2023

crenshaw-dev Oct 5, 2023

sahotay commented Mar 3, 2024

feat: add git sparse checkout #14272

Are you sure you want to change the base?

feat: add git sparse checkout #14272

Conversation

yordis commented Jun 29, 2023

yordis commented Aug 25, 2023

yordis commented Aug 25, 2023 • edited

yordis commented Aug 25, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Aug 25, 2023 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sahotay commented Mar 3, 2024

yordis commented Aug 25, 2023 •

edited

codecov bot commented Aug 25, 2023 •

edited