add function to invoke Git with disabled LFS filters #2453

larsxschneider · 2017-07-30T19:30:36Z

Git LFS invokes Git for various actions internally. A new Git instance is started with the default Git config and therefore might invoke another Git LFS process via the filter machinery. I have noticed that in #2439.

Teach Git LFS to invoke Git with disabled LFS filters to prevent unnecessary Git LFS processes.

I recommend to review the changes by commit.

Attention: I haven't performed extensive testing on this, yet.

ttaylorr · 2017-07-31T15:35:23Z

git/git.go

+}
+
+// Invoke Git with disabled LFS filters
+func Exec(args ...string) *subprocess.Cmd {


I would like to unexport this and SimpleExec below, so that outside callers cannot git.Exec(...). This type of code allows arbitrary usage of Git, which I would like to prevent. If we find ourselves needing a new git process that isn't already covered in this package, let's add it here where we can track it using static analysis tools, etc., instead of searching through the code.

You mean these kind of calls:
https://github.com/larsxschneider/git-lfs/blob/7418ab2974609e0fd2ec0d4e131126595feb31ed/commands/command_pointer.go#L120
... should go into the git package?

Yes, ideally I'd like that to turn into something like:

h, err := git.HashObject(someReader)

Where:

The exec("hash-object", ...) would live inside of the git package, and

It would be exposed with a function exported from package git called HashObject().

ttaylorr · 2017-07-31T15:35:30Z

git/git.go

+}
+
+// Invoke Git with disabled LFS filters
+func SimpleExec(args ...string) (string, error) {


Same note here as above.

ttaylorr · 2017-07-31T15:35:54Z

git/git.go

@@ -490,13 +490,45 @@ func (c *gitConfig) IsGitVersionAtLeast(ver string) bool {
 	return IsVersionAtLeast(gitver, ver)
 }

+// Prepend Git config instructions to disable Git LFS filter
+func PrependConfigToDisableLFS(args ...string) []string {


Do outside callers need this? If not, I'd like to un-export this.

No outside caller needs this. Un-export is done by a lower case function name?

ttaylorr · 2017-07-31T15:40:58Z

git/git.go

@@ -490,13 +490,45 @@ func (c *gitConfig) IsGitVersionAtLeast(ver string) bool {
 	return IsVersionAtLeast(gitver, ver)
 }

+// Prepend Git config instructions to disable Git LFS filter
+func PrependConfigToDisableLFS(args ...string) []string {


Additionally, I'm not sure that disabling LFS in all uses of Exec(), and SimpleExec() is safe. If you want to keep this function, I'd recommend that we instead change the calls from:

// Exec runs Git, disabling LFS, with the given arguments. func Something() ([]byte, error) { return Exec("something", ...) }

to:

@@ -1,5 +1,5 @@ -// Exec runs Git, disabling LFS, with the given arguments. +// Exec runs Git with the given arguments. func Something() ([]byte, error) { - return Exec("something", ...) + return Exec(PrependConfigToDisableLFS("something")...) }

I am not sure I understand what you're saying here.
Would you prefer it if I don't use the additional Git Exec wrapper functions?

In case of config changes disabling LFS wouldn't be safe. But everything else should be, no?

I am in favor of the wrapper functions, with two recommendations:

Un-export Exec() and SimpleExec().

Make exec() and simpleExec() execute their arguments literally, without implicitly prepending the LFS-disabling arguments to Git. If a caller wants to run a Git command that doesn't then go back and run LFS, they should call exec like: exec(disableLfs(args...)).

re 2) My thinking was that "disablingLFS" should be the "default" case, no? Although it's a bit verbose I agree with the explicit prepending!

re 2) My thinking was that "disablingLFS" should be the "default" case, no?

I would prefer that this not be the default case. I think that this approach will make it clearer that by calling exec()/simpleExec() you have to explicitly opt-in to not using LFS.

I agree though that this approach makes it harder for new callers to know when to add this and when not to. What do you think about execWithLFS and execWithoutLfs as another alternative?

larsxschneider · 2017-08-02T20:23:05Z

@ttaylorr subprocess/cmd and lfs/gitscanner_cmd look pretty similar. Would it make sense to extend subprocess and use it the lfs package? Or did you you introduce the duplication to keep the lfs package as independent as possible?

ttaylorr · 2017-08-02T23:08:26Z

@ttaylorr subprocess/cmd and lfs/gitscanner_cmd look pretty similar. Would it make sense to extend subprocess and use it the lfs package?

If lfs could import subprocess I think that would be a good idea 👍 .

Git LFS invokes Git for various actions internally. The new Git instance is started with the default Git config and therefore might invoke another Git LFS "filter-process". That is usually not be desired. Teach Git LFS to invoke Git with or without LFS filters.

The only side effect of this refactoring should be that the Git LFS is disabled for the Git subprocess. No other functional changes are intended.

larsxschneider · 2017-08-21T12:40:34Z

@ttaylorr I am pretty happy with the current state. If you have time, please give is thorough review 😄

larsxschneider · 2017-08-21T13:23:24Z

test: pre-push with their lock on non-lfs lockable file seems to be flaky 😢

technoweenie

Hey, this is a great refactor. Thanks :)

I do have some minor comments though:

technoweenie · 2017-08-21T14:27:32Z

git/git.go

+	return bufio.NewScanner(cmd.Stdout), nil
+}
+
+func HashObject(by []byte) (string, error) {


What do you think about changing this to accept a reader? This will allow it to take larger objects without being forced to read it into memory first. Then you can get rid of the bytes.NewReader() in here.

I realize that the usage in commands/command_pointer.go will be just fine buffering in memory. But, I think it makes sense to change this function since it's now an exported git package function.

technoweenie · 2017-08-21T14:34:39Z

git/git.go

-func UpdateIndex(file string) error {
-	_, err := subprocess.SimpleExec("git", "update-index", "-q", "--refresh", file)
-	return err
+func StartUpdateIndexFromStdin(output *bytes.Buffer) (io.WriteCloser, error) {


In the same vein as my git.HashObject() comment, what do you think about changing this to accept an io.Writer? It'll give the caller some flexibility in how the output is captured.

Both done! Thanks for the suggestion 👍

The only side effect of this refactoring should be that the Git LFS is disabled for the Git subprocess. No other functional changes are intended.

…output Replicate `lfs/gitscanner_cmd` in `subprocess/buffered_cmd` using subprocess/cmd. In subsequent commits we will migrate all functions users of `lfs/gitscanner_cmd` to `subprocess/buffered_cmd` and afterwards remove `lfs/gitscanner_cmd`.

Git is called with the LFS filter here. No functional changes are intended.

The only side effect of this refactoring should be that the Git LFS is disabled for the Git subprocess. No other functional changes are intended.

larsxschneider requested review from technoweenie and ttaylorr July 30, 2017 19:30

larsxschneider changed the title ~~add function to invoke Git with disabled LFS filters~~ WIP: add function to invoke Git with disabled LFS filters Jul 30, 2017

larsxschneider force-pushed the lars/gitexec branch from aa4c822 to 8bb8614 Compare July 30, 2017 22:33

ttaylorr suggested changes Jul 31, 2017

View reviewed changes

larsxschneider force-pushed the lars/gitexec branch 7 times, most recently from dd9ae18 to c86cd49 Compare August 2, 2017 20:20

git-lfs deleted a comment from ttaylorr Aug 2, 2017

larsxschneider removed the request for review from technoweenie August 2, 2017 20:25

larsxschneider force-pushed the lars/gitexec branch from c86cd49 to 83b5b8c Compare August 21, 2017 10:17

larsxschneider added 3 commits August 21, 2017 13:33

git: use wrapper functions to call Git with Git LFS disabled

4a68689

The only side effect of this refactoring should be that the Git LFS is disabled for the Git subprocess. No other functional changes are intended.

git: remove unused UpdateIndex function

b47ced3

larsxschneider force-pushed the lars/gitexec branch 2 times, most recently from a750271 to 6c0af76 Compare August 21, 2017 12:22

larsxschneider changed the title ~~WIP: add function to invoke Git with disabled LFS filters~~ add function to invoke Git with disabled LFS filters Aug 21, 2017

larsxschneider force-pushed the lars/gitexec branch from 6c0af76 to 2b92300 Compare August 21, 2017 12:46

larsxschneider closed this Aug 21, 2017

larsxschneider reopened this Aug 21, 2017

technoweenie suggested changes Aug 21, 2017

View reviewed changes

move git update-index invocation to git package

ae4d261

The only side effect of this refactoring should be that the Git LFS is disabled for the Git subprocess. No other functional changes are intended.

larsxschneider added 8 commits August 22, 2017 19:34

move gitHashObject() to git package

9c0f432

The only side effect of this refactoring should be that the Git LFS is disabled for the Git subprocess. No other functional changes are intended.

move git diff-index invocation to git package using gitBuffered

cddf98f

Git is called with the LFS filter here. No functional changes are intended.

move git cat-file invocation to git package using gitNoLFSBuffered

5bb498f

The only side effect of this refactoring should be that the Git LFS is disabled for the Git subprocess. No other functional changes are intended.

move git log invocation to git package using gitNoLFSBuffered

aba51b6

The only side effect of this refactoring should be that the Git LFS is disabled for the Git subprocess. No other functional changes are intended.

move git ls-tree invocation to git package using gitNoLFSBuffered

de8ece9

The only side effect of this refactoring should be that the Git LFS is disabled for the Git subprocess. No other functional changes are intended.

lfs: remove unused gitscanner_cmd.go

d113dc7

git: change gitHashObject() to accept an io.Reader

376464d

larsxschneider force-pushed the lars/gitexec branch from 2b92300 to 376464d Compare August 22, 2017 17:35

technoweenie approved these changes Aug 22, 2017

View reviewed changes

ttaylorr approved these changes Aug 24, 2017

View reviewed changes

ttaylorr merged commit 39c7bf6 into master Aug 24, 2017

ttaylorr deleted the lars/gitexec branch August 24, 2017 20:24

larsxschneider mentioned this pull request Aug 26, 2017

git-lfs died of signal 9 #2439

Closed

larsxschneider mentioned this pull request Sep 4, 2017

Windows implementation leaks handles #2350

Open

technoweenie mentioned this pull request Oct 4, 2017

git lfs pull marks files as modified if git lfs configured with --skip-smudge #2646

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add function to invoke Git with disabled LFS filters #2453

add function to invoke Git with disabled LFS filters #2453

larsxschneider commented Jul 30, 2017

ttaylorr Jul 31, 2017

larsxschneider Jul 31, 2017

ttaylorr Jul 31, 2017

ttaylorr Jul 31, 2017

ttaylorr Jul 31, 2017

larsxschneider Jul 31, 2017

ttaylorr Jul 31, 2017

ttaylorr Jul 31, 2017

larsxschneider Jul 31, 2017

ttaylorr Jul 31, 2017

larsxschneider Jul 31, 2017

ttaylorr Jul 31, 2017

larsxschneider commented Aug 2, 2017

ttaylorr commented Aug 2, 2017

larsxschneider commented Aug 21, 2017

larsxschneider commented Aug 21, 2017

technoweenie left a comment

technoweenie Aug 21, 2017

technoweenie Aug 21, 2017

larsxschneider Aug 22, 2017

add function to invoke Git with disabled LFS filters #2453

add function to invoke Git with disabled LFS filters #2453

Conversation

larsxschneider commented Jul 30, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

larsxschneider commented Aug 2, 2017

ttaylorr commented Aug 2, 2017

larsxschneider commented Aug 21, 2017

larsxschneider commented Aug 21, 2017

technoweenie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment