[JENKINS-35687] Add simple git lfs support

[matthauck]

Note: I had to do some minor changes to pom.xml to make this build with changes for jenkinsci/git-client-plugin#206, so it may well not pass the CI build...

[MarkEWaite]

How does the user tell the plugin to enable LFS support for a specific job? I don't see any user interface change in the plugin and can't understand what calls this new code if there is no user interface enabling it for the job.

Ah, I wasn't attentive enough. I assume that because it is a GitSCMExtension, the "Enable Git LFS support" will appear as a drop down choice in the "Additional Behaviours" section.

[matthauck]

Yeah, was just about to add that it is an additional behavior. I can add a help.html, however, which would probably make it more useful.

[HRMPW]

Is this stalled or something that might still make it into the git plugin. Also, the most recent version of jgit has support for lfs but it doesn't look like it is being pulled in.

[MarkEWaite]

It is something that might still make it into the git plugin, but not before my hoped September 2016 release of git plugin 3.0.0 and git client plugin 2.0.0. Those releases are planned to be in time for Jenkins World 2016.

Git plugin 3.0.0 adds submodule authentication support and drops support for Java 6. It requires at least Jenkins 1.625. It has been in beta test in various incarnations for almost a year.

Git client plugin 2.0.0 adds submodule authentication support, drops support for Java 6, and upgrades the JGit implementation from JGit 3 to JGit 4. It requires at least Jenkins 1.625. It has been in beta test in various incarnations for almost a year.

The git client plugin release is particularly substantial because the JGit API changes from JGit 3 to JGit 4 will break users of plugins which depend on the JGit implementation provided by the git client plugin. You can read the jenkins-developers mailing list posting for more details.

[matthauck]

Glad to hear that lfs support may yet have a future here! 😅

[HRMPW]

@MarkEWaite I hope JW went well for you. There was quite a of activity at the booth. Now that 3.0 has been released I thought I might ask about LFS support again. I have had several people ask me about this feature in Jenkins.

[MarkEWaite]

I have several other reviews in the queue before this one, so it will likely be a month or more before the reviews are complete. One Jira user has started evaluating this pull request (and the matching pull request for the git client plugin). If others can do the same evaluation of a pre-release build, it makes me more confident in the change, and reduces the amount of verification that I need to do.

[pyther]

I am extremely interested in this pull request. git-lfs support is Jenkins is something that my organization needs like yesterday. I attempted to build both the git plugin with a patched git-client plugin, but got build errors that I could not figure out. I'd be glad to test on my instance if I can get some help building the plugin.

[MarkEWaite]

I've built a version of the git client plugin which includes the proposed lfs change and a version of the git plugin which includes the proposed change. Since I'm not a git lfs user, I'll need to rely on others to decide if the functionality is sufficiently rich to justify including it in the plugin

[zoomequipd]

Having been suffering from builds hanging (i assume waiting for auth) in LFS enabled repos -
I pulled down and installed the git-client and git plugin linked to by @MarkEWaite [https://github.com//pull/414#issuecomment-258318580]

added the "GIT LFS pull after checkout" option and it errored out - it is this actually trying to do a git lfs pull before it does the checkout?

00:13:54 Started by user Brandon Murphy
00:13:54 Building in workspace /var/lib/jenkins/workspace/website
00:13:54 [WS-CLEANUP] Deleting project workspace...
00:13:54 [WS-CLEANUP] Done
00:13:54 Cloning the remote Git repository
00:13:54 Using shallow clone
00:13:54 Cloning repository https://bitbucket.org/secdsm/website.git
00:13:54  > git init /var/lib/jenkins/workspace/website # timeout=10
00:13:54 Fetching upstream changes from https://bitbucket.org/secdsm/website.git
00:13:54  > git --version # timeout=10
00:13:54 using GIT_ASKPASS to set credentials 
00:13:54  > git fetch --tags --progress https://bitbucket.org/secdsm/website.git +refs/heads/*:refs/remotes/origin/* --depth=1
00:13:57  > git config remote.origin.url https://bitbucket.org/secdsm/website.git # timeout=10
00:13:57  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
00:13:57  > git config remote.origin.url https://bitbucket.org/secdsm/website.git # timeout=10
00:13:57 Enabling Git LFS pull
00:13:57 Fetching upstream changes from https://bitbucket.org/secdsm/website.git
00:13:57 using GIT_ASKPASS to set credentials 
00:13:57  > git fetch --tags --progress https://bitbucket.org/secdsm/website.git +refs/heads/*:refs/remotes/origin/* --depth=1
00:14:00 using GIT_ASKPASS to set credentials 
00:14:00  > git lfs pull
00:14:00 ERROR: Error fetching remote repo 'origin'
00:14:00 hudson.plugins.git.GitException: Failed to fetch from https://bitbucket.org/secdsm/website.git
00:14:00    at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:803)
00:14:00    at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1063)
00:14:00    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1094)
00:14:00    at hudson.scm.SCM.checkout(SCM.java:495)
00:14:00    at hudson.model.AbstractProject.checkout(AbstractProject.java:1278)
00:14:00    at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
00:14:00    at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
00:14:00    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
00:14:00    at hudson.model.Run.execute(Run.java:1720)
00:14:00    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
00:14:00    at hudson.model.ResourceController.execute(ResourceController.java:98)
00:14:00    at hudson.model.Executor.run(Executor.java:404)
00:14:00 Caused by: hudson.plugins.git.GitException: Command "git lfs pull" returned status code 2:
00:14:00 stdout: 
00:14:00 stderr: Could not pull
00:14:00 
00:14:00 Errors logged to /var/lib/jenkins/workspace/website/.git/lfs/objects/logs/20161113T001400.052039.log
00:14:00 Use `git lfs logs last` to view the log.
00:14:00 
00:14:00    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1758)
00:14:00    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1502)
00:14:00    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:64)
00:14:00    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:327)
00:14:00    at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:801)
00:14:00    ... 11 more
00:14:00 ERROR: null
00:14:00 [WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] Skipped based on build state FAILURE
00:14:00 Finished: FAILURE

`

the log from 'git lfs logs last' shows

git-lfs/1.4.4 (GitHub; linux amd64; go 1.7.3)
git version 2.7.4

$ git-lfs pull
Could not pull

Git can't resolve ref: "HEAD"

ENV:
[snip]

[matthauck]

Hmm. Interesting, it should be doing the lfs pull after the checkout so that it pulls the files for the right branch...

[estyrke]

@matthauck Since the decorator is added on the fetch command I guess it will happen before the checkout. Would it be better to do a two-step process of adding "git lfs fetch" to the Fetch command and "git lfs checkout" to the Checkout command? It seems like it would solve this problem and also looks more symmetrical to me.

I made a try at this, working from your existing modifications:
jenkinsci/git-client-plugin@master...estyrke:git-lfs
master...estyrke:git-lfs

However, I did not manage to get a build environment working, so these changes are completely untested. Feel free to use them if you like.

[zoomequipd]

@estyrke - If you can compile the packages into a SNAPSHOT* for me, (sorry, I'm clueless how to do that) I can test it out.
EDIT - called it a SNAPSHOT

[matthauck]

@estyrke, I'm not entirely available at present to test that, we have something working now that I don't really want to / can change rihgt now. But your approach seems like the right way to do it. I'd suggest opening a PR and going with that solution? We can just close this one.

[MarkEWaite]

I'm closing this pull request so that I can focus my evaluation efforts on #463 .

[matthauck]

👍