Jenkins pipeline - smudge filter lfs failed

[GouravIN]

Getting the following error during my git lfs after checkout step.

Started by user Gourav Pattnaik
Checking out git git@github.com:XXXXX into /var/jenkins_home/jobs/in-ci-pipeline-dev/workspace@script to read Jenkinsfile
using credential XXXXXX

git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url git@github.com:XXXXXX # timeout=10
Fetching upstream changes from git@github.com:XXXXXX
git --version # timeout=10
using GIT_SSH to set credentials XXXXXXXXX SSH user
git fetch --tags --progress -- git@github.com:XXXXXX +refs/heads/:refs/remotes/origin/
git rev-parse XXXXX^{commit} # timeout=10
git rev-parse refs/remotes/origin/XXXXXXX^{commit} # timeout=10
Checking out Revision 438b9cbb48f1c2cceeb54300753bb608dccc24c6 (refs/remotes/origin/XXXXXXXX)
git config core.sparsecheckout # timeout=10
git checkout -f 438b9cbb48f1c2cceeb54300753bb608dccc24c6
ERROR: Checkout failed
hudson.plugins.git.GitException: Command "git checkout -f 438b9cbb48f1c2cceeb54300753bb608dccc24c6" returned status code 128:
stdout:
stderr: Downloading app-new/config/browscap.ini (88 MB)
Error downloading object: app-new/config/browscap.ini (ba81f69): Smudge error: Error downloading app-new/config/browscap.ini (ba81f6956fa350f2837ec9d3b61842719e8b984dbec3fd54140b66df0cf7e13d): batch request: Repository not found.: exit status 1

Errors logged to /var/jenkins_home/jobs/in-ci-pipeline-dev/workspace@script/.git/lfs/logs/20200914T144748.824480105.log
Use git lfs logs last to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: app-new/config/browscap.ini: smudge filter lfs failed

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2174)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$1000(CliGitAPIImpl.java:78)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:2473)

Caused: hudson.plugins.git.GitException: Could not checkout 438b9cbb48f1c2cceeb54300753bb608dccc24c6
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:2497)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1242)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)
at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:155)
at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:69)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:309)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
ERROR: Maximum checkout retry attempts reached, aborting
Finished: FAILURE

[bk2204]

Hey,

Sorry to hear you're having trouble. What kind of SSH key are you using to perform this clone (personal or deploy key)? Does it work if you don't enable sparse checkout? What OS is this on, and what version of Git LFS are you using (git lfs env)? What is GIT_SSH set to? Does it use OpenSSH under the hood or something different?

[GouravIN]

@bk2204
root@jenkins:/# git lfs env
git-lfs/2.10.0 (GitHub; linux amd64; go 1.13.4)
git version 2.11.0

OS: Debian
Does it work if you don't enable sparse checkout? No
What kind of SSH key are you using to perform this clone (personal or deploy key)? Personal, we created a set of private and public key and are using it.
What is GIT_SSH set to? Does it use OpenSSH under the hood or something different? Using the SSH method of Github. Yes OpenSSH.

[bk2204]

Can you show us the program you're using for GIT_SSH? I'm wondering if we're failing to handle some invocation with that properly and that might be the cause of the issues you're seeing.

[GouravIN]

@bk2204 I'm using the Git plugin of Jenkins. Below is my code.

def checkoutStep(branchname, repo, targetdir = "") {
checkout changelog: false, poll: false,
scm: [$class : 'GitSCM',
branches : [[name: "${branchname}"]],
browser : [$class: 'GithubWeb', repoUrl: "XXXXXXXX"],
doGenerateSubmoduleConfigurations: false,
extensions : [[$class: 'CleanCheckout'],
[$class: 'CleanBeforeCheckout'],
[$class: 'RelativeTargetDirectory', relativeTargetDir: "${targetdir}"],
[$class: 'CloneOption', depth: 0, noTags: true, reference: '', shallow: false],
[$class: 'GitLFSPull']],
submoduleCfg : [],
userRemoteConfigs : [[credentialsId: 'XXXXXXXXX', url: "git@github.com:XXXXXXX"]]]
}

[bk2204]

Hey,

I'm not sure what's going on here, but it seems like you're seeing some sort of authentication problem with GitHub using this key. It's not clear to me what the problem is, though. If you can create a set of reproduction steps as a shell script, it should be possible for us to look at this to see if it's a Git LFS issue, but I don't know anything about Jenkins and can't help debug the problem there.

[GouravIN]

@bk2204 I will recreate the steps and get back to you.

[GouravIN]

@bk2204 So far this is the thing I have figured out.

The authentication to the Github Repo works, but it doesn't work with GITLFS both via SSH and HTTPS method.

Checking out git https://github.com/XXXXXXX into /var/jenkins_home/jobs/in-ci-pipeline-dev/workspace@script to read Jenkinsfile
using credential XXXXXXXXXX

git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/XXXXXXX# timeout=10
Fetching upstream changes from https://github.com/XXXXXXX
git --version # timeout=10
using GIT_ASKPASS to set credentials XXXXXXXX credentials for the web
git fetch --tags --progress -- https://github.com/XXXXXXXX +refs/heads/:refs/remotes/origin/
git rev-parse dev/in-ci-jenkinsfile^{commit} # timeout=10
git rev-parse refs/remotes/origin/dev/in-ci-jenkinsfile^{commit} # timeout=10
Checking out Revision ffc2f625856285fe418ee344c46291fbb6c03b35 (refs/remotes/origin/dev/in-ci-jenkinsfile)
git config core.sparsecheckout # timeout=10
git checkout -f ffc2f625856285fe418ee344c46291fbb6c03b35
ERROR: Checkout failed
hudson.plugins.git.GitException: Command "git checkout -f ffc2f625856285fe418ee344c46291fbb6c03b35" returned status code 128:
stdout:
stderr: Downloading app-new/config/browscap.ini (88 MB)
Error downloading object: app-new/config/browscap.ini (ba81f69): Smudge error: Error downloading app-new/config/browscap.ini (ba81f6956fa350f2837ec9d3b61842719e8b984dbec3fd54140b66df0cf7e13d): batch response: Bad credentials

From the stderr output: You can see here that it has already got inside the repo, and when it is trying to pull the file which is of 88 MB using gitlfs, it throws error.

Via: HTTPS it throws Bad Credential error
Via: SSH it throws repo not found error.

And the most strange thing is that, this only happens when I'm using the Jenkinsfile in a pipeline, but when I use it a scripted method, it works completely fine.

[bk2204]

Hey,

I don't know why this is happening, but I suppose it's possible that you're setting GIT_ASKPASS or GIT_SSH only for the fetch and not for the checkout. If so, that won't work, because Git LFS won't be invoked as part of the fetch, but as part of the checkout. Beyond that, I don't have any ideas about why this isn't working. If you can't reproduce this with a shell script, I'd reach out to the Jenkins folks and ask them what's going on.

[GouravIN]

@bk2204 You can close this issue. I have figured out that it was an issue from Jenkins side. Which I have fixed it.

[jeffreyssmith2nd]

Sorry to revive a closed thread that's not exactly related to this repo, but @GouravIN can you share what you changed in Jenkins to make things work?

[GouravIN]

@jeffreyssmith2nd The GitLFS option was not enabled on the job config. So when it was trying to download the files it was failing.

[jeffreyssmith2nd]

Thanks @GouravIN, we ended up solving our issue by following the recommendation here and setting GIT_LFS_SKIP_SMUDGE=1

[Ezward]

I know I am commenting is a closed issue, but it is worth noting for others that this workaround means that the initial checkout will not checkout your LFS files. So that is not exactly a fix.