Does not work with Pipeline script from SCM

[mikz]

I'm using the Pipeline Plugin. With the script directly in the configuration.
I tried to switch to "Pipeline script from SCM", configuring git as said on the wiki. But it fails:

GitHub PR #<a href="https://github.com/org/repo/pull/6504">6504</a>, Commit changed
Cloning the remote Git repository
Cloning repository git@github.com:org/repo.git
 > git init /var/lib/jenkins/jobs/system-pipeline/workspace@script # timeout=10
Fetching upstream changes from git@github.com:org/repo.git
 > git --version # timeout=10
using GIT_SSH to set credentials 
 > git -c core.askpass=true fetch --tags --progress git@github.com:org/repo.git +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url git@github.com:org/repo.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url git@github.com:org/repo.git # timeout=10
Fetching upstream changes from git@github.com:org/repo.git
using GIT_SSH to set credentials 
 > git -c core.askpass=true fetch --tags --progress git@github.com:org/repo.git +refs/pull/${GITHUB_PR_NUMBER}/${GITHUB_PR_COND_REF}:refs/remotes/origin/pull/${GITHUB_PR_NUMBER}/${GITHUB_PR_COND_REF}
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from git@github.com:org/repo.git
    at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:766)
    at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1022)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1053)
    at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109)
    at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:109)
    at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:58)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:206)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:410)
Caused by: hudson.plugins.git.GitException: Command "git -c core.askpass=true fetch --tags --progress git@github.com:org/repo.git +refs/pull/${GITHUB_PR_NUMBER}/${GITHUB_PR_COND_REF}:refs/remotes/origin/pull/${GITHUB_PR_NUMBER}/${GITHUB_PR_COND_REF}" returned status code 128:
stdout: 
stderr: fatal: Couldn't find remote ref refs/pull/${GITHUB_PR_NUMBER}/${GITHUB_PR_COND_REF}

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1719)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1463)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:63)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:314)
    at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:764)
    ... 8 more
ERROR: null
Finished: FAILURE

When I replace all ${GITHUB_PR_NUMBER}/${GITHUB_PR_COND_REF} with 6504/merge it checks out OK. So the problem might be that those variables are not available for the git checkout?

[KostyaSha]

Could you provide your pipeline config?
Pipeline is expanding variables that written in DSL code... this vars should be attached to job... and then you should have access to this vars from DSL code. Maybe try ${env.GITHUB_PR_NUMBER}?

[mikz]

Tried with ${env.}, but no luck.

Here is the Pipeline configuration:

Other than that, there is not really much in the Job. It works correctly when I write the Pipeline script in the configuration. In that script ${GITHUB_PR_NUMBER} is working correctly.

[KostyaSha]

CC @abayer that is covering workflow+community issues.

[jglick]

Ah this is just JENKINS-28447 then. The workaround would be to use an inline bootstrap script that just uses load to evaluate the real script.

[KostyaSha]

Sounds like not plugin issue. Closing.

[kad]

Too bad :( practically making pipeline jobs unusable for anything serious in production.

[oleg-nenashev]

@kad There is a hacky workaround, but yes it sucks. Seems to be stuck

[kad]

Well, this hack is not really usable. I tried to trigger pipeline job with this plugin, and try to get information about what actually triggered it inside simple static pipeline -> fail :(
I'm ok with inside pipeline to fetch actual git repo and then handover pipeline execution to script there...but at least to fetch it, in static pipeline I need to get GITHUB_PR_NUMBER or similar variables, and they are not visible :(

[KostyaSha]

If you are on newer jenkins then it sucking securitu-170 fix that you need disable.

[kad]

I am trying this on 2.5 and jenkins runs as

/usr/lib64/jvm/jre/bin/java -Djava.awt.headless=true -Dhudson.slaves.WorkspaceList=_ -Dhudson.model.ParametersAction.keepUndefinedParameters=true -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --javaHome=/usr/lib64/jvm/jre --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --httpPort=8080 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20 --prefix=/jenkins

still no luck of getting those vars visible from trigger.
I've tried either to use them directly "${GITHUB_PR_NUMBER}" or via "${env.GITHUB_PR_NUMBER}", same result, null... seems like pipeline don't consume them from trigger.

[kad]

Actually, it was mine bad. Single quotes vs. double quotes in pipeline scripts. I love snippet generator defaults :( In the end of the day, workaround was done via something like this: https://gist.github.com/kad/4a5a8f669d4a4090b43be1f1c5461df3
Not nice, but works. To get it better, it would be great to get few more parameters exported by this plugin: #94

About arguments and SECURITY-170, maybe it's possible to do something similar as in Gerrit trigger
jenkinsci/gerrit-trigger-plugin/pull/285 ? (see their changes in EventListener.java)

[KostyaSha]

@kad as you see gerrit change requires core change, that would require the newest jenkins core version and cut a lot of users. SECURITY-170 is not a security fix, that just sucking/breakable/not-designed/unreviewed change (such things happening in jenkinsci project periodically). I wouldn't do anything until normal API would be available for the most number of jenkins users.
Trigger shouldn't use any Listeners without any real reasons (:atm: pr trigger has only one listener because of bad APIs in git-plugin).

[kad]

then probably add something about that to https://wiki.jenkins-ci.org/display/JENKINS/Plugins+affected+by+fix+for+SECURITY-170 ?

[KostyaSha]

I don't care about this list, i expect they broke ALL trigger plugins, filling such list is stupid imho.

[KostyaSha]

@kad sorry, how did you edit wiki page without permissions?

[kad]

well, in your repository you have default settings for public repos. which means "Restrict editing to collaborators only" checkbox is not checked, thus any logged in users can edit your wiki pages.

[KostyaSha]

yes, i found :) was really surprised. I created special repo for accepting wiki changes https://github.com/KostyaSha/github-integration-plugin-wiki

[kad]

I think no one knows about repository. If you'd like to have changes for wiki page going via that repo, probably say about that on first page of wiki.

[KostyaSha]

Sure, created when somebody wanted contribute, will update docs later...

[kad]

BTW, 2.6 just released that has way for trigger plugins to whitelist their environment variables. ghprb seems to be already have corresponding fix for that.

[KostyaSha]

ghprb has no fix, that's crap.