version 1.0.17: first call of OpenShiftDSL function, e.g. raw or process, fails with java.io.IOException: Cannot run program "oc" #180

TomSchmitz · 2018-09-19T14:00:55Z

Hello,

after updating jenkins-client-plugin version from 1.0.16 to 1.0.17 any OpenShiftDSL function will fail with a java.io.IOException: Cannot run program "oc" (in directory ...): error=2, No such file or directory.

Calling withCluster() and withProject() works just fine, but any other OpenShiftDSL function, e.g. raw or process, will fail.

You can find the full stacktrace in the attached text file.

If you need any further info, let me know.
Thank you for your help.
Tom
stacktraceIOException.txt

gomesp · 2018-09-20T02:25:43Z

@TomSchmitz Does your Jenkins instance have the oc CLI installed and in the PATH?
What is the OS that Jenkins is running on? I know that there's an issue with Alpine Linux that even if oc is in the PATH it won't execute it because it cannot load the libraries dynamically (oc used to have its libraries statically linked before).

TomSchmitz · 2018-09-20T07:33:48Z

hello @gomesp,

our jenkins instance is running on OS RHEL 6. We have several oc CLIs installed and configured via the jenkins-client-plugin, see attached file. We don't set the oc CLI in the PATH explicity.

We just switch to version 1.0.16 again and it is running successfully without configuring anything special for the oc CLIs.

fshehadeh · 2018-09-21T22:21:33Z

@gomesp We saw a similar issue when upgrading from 1.0.13 to 1.0.17. We configured Jenkins with multiple version of OC and we use code like the following to update the PATH environment variable:
` def ocDir = tool ocVersion
echo "Using OC version: ${ocVersion}, at: ${ocDir}"

	this.script.withEnv(["PATH+OC=${ocDir}"]) {
		// verify the actual project name
		def actualProjectName = openshift.project()
		echo "Using Openshift project: ${actualProjectName}"

		// do something with it
	}

`
I verified using 'echo' that the PATH environment was updated inside the withEnv block, but I don't think it is propagating to the actual call for the "oc" command. Rolling back to 1.0.13 fixed the issue. I think this has to do with the recent change to how the "oc" command is invoked (used to rely on BourneShellScript, which might have been aware of the PATH environment)

TomSchmitz · 2018-09-24T06:47:34Z

@gomesp , in order to confirm @fshehadeh's post, we are using a similar code to define oc tool and PATH:
`
def ocDir = steps.tool ocTool

steps.withEnv(["PATH+OC=${ocDir}"]) {

//do something with OpenShiftDSL

}
`

vutkin · 2018-09-24T11:56:38Z

I have the same issue.

gabemontero · 2018-09-24T18:34:12Z

I think the env propagation noted here is correct, with the referenced change the culprit. I also think it serves as the root cause for #177

I'll be looking and fixing the env propagation with the new means of launching. If it proves untenable I'll revert.

@bparees @vfreex fyi ^^

gabemontero · 2018-09-25T00:20:04Z

It is not quite ready yet, but I believe I have a path to get the envs, including a PATH+OC, with the new form of process launching.

I'll update again tomorrow after a few more cycles or work.

vfreex · 2018-09-25T04:47:41Z

@gabemontero I haven't taken a look yet, but I think I missed the part of locating oc executable from the overriden PATH environment variable when getting rid of using bash in the new implementation of process launching.

Do you have any findings? Will you fix it or I will do the fix?

elbaer · 2018-09-25T11:11:46Z

I can confirm this issue too. I used the following script for my test:

script {
    def ocDir = tool "oc"
    withEnv(["PATH+OC=${ocDir}/bin:$PATH"]) {
        openshift.withCluster("my-ose-prod") {
            openshift.withCredentials("my-project-sa-token") {
                openshift.withProject("my-project") {
                    println "PATH: ${PATH}"
                    println "OC Version from Shell:"
                    sh "oc version"
                    println "OC Version from Plugin:"
                    println openshift.raw('version').out
                }
            }
        }
    }
}

It generated the following output:

[Pipeline] {
[Pipeline] stage
[Pipeline] { (build)
[Pipeline] script
[Pipeline] {
[Pipeline] tool
[oc] $ sh -e /var/lib/jenkins/workspace/tools/com.cloudbees.jenkins.plugins.customtools.CustomTool/oc/hudson4890815691766962028.sh
[Pipeline] withEnv
[Pipeline] {

[Pipeline] echo

[Pipeline] _OcContextInit
[Pipeline] _OcContextInit
[Pipeline] _OcContextInit
[Pipeline] echo
PATH: /var/lib/jenkins/workspace/tools/com.cloudbees.jenkins.plugins.customtools.CustomTool/oc/root/bin:/usr/local/bin:/usr/bin:/usr/local/bin:/usr/bin
[Pipeline] echo
OC Version from Shell:
[Pipeline] sh

[test-oc-client] Running shell script
+ oc version
oc v3.7.44
kubernetes v1.7.6+a08f5eeb62
features: ***
[Pipeline] echo
OC Version from Plugin:
[Pipeline] _OcAction
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
java.io.IOException: error=2, No such file or directory
...

It looks like the modified PATH variable is not passed to the plugin?

vfreex · 2018-09-25T12:39:39Z

@elbaer Yes, the modified environment variable given in withEnv is not handed in the new process launching mechanism.

gabemontero · 2018-09-25T13:52:54Z

I'm close to having a fix @vfreex but I'm going to tag you in the PR for review, to cross reference with the prior situations you were trying to address. Hopefully the common workarounds I've found to properly set up ProcessBuilder to not interfere with your prior fix.

gabemontero · 2018-09-25T17:50:20Z

I have a PR up that fixes my reproducer

gabemontero · 2018-09-28T15:05:55Z

v1.0.18 of the plugin has been initiated with the jenkins update center

kajh · 2019-02-11T06:34:51Z

I see the issue with v1.0.24 on jenkins 2.163. I havn't tried on earlier versions. It looks like the PATH is not passed to the plugin. sh "oc version" works fine, but not openshift.raw('version').

oc is installed under /home/jenkins/openshift-origin-client-tools.

node('mynode') {
    def ocDir = tool name: 'oc3.11'
    echo "dir " + ocDir
    withEnv(["PATH+OC=${ocDir}"]) {
        openshift.withCluster('https://myhost:8443/', 'my-token') {
            openshift.withProject( 'myproject' ) {
                    println "PATH: ${PATH}"
                    println "OC Version from Shell:"
                    sh "oc version"
                    println "OC Version from Plugin:"
                    println openshift.raw('version').out
            }
        }
    }
}

gives the following output:

[Pipeline] {
[Pipeline] tool
[Pipeline] echo
dir /home/jenkins/openshift-origin-client-tools
[Pipeline] withEnv
[Pipeline] {
[Pipeline] echo

[Pipeline] _OcContextInit
[Pipeline] _OcContextInit
[Pipeline] echo
Hello from project myproject in cluster https://myhost:8443/
[Pipeline] echo
PATH: /home/jenkins/openshift-origin-client-tools:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin
[Pipeline] echo
OC Version from Shell:
[Pipeline] sh
+ oc version
oc v3.11.0+0cbc58b
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO
[Pipeline] echo
OC Version from Plugin:
[Pipeline] _OcAction
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
java.io.IOException: error=2, No such file or directory
	at java.lang.UNIXProcess.forkAndExec(Native Method)
	at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
	at java.lang.ProcessImpl.start(ProcessImpl.java:134)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to wapp-jenkins05
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:957)
		at hudson.Launcher$RemoteLauncher.launch(Launcher.java:1059)
		at hudson.Launcher$ProcStarter.start(Launcher.java:455)
		at com.openshift.jenkins.plugins.pipeline.OcAction$Execution.run(OcAction.java:182)
		at com.openshift.jenkins.plugins.pipeline.OcAction$Execution.run(OcAction.java:125)
		at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
		at hudson.security.ACL.impersonate(ACL.java:290)
		at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
Caused: java.io.IOException: Cannot run program "oc" (in directory "/home/jenkins/workspace/kajhtest/myproject-pipeline"): error=2, No such file or directory
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
	at hudson.Proc$LocalProc.<init>(Proc.java:249)
	at hudson.Proc$LocalProc.<init>(Proc.java:218)
	at hudson.Launcher$LocalLauncher.launch(Launcher.java:936)
	at hudson.Launcher$ProcStarter.start(Launcher.java:455)
	at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1313)
	at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1266)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

kajh · 2019-02-11T16:49:14Z

I see that the issue I describe here is the same as #224. Sorry for not noticing that issue!

gabemontero mentioned this issue Sep 24, 2018

Escaping parameters with spaces #177

Closed

gabemontero mentioned this issue Sep 25, 2018

Fix env propagation #182

Merged

openshift-merge-robot closed this as completed in #182 Sep 28, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

version 1.0.17: first call of OpenShiftDSL function, e.g. raw or process, fails with java.io.IOException: Cannot run program "oc" #180

version 1.0.17: first call of OpenShiftDSL function, e.g. raw or process, fails with java.io.IOException: Cannot run program "oc" #180

TomSchmitz commented Sep 19, 2018 •

edited

gomesp commented Sep 20, 2018

TomSchmitz commented Sep 20, 2018

fshehadeh commented Sep 21, 2018

TomSchmitz commented Sep 24, 2018 •

edited

vutkin commented Sep 24, 2018

gabemontero commented Sep 24, 2018

gabemontero commented Sep 25, 2018

vfreex commented Sep 25, 2018 •

edited

elbaer commented Sep 25, 2018 •

edited

vfreex commented Sep 25, 2018

gabemontero commented Sep 25, 2018

gabemontero commented Sep 25, 2018

gabemontero commented Sep 28, 2018

kajh commented Feb 11, 2019 •

edited

kajh commented Feb 11, 2019

version 1.0.17: first call of OpenShiftDSL function, e.g. raw or process, fails with java.io.IOException: Cannot run program "oc" #180

version 1.0.17: first call of OpenShiftDSL function, e.g. raw or process, fails with java.io.IOException: Cannot run program "oc" #180

Comments

TomSchmitz commented Sep 19, 2018 • edited

gomesp commented Sep 20, 2018

TomSchmitz commented Sep 20, 2018

fshehadeh commented Sep 21, 2018

TomSchmitz commented Sep 24, 2018 • edited

vutkin commented Sep 24, 2018

gabemontero commented Sep 24, 2018

gabemontero commented Sep 25, 2018

vfreex commented Sep 25, 2018 • edited

elbaer commented Sep 25, 2018 • edited

vfreex commented Sep 25, 2018

gabemontero commented Sep 25, 2018

gabemontero commented Sep 25, 2018

gabemontero commented Sep 28, 2018

kajh commented Feb 11, 2019 • edited

kajh commented Feb 11, 2019

TomSchmitz commented Sep 19, 2018 •

edited

TomSchmitz commented Sep 24, 2018 •

edited

vfreex commented Sep 25, 2018 •

edited

elbaer commented Sep 25, 2018 •

edited

kajh commented Feb 11, 2019 •

edited