Jenkins connection issue on test-osuosl-ubuntu14-ppc64_be-2

[gibfahn]

Fixed by restarting the Jenkins service with

service jenkins stop
service jenkins start

https://ci.nodejs.org/job/node-test-commit-plinux/5208/nodes=ppcbe-ubuntu1404/console

[EnvInject] - Loading node environment variables.
Building remotely on test-osuosl-ubuntu14-ppc64_be-2 (ppcbe-ubuntu1404) in workspace /home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcbe-ubuntu1404
java.io.IOException: remote file operation failed: /home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcbe-ubuntu1404 at hudson.remoting.Channel@57fe741f:test-osuosl-ubuntu14-ppc64_be-2: java.io.IOException: Remote call on test-osuosl-ubuntu14-ppc64_be-2 failed
	at hudson.FilePath.act(FilePath.java:1014)
	at hudson.FilePath.act(FilePath.java:996)
	at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:137)
	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:739)
	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:731)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1076)
	at hudson.scm.SCM.checkout(SCM.java:495)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1278)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
	at hudson.model.Run.execute(Run.java:1720)
	at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:404)
Caused by: java.io.IOException: Remote call on test-osuosl-ubuntu14-ppc64_be-2 failed
	at hudson.remoting.Channel.call(Channel.java:805)
	at hudson.FilePath.act(FilePath.java:1007)
	... 14 more
Caused by: java.lang.OutOfMemoryError: PermGen space
	at sun.misc.Unsafe.defineClass(Native Method)
	at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
	at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399)
	at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:396)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:395)
	at sun.reflect.MethodAccessorGenerator.generateSerializationConstructor(MethodAccessorGenerator.java:113)
	at sun.reflect.ReflectionFactory.newConstructorForSerialization(ReflectionFactory.java:331)
	at java.io.ObjectStreamClass.getSerializableConstructor(ObjectStreamClass.java:1420)
	at java.io.ObjectStreamClass.access$1500(ObjectStreamClass.java:72)
	at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:497)
	at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)
	at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)
	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:598)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1630)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1630)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1779)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2016)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1940)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1806)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
	at hudson.remoting.UserRequest.deserialize(UserRequest.java:216)
	at hudson.remoting.UserRequest.perform(UserRequest.java:130)
	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
	at hudson.remoting.Request$2.run(Request.java:332)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
	at ......remote call to test-osuosl-ubuntu14-ppc64_be-2(Native Method)
	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1433)
	at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
	at hudson.remoting.Channel.call(Channel.java:797)
	... 15 more
Run condition [Always] enabling perform for step [[Execute a set of scripts, Execute a set of scripts]]

[jbergstroem]

Suggesting you bump the ram from 128m to 192m and see if it changes anything.

[mhdawson]

I did not look at the reason, as its already 8PM my time but I had to restart the service on test-osuosl-ubuntu14-ppc64_be-1, test-osuosl-ubuntu14-ppc64_le-1 and test-osuosl-ubuntu14-ppc64_be-2

We'll have to dig deeper if this is required again as we've not had any issues like this for a long time.

[phillipj]

Seeing this again today on test-osuosl-ubuntu14-ppc64_be-1.

Refs https://ci.nodejs.org/job/node-test-commit-plinux/6073/nodes=ppcbe-ubuntu1404/console

[phillipj]

FYI just bumped test-osuosl-ubuntu14-ppc64_be-1 to 256mb and restarted Jenkins service.

[mhdawson]

Thanks for the heads, up, lets watch for a few days. If this seems to have resolved it I'll go ahead an d update the ansible configs.

[mhdawson]

Ok have not seen issues since change, created #605 to modify ansible configs.