Incompatability with Docker 1.2.0 #90

Closed
bartmeuris opened this Issue Aug 25, 2014 · 25 comments

Projects

None yet
@bartmeuris

When I upgraded my Docker from version 1.1.2 to 1.2.0 and attempted to build a project, the cloud plugin would keep spawning new containers (which kept running), never detecting that they were active.

I ended up killing and removing about 100 containers manually, and downgrading Docker to 1.1.2 - which resolved the problem.

@maxfields2000

Can confirm, have been tracking this exact same issue this morning. It appears that the containers no longer port map port 22 correctly, and as such cannot become slaves because they are unreachable.

typically the port mapping on a running docker plugin container has: 0.0.0.0:49153->22/tcp however with docker 1.2 it just shows a: 22/tcp.

@ivey
ivey commented Aug 25, 2014

What's interesting is that when I copied the JSON payloads via tcpdump, then POSTed them with curl ... it mapped appropriately.

@hcguersoy

Same issue here. Have downgraded to 1.1.1 and works again.

@mkostrikin

After upgrade to 1.2.0 while docker plugin has cap on running instances:
WARNING: Failed to count the # of live instances on Docker
javax.ws.rs.ProcessingException: Error reading entity from input stream.
at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:868)

When cap was set to 0:
WARNING: Provisioned slave Image of jslave failed to launch
java.lang.RuntimeException: No mapped port 22 in host for SSL. Config=ContainerInspectResponse{id=aa77af7974e67ab53ec0eb04a1cc33d228fe028afb1a89a6cb3c6da5756abfd4, created=2014-08-26T10:02:56.633228801Z, path=/usr/sbin/sshd, args=....
And new containers get starting without external ssh port mapped:
b559d23421dc jsn:latest "/usr/sbin/sshd -D" 6 seconds ago Up 5 seconds 22/tcp sad_carson
27f0234912a jsn:latest "/usr/sbin/sshd -D" 16 seconds ago Up 15 seconds 22/tcp angry_fermi
2e4d4234eec jsn:latest "/usr/sbin/sshd -D" 26 seconds ago Up 25 seconds 22/tcp sad_feynman

@frosenberg

I have a similar issue with docker v1.2.0. Whenever a slave is supposed to be launched, I see this in the logs

provision
WARNING: Failed to count the # of live instances on Docker
javax.ws.rs.ProcessingException: Error reading entity from input stream.
    at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:868)
    at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:812)
    at org.glassfish.jersey.client.ClientResponse.readEntity(ClientResponse.java:309)
    at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:813)
    at org.glassfish.jersey.client.JerseyInvocation.access$600(JerseyInvocation.java:90)
    at org.glassfish.jersey.client.JerseyInvocation$3.call(JerseyInvocation.java:693)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:422)
    at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:689)
    at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:405)
    at org.glassfish.jersey.client.proxy.WebResourceFactory.invoke(WebResourceFactory.java:318)
    at com.sun.proxy.$Proxy78.inspectContainer(Unknown Source)
    at com.nirima.docker.client.DockerClient$Container.inspect(DockerClient.java:248)
    at com.nirima.jenkins.plugins.docker.DockerCloud$2.apply(DockerCloud.java:268)
    at com.nirima.jenkins.plugins.docker.DockerCloud$2.apply(DockerCloud.java:266)
    at com.google.common.collect.Iterators$7.computeNext(Iterators.java:649)
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
    at com.google.common.collect.Iterators.size(Iterators.java:151)
    at com.google.common.collect.Collections2$FilteredCollection.size(Collections2.java:222)
    at com.nirima.jenkins.plugins.docker.DockerCloud.countCurrentDockerSlaves(DockerCloud.java:272)
    at com.nirima.jenkins.plugins.docker.DockerCloud.addProvisionedSlave(DockerCloud.java:284)
    at com.nirima.jenkins.plugins.docker.DockerCloud.provision(DockerCloud.java:154)
    at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:281)
    at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51)
    at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:368)
    at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.NullPointerException) (through reference chain: com.nirima.docker.client.model.ContainerInspectResponse["NetworkSettings"]->com.nirima.docker.client.model.NetworkSettings["Ports"])
    at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:232)
    at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:197)
    at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1311)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:208)
    at com.fasterxml.jackson.databind.deser.impl.InnerClassProperty.deserializeAndSet(InnerClassProperty.java:101)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:295)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:121)
    at com.fasterxml.jackson.databind.ObjectReader._bind(ObjectReader.java:1179)
    at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:635)
    at com.fasterxml.jackson.jaxrs.base.ProviderBase.readFrom(ProviderBase.java:660)
    at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.invokeReadFrom(ReaderInterceptorExecutor.java:239)
    at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.aroundReadFrom(ReaderInterceptorExecutor.java:211)
    at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:139)
    at org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1109)
    at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:853)
    ... 35 more
Caused by: java.lang.NullPointerException
@thomassuckow
Contributor

After CoreOS updated docker to 1.2.0, now the plugin is not launching slaves properly. In my case it is filling the maximum number of instances, but they are not being added to Jenkins as slaves.

I feel like the create request is missing

"ExposedPorts":{
    "22/tcp": {}
}

But I am not familiar enough with the api to be certain.

@mastef
mastef commented Aug 31, 2014

same issue here with docker 1.2.0 and shipyard, which uses docker-py 0.3.1

the containers are launched without publishing the ports 80/tcp instead of 0.0.0.0:44034->80/tcp

are there maybe any breaking changes with Docker 1.2.0 such as removal of depreciated parameters?

@lielran
lielran commented Sep 1, 2014

i am also getting errors with CoreOS, Docker 1.2.0 ,and Jenkins 1.578

Excess workload after pending Spot instances: 1
Sep 01, 2014 7:18:54 PM INFO com.nirima.jenkins.plugins.docker.DockerCloud countCurrentDockerSlaves
Images found: [Image{repository=null, tag=null, id=f3ac335af281c1ece22f2b76f79b62c0eb9dc75e27fb4bc6fba063135b20fbfc, created=1397576253, size=0, virtualSize=436605835}]
Sep 01, 2014 7:18:54 PM INFO com.nirima.jenkins.plugins.docker.DockerCloud addProvisionedSlave
AMI Instance cap of 2 reached for ami evarga/jenkins-slave, not provisioning.
Sep 01, 2014 7:18:54 PM SEVERE com.nirima.jenkins.plugins.docker.DockerCloud$1 call
Error in provisioning; slave=null, template=DockerTemplate{image=evarga/jenkins-slave, parent=DockerCloud{name=dokcer-coreos, serverUrl=http://10.1.42.1:2375}}
Sep 01, 2014 7:19:03 PM WARNING hudson.slaves.NodeProvisioner update
Provisioned slave Image of evarga/jenkins-slave failed to launch
java.lang.RuntimeException: No mapped port 22 in host for SSL. Config=ContainerInspectResponse{id=75566d81a7315099c23eb32db7d35ceaf09f492bd969def3ec7ef32e8a26a442, created=2014-09-01T19:18:54.272045514Z, path=/usr/sbin/sshd, args=[Ljava.lang.String;@27b52a0, config=ContainerConfig{hostName=75566d81a731, portSpecs=null, user=, tty=false, stdinOpen=false, stdInOnce=false, memoryLimit=0, memorySwap=0, cpuShares=0, attachStdin=false, attachStdout=false, attachStderr=false, env=[Ljava.lang.String;@6e6d87a5, cmd=[Ljava.lang.String;@58e49a46, dns=null, image=evarga/jenkins-slave, volumes=null, volumesFrom=, entrypoint=null, networkDisabled=false, privileged=false, workingDir=, domainName=, exposedPorts={22/tcp={}}, onBuild=null}, state=ContainerState{running=true, pid=7847, exitCode=0, startedAt=2014-09-01T19:18:54.470714684Z, ghost=false, finishedAt=0001-01-01T00:00:00Z}, image=f3ac335af281c1ece22f2b76f79b62c0eb9dc75e27fb4bc6fba063135b20fbfc, networkSettings=NetworkSettings{ipAddress=10.1.0.5, ipPrefixLen=16, gateway=10.1.42.1, bridge=docker0, portMapping=null, ports=Ports{ports={}, mapping=null}}, sysInitPath=null, resolvConfPath=/var/lib/docker/containers/75566d81a7315099c23eb32db7d35ceaf09f492bd969def3ec7ef32e8a26a442/resolv.conf, volumes={}, volumesRW={}, hostnamePath=/var/lib/docker/containers/75566d81a7315099c23eb32db7d35ceaf09f492bd969def3ec7ef32e8a26a442/hostname, hostsPath=/var/lib/docker/containers/75566d81a7315099c23eb32db7d35ceaf09f492bd969def3ec7ef32e8a26a442/hosts, name=/furious_poincare, driver=btrfs}
    at com.nirima.jenkins.plugins.docker.DockerComputerLauncher.getSSHLauncher(DockerComputerLauncher.java:62)
    at com.nirima.jenkins.plugins.docker.DockerComputerLauncher.makeLauncher(DockerComputerLauncher.java:41)
    at com.nirima.jenkins.plugins.docker.DockerComputerLauncher.<init>(DockerComputerLauncher.java:37)
    at com.nirima.jenkins.plugins.docker.DockerTemplate.provision(DockerTemplate.java:206)
    at com.nirima.jenkins.plugins.docker.DockerCloud$1.call(DockerCloud.java:164)
    at com.nirima.jenkins.plugins.docker.DockerCloud$1.call(DockerCloud.java:159)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

@yongjunj
yongjunj commented Sep 2, 2014

+1 - I can confirm the issue as well.

Jenkins logs - note portMapping=null, ports=Ports{ports={}, mapping=null}}:

Sep 02, 2014 5:41:47 PM hudson.slaves.NodeProvisioner update
WARNING: Provisioned slave Image of jenkins-slave failed to launch
java.lang.RuntimeException: No mapped port 22 in host for SSL. Config=ContainerInspectResponse{id=6a1265afb5a9b477afd99b31fde24a01e5928a4a0dbbc5b38a96437bd500f64e, created=2014-09-02T17:41:38.415326418Z, path=/usr/bin/supervisord, args=[Ljava.lang.String;@591dfb79, config=ContainerConfig{hostName=6a1265afb5a9, portSpecs=null, user=, tty=false, stdinOpen=false, stdInOnce=false, memoryLimit=0, memorySwap=0, cpuShares=0, attachStdin=false, attachStdout=false, attachStderr=false, env=[Ljava.lang.String;@3f68f2a4, cmd=[Ljava.lang.String;@3120d82, dns=null, image=veblen-jenkins-slave, volumes=null, volumesFrom=, entrypoint=null, networkDisabled=false, privileged=false, workingDir=, domainName=, exposedPorts={22/tcp={}}, onBuild=null}, state=ContainerState{running=true, pid=79236, exitCode=0, startedAt=2014-09-02T17:41:38.593935975Z, ghost=false, finishedAt=0001-01-01T00:00:00Z}, image=ae26c00a0e206b81c1a4f084f57671e44c2544f53d5a0da0e42deb2a027b6dc2, networkSettings=NetworkSettings{ipAddress=172.17.0.108, ipPrefixLen=16, gateway=172.17.42.1, bridge=docker0, portMapping=null, ports=Ports{ports={}, mapping=null}}, sysInitPath=null, resolvConfPath=/var/lib/docker/containers/6a1265afb5a9b477afd99b31fde24a01e5928a4a0dbbc5b38a96437bd500f64e/resolv.conf, volumes={}, volumesRW={}, hostnamePath=/var/lib/docker/containers/6a1265afb5a9b477afd99b31fde24a01e5928a4a0dbbc5b38a96437bd500f64e/hostname, hostsPath=/var/lib/docker/containers/6a1265afb5a9b477afd99b31fde24a01e5928a4a0dbbc5b38a96437bd500f64e/hosts, name=/high_albattani, driver=aufs}
        at com.nirima.jenkins.plugins.docker.DockerComputerLauncher.getSSHLauncher(DockerComputerLauncher.java:62)
        at com.nirima.jenkins.plugins.docker.DockerComputerLauncher.makeLauncher(DockerComputerLauncher.java:41)
        at com.nirima.jenkins.plugins.docker.DockerComputerLauncher.<init>(DockerComputerLauncher.java:37)
        at com.nirima.jenkins.plugins.docker.DockerTemplate.provision(DockerTemplate.java:206)
        at com.nirima.jenkins.plugins.docker.DockerCloud$1.call(DockerCloud.java:164)
        at com.nirima.jenkins.plugins.docker.DockerCloud$1.call(DockerCloud.java:159)
        at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

Jenkins version 1.578
Docker plugin 0.7

Docker version 1.2.0:

user@server:~$ docker version
Client version: 1.2.0
Client API version: 1.14
Go version (client): go1.3.1
Git commit (client): fa7b24f
OS/Arch (client): linux/amd64
Server version: 1.2.0
Server API version: 1.14
Go version (server): go1.3.1
Git commit (server): fa7b24f
@docwhat
docwhat commented Sep 4, 2014

@magnayn Any clue what's going on? All the above are slightly different. Did docker change its API?

@ivey
ivey commented Sep 5, 2014

Probably related to docker/docker#7869

@magnayn
Member
magnayn commented Sep 5, 2014

I think it's probably they have either changed their API, or (possibly more likely) removed an item that used to be deprecated.

It's probably really simple to fix, just at this moment I don't have a lot of time to devote to it (these things likely get bumped in priority the moment someone at $WORK installs a new docker :-) )

If you're not scared of Java, you could try building the library (https://github.com/nirima/jDocker) - the tests will likely fail at some point, and give you a bigger indication of what needs fixing (and you don't need all the Jenkins stuff in order to test it).

@jlovison jlovison referenced this issue in gliderlabs/registrator Sep 6, 2014
Closed

Not recognizing containers #16

@thomassuckow
Contributor

None of the tests appear to test exposing ports. The NetCat test looks like it may have at one time but I don't think it tested correctly on 1.1.2 either.

@gaillou
gaillou commented Sep 9, 2014

A workaround is to downgrade jenkins docker plugin to 0.6.1

@thomassuckow
Contributor

Interesting, I'll have to see if I can reproduce that.

@thomassuckow
Contributor

jDocker 1.4 does successfully port forward. It appears to be the transfer-encoding:chunked. I am looking into why the new jDocker uses chunking so I can disable it temporarily as well as finding the breaking change in docker.

@thomassuckow thomassuckow referenced this issue in nirima/jDocker Sep 9, 2014
Merged

Docker 1.2 Workaround #4

@lielran
lielran commented Sep 12, 2014

Cool ! Thanks
On Sep 12, 2014 4:32 PM, "magnayn" notifications@github.com wrote:

Closed #90 #90 via
nirima/jDocker#4 nirima/jDocker#4.


Reply to this email directly or view it on GitHub
#90 (comment).

This e-mail, as well as any attached document, may contain material which
is confidential and privileged and may include trademark, copyright and
other intellectual property rights that are proprietary to Kenshoo Ltd,
its subsidiaries or affiliates ("Kenshoo"). This e-mail and its
attachments may be read, copied and used only by the addressee for the
purpose(s) for which it was disclosed herein. If you have received it in
error, please destroy the message and any attachment, and contact us
immediately. If you are not the intended recipient, be aware that any
review, reliance, disclosure, copying, distribution or use of the contents
of this message without Kenshoo's express permission is strictly prohibited.

@mamciek
Contributor
mamciek commented Sep 15, 2014

it works with DOcker 1.2 and jDocker 1.6-SNAPSHOT (installed manually)

@magnayn
Member
magnayn commented Sep 15, 2014

Cool.

I'm just rolling up some other changes, I'll cut a release some time this
week after smoke testing it in our Jenkins rig.

Thanks to all for the various diagnoses, esp. @thomassuckow for figuring
out it's the chunked issue.

On Mon, Sep 15, 2014 at 3:28 PM, Maciej Mazur notifications@github.com
wrote:

it works with DOcker 1.2 and jDocker 1.6-SNAPSHOT (installed manually)


Reply to this email directly or view it on GitHub
#90 (comment)
.

@lielran
lielran commented Sep 15, 2014

how can you work with jDocker 1.6-SNAPSHOT ?

On Mon, Sep 15, 2014 at 5:28 PM, Maciej Mazur notifications@github.com
wrote:

it works with DOcker 1.2 and jDocker 1.6-SNAPSHOT (installed manually)


Reply to this email directly or view it on GitHub
#90 (comment)
.

This e-mail, as well as any attached document, may contain material which
is confidential and privileged and may include trademark, copyright and
other intellectual property rights that are proprietary to Kenshoo Ltd,
its subsidiaries or affiliates ("Kenshoo"). This e-mail and its
attachments may be read, copied and used only by the addressee for the
purpose(s) for which it was disclosed herein. If you have received it in
error, please destroy the message and any attachment, and contact us
immediately. If you are not the intended recipient, be aware that any
review, reliance, disclosure, copying, distribution or use of the contents
of this message without Kenshoo's express permission is strictly prohibited.

@mkostrikin

recent 0.8 works with 1.2.0

@BenjaminHerbert

I can confirm docker-plugin 0.8 works with Docker 1.2.0 and Jenkins LTS ver. 1.565.3.

@constructs

I tried today with docker 1.3.1, faced a similar problem.
The docker containers started successfully from jenkins but jenkins rejected them as the there were some echos in my bashrc.
I deleted these nodes manually from Docker server.

Now, I need to delete these nodes from jenkins and I cannot delete them.

Any change to Jenkins configuration also hangs.

just before slave 082c17bc0ae9@Docker-Host gets launched ...
executing pre-launch scripts ...
[11/05/14 15:54:55] [SSH] Opening SSH connection to :49302.
[11/05/14 15:54:55] [SSH] Authentication successful.
SSH connection reports a garbage before a command execution.
Check your .bashrc, .profile, and so on to make sure it is quiet.
The received junk text is as follows:
.......
hudson.AbortException
at hudson.plugins.sshslaves.SSHLauncher.verifyNoHeaderJunk(SSHLauncher.java:854)
at hudson.plugins.sshslaves.SSHLauncher.access$100(SSHLauncher.java:134)
at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:698)
at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:691)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
[11/05/14 15:54:55] Launch failed - cleaning up connection
[11/05/14 15:54:55] [SSH] Connection closed.
Any suggestions please?

@yongjunj
yongjunj commented Nov 7, 2014

I can confirm the following setup works:

  • Docker v1.3.1
  • Jenkins v1.588
  • docker-plugin v0.8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment