New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI failure: acceptance test #906

Closed
olix0r opened this Issue Jan 9, 2017 · 1 comment

Comments

Projects
None yet
2 participants
@olix0r
Member

olix0r commented Jan 9, 2017

Recently, we've seen a lot of transient CI failures, which slows everything down:

I 0109 12:46:02.442 THREAD158: linkerd 0.8.5 (rev=7226e5003076a20faf9927f8b60954ed7d760e68) built at 20170109-124550
I 0109 12:46:02.600 THREAD158: Finagle version 6.41.0 (rev=95eedf5f41f78414fae25d93cc8fae02eeb5a75d) built at 20161220-164342
I 0109 12:46:06.384 THREAD158: tracer: io.buoyant.linkerd.tracer.ZipkinConfig$$anon$1@573ae685
I 0109 12:46:06.384 THREAD158: tracer: io.buoyant.telemetry.TracelogTracer@6e3c709b
I 0109 12:46:06.484 THREAD158: Tracer: com.twitter.finagle.zipkin.thrift.ScribeZipkinTracer
I 0109 12:46:06.652 THREAD158: Resolver[inet] = com.twitter.finagle.InetResolver(com.twitter.finagle.InetResolver@82c9154)
I 0109 12:46:06.658 THREAD158: Resolver[fixedinet] = com.twitter.finagle.FixedInetResolver(com.twitter.finagle.FixedInetResolver@4c61fd2f)
I 0109 12:46:06.660 THREAD158: Resolver[neg] = com.twitter.finagle.NegResolver$(com.twitter.finagle.NegResolver$@51a9a13e)
I 0109 12:46:06.661 THREAD158: Resolver[nil] = com.twitter.finagle.NilResolver$(com.twitter.finagle.NilResolver$@3c096de4)
I 0109 12:46:06.661 THREAD158: Resolver[fail] = com.twitter.finagle.FailResolver$(com.twitter.finagle.FailResolver$@50c3ed00)
I 0109 12:46:06.662 THREAD158: Resolver[flag] = com.twitter.server.FlagResolver(com.twitter.server.FlagResolver@1388fab2)
I 0109 12:46:06.663 THREAD158: Resolver[zk] = com.twitter.finagle.zookeeper.ZkResolver(com.twitter.finagle.zookeeper.ZkResolver@f3409ac)
I 0109 12:46:06.664 THREAD158: Resolver[zk2] = com.twitter.finagle.serverset2.Zk2Resolver(com.twitter.finagle.serverset2.Zk2Resolver@78a523e3)
I 0109 12:46:07.229 THREAD158: serving http admin on /0.0.0.0:9990
I 0109 12:46:07.249 THREAD158: serving int on /0.0.0.0:4140
I 0109 12:46:07.301 THREAD158: announcing /10.0.4.1:4140 as /#/io.l5d.serversets/foo/bar to zk-serversets
I 0109 12:46:07.441 THREAD158: announcing /172.17.145.128:4140 as /#/io.l5d.serversets/foo/bar to zk-serversets
I 0109 12:46:07.453 THREAD158: serving /host/thrift-framed on /0.0.0.0:4141
I 0109 12:46:07.658 THREAD158: serving /host/thrift-buffered on /0.0.0.0:4142
I 0109 12:46:07.694 THREAD158: serving /host/mux on /0.0.0.0:4143
I 0109 12:46:07.885 THREAD158: serving tlsStatic on /0.0.0.0:4144
I 0109 12:46:07.905 THREAD158: serving tlsBoundPath on /0.0.0.0:4145
I 0109 12:46:07.913 THREAD158: initialized
WARN 0109 12:46:08.042 ZkAnnouncer Mutator-SendThread(localhost:2181): Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:356)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1192)
WARN 0109 12:46:09.151 ZkAnnouncer Mutator-SendThread(localhost:2181): Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:356)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1192)
127.0.0.1 - - [09/Jan/2017 12:46:09] "GET / HTTP/1.1" 200 -
E 0109 12:46:09.195 THREAD160: service failure
com.twitter.finagle.IndividualRequestTimeoutException: exceeded 1.seconds to 0.0.0.0/4140 while waiting for a response for an individual request, excluding retries. Remote Info: Not Available

HTTP REQUEST FAILED
I 0109 12:46:09.321 THREAD160 TraceId:72c750538cac31ff: FailureAccrualFactory marking connection to "$/inet/127.1/9999" as dead. Remote Address: Inet(/127.0.0.1:9999,Map())
127.0.0.1 - - [09/Jan/2017 12:46:09] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [09/Jan/2017 12:46:09] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [09/Jan/2017 12:46:09] "GET / HTTP/1.1" 200 -
E 0109 12:46:09.892 THREAD160 TraceId:72c750538cac31ff: service failure
Failure(1.seconds, flags=0x03) with RemoteInfo -> Upstream Address: /127.0.0.1:40756, Upstream Client Id: Not Available, Downstream Address: /127.0.0.1:9999, Downstream Client Id: $/inet/127.1/9999, Trace Id: 72c750538cac31ff.72c750538cac31ff<:72c750538cac31ff
Caused by: com.twitter.util.TimeoutException: 1.seconds
	at com.twitter.util.Future$$anonfun$within$1.apply(Future.scala:992)
	at com.twitter.util.Future$$anonfun$within$1.apply(Future.scala:992)
	at com.twitter.util.Future$$anonfun$2.apply$mcV$sp(Future.scala:1039)
	at com.twitter.util.Monitor$$anonfun$apply$1.apply$mcV$sp(Monitor.scala:38)
	at com.twitter.util.Monitor$$anonfun$apply$1.apply(Monitor.scala:38)
	at com.twitter.util.Monitor$$anonfun$apply$1.apply(Monitor.scala:38)
	at com.twitter.util.Monitor$$anonfun$using$1.apply(Monitor.scala:110)
	at com.twitter.util.Monitor$.restoring(Monitor.scala:117)
	at com.twitter.util.Monitor$.using(Monitor.scala:108)
	at com.twitter.util.Monitor$class.apply(Monitor.scala:37)
	at com.twitter.finagle.util.DefaultMonitor.apply(Monitor.scala:60)
	at com.twitter.util.Timer$$anonfun$schedule$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Timer.scala:33)
	at com.twitter.util.Timer$$anonfun$schedule$1$$anonfun$apply$mcV$sp$1.apply(Timer.scala:33)
	at com.twitter.util.Timer$$anonfun$schedule$1$$anonfun$apply$mcV$sp$1.apply(Timer.scala:33)
	at com.twitter.util.Local$.let(Local.scala:71)
	at com.twitter.util.Timer$$anonfun$schedule$1.apply$mcV$sp(Timer.scala:33)
	at com.twitter.util.Monitor$$anonfun$apply$1.apply$mcV$sp(Monitor.scala:38)
	at com.twitter.util.Monitor$$anonfun$apply$1.apply(Monitor.scala:38)
	at com.twitter.util.Monitor$$anonfun$apply$1.apply(Monitor.scala:38)
	at com.twitter.util.Monitor$$anonfun$using$1.apply(Monitor.scala:110)
	at com.twitter.util.Monitor$.restoring(Monitor.scala:117)
	at com.twitter.util.Monitor$.using(Monitor.scala:108)
	at com.twitter.util.Monitor$class.apply(Monitor.scala:37)
	at com.twitter.finagle.util.DefaultMonitor.apply(Monitor.scala:60)
	at com.twitter.util.Timer$$anonfun$schedule$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Timer.scala:33)
	at com.twitter.util.Timer$$anonfun$schedule$1$$anonfun$apply$mcV$sp$1.apply(Timer.scala:33)
	at com.twitter.util.Timer$$anonfun$schedule$1$$anonfun$apply$mcV$sp$1.apply(Timer.scala:33)
	at com.twitter.util.Local$.let(Local.scala:71)
	at com.twitter.util.Timer$$anonfun$schedule$1.apply$mcV$sp(Timer.scala:33)
	at com.twitter.finagle.util.HashedWheelTimer$$anon$5.run(HashedWheelTimer.scala:18)
	at org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:556)
	at org.jboss.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:632)
	at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:369)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at java.lang.Thread.run(Thread.java:745)

127.0.0.1 - - [09/Jan/2017 12:46:09] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [09/Jan/2017 12:46:10] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [09/Jan/2017 12:46:10] "GET / HTTP/1.1" 200 -
WARN 0109 12:46:10.267 ZkAnnouncer Mutator-SendThread(localhost:2181): Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:356)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1192)
127.0.0.1 - - [09/Jan/2017 12:46:10] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [09/Jan/2017 12:46:10] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [09/Jan/2017 12:46:10] "GET / HTTP/1.1" 200 -
I 0109 12:46:10.780 THREAD191 TraceId:70c18113615ad35c: ignoring [/admin/shutdown] from 127.0.0.1, because it is a GET, not a POST
At least one request to linkerd failed.
 => Ensure that you have a local service running on port 9999
 => Inspect logs at ./logs

sh -x ci/test.sh returned exit code 1
@adleong

This comment has been minimized.

Show comment
Hide comment
@adleong

adleong Jan 11, 2017

Member

Fixed by #914

Member

adleong commented Jan 11, 2017

Fixed by #914

@adleong adleong closed this Jan 11, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment