Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kubernetes ingress and needed port to use vespa #4511

Closed
lundin opened this issue Dec 20, 2017 · 3 comments
Closed

kubernetes ingress and needed port to use vespa #4511

lundin opened this issue Dec 20, 2017 · 3 comments

Comments

@lundin
Copy link

lundin commented Dec 20, 2017 •
edited
Loading

Hi!
I looked into this issue:
#3640
But could not make every part of vespa work to for example deploy apps and start using it.

Did you ever managed to created a ingress (i am on GKE) and manage it to work with vespa tools and endpoints ?
I can reach the config endpoint mydomain.com/ApplicationStatus due to the mapping in the ingress pointing to the config port (nodeport service).
But now i am trying to use the deploy API with the endpoints in:
http://docs.vespa.ai/documentation/cloudconfig/deploy-rest-api-v2.html#use-cases

It seems that i need to have multiple (at least two ports 80(80) and 19071 on the same domain open) and ingress does not support that without specify different paths
kubernetes/ingress-nginx#1655

POSTING to
http://mydomain.com/application/v2/tenant/mytenant/session
just return 404 -default backend

and specify
http://mydomain.com:19701/application/v2/tenant/mytenant/session
hangs.

Anyone has a suggestion how i can create the ingress exposing needed ports ? Mabey one can create multiple services maped to different hosts (vespaapi.mydomain.com using 19071 and mydomain.com using 80)

Service:

apiVersion: v1
kind: Service
metadata:
  name: vespa
  labels:
    app: vespa
spec:
  selector:
    app: vespa
  type: NodePort
  ports:
  - name: sample-app
    port: 80
    targetPort: 8080
    protocol: TCP
  - name: config
    port: 19071
    targetPort: 19071
    protocol: TCP

Ingress:

- host: mydomain.com
    http:
      paths:
      - path: /
        backend:
          serviceName: vespa
          servicePort: 80
      - path: /ApplicationStatus
        backend:
          serviceName: vespa
          servicePort: 19071
@lundin
Copy link
Author

lundin commented Dec 22, 2017 •
edited
Loading

I think it is due to the config server addr is not fully resolved with the hostname at boot time so i cant reach it outside of the container/pod. exec into it and curl -s --head localhost:19071/ApplicationStatus gets 200 OK.

kubectl get endpoints vespa
NAME      ENDPOINTS                        AGE
vespa     10.40.7.3:19071,10.40.7.3:8080   19m

wget -qO- 10.40.7.3:19071
Nothing

But the logs

kubectl logs vespa-1454932670-48t5x

hostname: you must be root to change the host name
Running /opt/vespa/libexec/vespa/start-configserver
Creating data directory /opt/vespa/conf/zookeeper
Creating data directory /opt/vespa/var/zookeeper/version-2
No value found for 'services.addr_configserver' or 'vespa_base.addr_configserver'; using 'vespa-1454932670-48t5x'
runserver(configserver) running with pid: 276
Running /opt/vespa/libexec/vespa/start-vespa-base.sh
Starting config proxy using tcp/localhost:19070 as config source(s)
runserver(configproxy) running with pid: 528
Waiting for config proxy to start
config proxy started (runserver pid 528)
runserver(config-sentinel) running with pid: 643

@lundin
Copy link
Author

lundin commented Dec 23, 2017 •
edited
Loading

Can anyone from vespa see if they can find any wrong with this log?

mydomain/ApplicationStatus works (200 OK). But i fail to create a tenant with the rest API (http://docs.vespa.ai/documentation/cloudconfig/tenant-rest-api.html) and thus using the deploy API. The config server seems not be started (same as the previous thread opened) but this time i can see that the file
cat /opt/vespa/var/db/vespa/config_server/serverdb/serverdefs/search.config.qr-start.def
Exists.

Is it due to out of memory ? It is on a VM 3.75 GB

[root@vespa-7dc749d59c-b72rd vespa]# cat vespa.log
1514056293.738872       vespa-7dc749d59c-b72rd  276/64024       configserver    runserver       event   starting/1 name="java -Xms128m -Xmx2048m -XX:+PreserveFramePointer -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=
/opt/vespa/var/crash -XX:OnOutOfMemoryError=kill -9 %p -Djava.library.path=/opt/vespa/lib64 -Djava.awt.headless=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.net.client.defaultConnectTimeout=5000 -Dsun.net.client.de
faultReadTimeout=60000 -Djavax.net.ssl.keyStoreType=JKS -Djdisc.config.file=/opt/vespa/var/jdisc_core/configserver.properties -Djdisc.export.packages= -Djdisc.cache.path=/opt/vespa/var/vespa/bundlecache/configserver -Djd
isc.debug.resources=false -Djdisc.bundle.path=/opt/vespa/lib/jars -Djdisc.logger.enabled=true -Djdisc.logger.level=ALL -Djdisc.logger.tag=jdisc/configserver -Dfile.encoding=UTF-8 -Dzookeeperlogfile=/opt/vespa/logs/vespa/
zookeeper.configserver.log -cp /opt/vespa/lib/jars/jdisc_core-jar-with-dependencies.jar com.yahoo.jdisc.core.StandaloneMain standalone-container-jar-with-dependencies.jar (pid 278)"
1514056295.136044       vespa-7dc749d59c-b72rd  525/27337       configproxy     runserver       event   starting/1 name="java -Xms32M -Xmx256M -XX:ThreadStackSize=256 -XX:MaxJavaStackTraceDepth=-1 -XX:OnOutOfMemoryError=
kill -9 %p -Dproxyconfigsources=tcp/localhost:19070 -cp libexec/vespa/patches/configproxy:lib/jars/config-proxy-jar-with-dependencies.jar com.yahoo.vespa.config.proxy.ProxyServer 19090 (pid 527)"
1514056295.207474       vespa-7dc749d59c-b72rd  278     configserver    stdout  debug   Initializing application without privileges.
1514056295.482  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.apache.felix.framework      info    ServiceEvent REGISTERED
1514056295.484  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.ApplicationLoader debug   App location is not a class name. Installing bundle
1514056295.486  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/standalone-container-jar-with-dependencies.jar

1514056304.426  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.container.core.config.BundleLoader  info    0 bundles were removed, and 7 bundles were installed.
'.
1514056295.574  vespa-7dc749d59c-b72rd  278     jdisc/configserver      standalone-container    info    BundleEvent INSTALLED
1514056295.575  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'standalone-container' requires install from 'configdefinitions-jar-with-dependencies.jar'.
1514056295.576  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/configdefinitions-jar-with-dependencies.jar'.
1514056295.592  vespa-7dc749d59c-b72rd  278     jdisc/configserver      configdefinitions       info    BundleEvent INSTALLED
1514056295.593  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'standalone-container' requires install from 'config-provisioning-jar-with-dependencies.jar'.
1514056295.594  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/config-provisioning-jar-with-dependencies.jar'.
1514056295.600  vespa-7dc749d59c-b72rd  278     jdisc/configserver      config-provisioning     info    BundleEvent INSTALLED
1514056295.601  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'standalone-container' requires install from 'config-bundle-jar-with-dependencies.jar'.
1514056295.601  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/config-bundle-jar-with-dependencies.jar'.
1514056295.630  vespa-7dc749d59c-b72rd  278     jdisc/configserver      config-bundle   info    BundleEvent INSTALLED
1514056295.631  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'standalone-container' requires install from 'config-model-api-jar-with-dependencies.jar'.
1514056295.631  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/config-model-api-jar-with-dependencies.jar'.
1514056295.646  vespa-7dc749d59c-b72rd  278     jdisc/configserver      config-model-api        info    BundleEvent INSTALLED
1514056295.647  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'standalone-container' requires install from 'config-model-jar-with-dependencies.jar'.
1514056295.648  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/config-model-jar-with-dependencies.jar'.
1514056295.714  vespa-7dc749d59c-b72rd  278     jdisc/configserver      config-model    info    BundleEvent INSTALLED
1514056295.715  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'standalone-container' requires install from 'container-disc-jar-with-dependencies.jar'.
1514056295.716  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/container-disc-jar-with-dependencies.jar'.
1514056296.043  vespa-7dc749d59c-b72rd  278     jdisc/configserver      container-disc  info    BundleEvent INSTALLED
1514056296.043  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'config-bundle-jar-with-dependencies.jar'.
1514056296.044  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle from 'file:/opt/vespa/lib/jars/config-bundle-jar-with-dependencies.jar' already installed.
1514056296.044  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'configdefinitions-jar-with-dependencies.jar'.
1514056296.044  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle from 'file:/opt/vespa/lib/jars/configdefinitions-jar-with-dependencies.jar' already installed.
1514056296.044  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'container-jersey2-jar-with-dependencies.jar'.
1514056296.044  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/container-jersey2-jar-with-dependencies.jar'.
1514056296.086  vespa-7dc749d59c-b72rd  278     jdisc/configserver      container-jersey2       info    BundleEvent INSTALLED
1514056296.089  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'container-search-and-docproc-jar-with-dependencies.jar'.
1514056296.093  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/container-search-and-docproc-jar-with-dependencies.jar'.
1514056296.095  vespa-7dc749d59c-b72rd  527/1   configproxy     configproxy.com.yahoo.vespa.config.proxy.ProxyServer    event   started/1 name=configproxy
1514056296.200  vespa-7dc749d59c-b72rd  278     jdisc/configserver      container-search-and-docproc    info    BundleEvent INSTALLED
1514056296.200  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'docprocs-jar-with-dependencies.jar'.
1514056296.206  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/docprocs-jar-with-dependencies.jar'.
1514056296.227  vespa-7dc749d59c-b72rd  278     jdisc/configserver      docprocs        info    BundleEvent INSTALLED
1514056296.227  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'jdisc_http_service-jar-with-dependencies.jar'.
1514056296.228  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jdisc_http_service-jar-with-dependencies.jar'.
1514056296.270  vespa-7dc749d59c-b72rd  278     jdisc/configserver      jdisc_http_service      info    BundleEvent INSTALLED
1514056296.272  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'jdisc_http_service' requires install from 'asm-debug-all-5.0.3.jar'.
1514056296.272  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/asm-debug-all-5.0.3.jar'.
1514056296.289  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.objectweb.asm.all.debug     info    BundleEvent INSTALLED
1514056296.289  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'jdisc_http_service' requires install from 'javax.servlet-api-3.1.0.jar'.
1514056296.290  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/javax.servlet-api-3.1.0.jar'.
1514056296.304  vespa-7dc749d59c-b72rd  278     jdisc/configserver      javax.servlet-api       info    BundleEvent INSTALLED
1514056296.305  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'jdisc_http_service' requires install from 'jetty-continuation-9.4.8.v20171121.jar'.
1514056296.305  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jetty-continuation-9.4.8.v20171121.jar'.
1514056296.327  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.continuation  info    BundleEvent INSTALLED
1514056296.328  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'jdisc_http_service' requires install from 'jetty-http-9.4.8.v20171121.jar'.
1514056296.328  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jetty-http-9.4.8.v20171121.jar'.
1514056296.345  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.http  info    BundleEvent INSTALLED
1514056296.345  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'jdisc_http_service' requires install from 'jetty-io-9.4.8.v20171121.jar'.
1514056296.345  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jetty-io-9.4.8.v20171121.jar'.
1514056296.353  vespa-7dc749d59c-b72rd  527/1   configproxy     configproxy.com.yahoo.vespa.config.proxy.RpcConfigSourceClient  info    Could not connect to config source at tcp/localhost:19070
1514056296.358  vespa-7dc749d59c-b72rd  527/1   configproxy     configproxy.com.yahoo.vespa.config.proxy.RpcConfigSourceClient  info    Could not connect to any config source in set [tcp/localhost:19070], please make sure config server(s) are running.
1514056296.389  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.io    info    BundleEvent INSTALLED
1514056296.394  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'jdisc_http_service' requires install from 'jetty-jmx-9.4.8.v20171121.jar'.
1514056296.400  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jetty-jmx-9.4.8.v20171121.jar'.
1514056296.433  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.jmx   info    BundleEvent INSTALLED
1514056296.434  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'jdisc_http_service' requires install from 'jetty-security-9.4.8.v20171121.jar'.
1514056296.435  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jetty-security-9.4.8.v20171121.jar'.
1514056296.449  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.security      info    BundleEvent INSTALLED
1514056296.450  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'jdisc_http_service' requires install from 'jetty-server-9.4.8.v20171121.jar'.
1514056296.450  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jetty-server-9.4.8.v20171121.jar'.
1514056296.481  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.server        info    BundleEvent INSTALLED
1514056296.481  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'jdisc_http_service' requires install from 'jetty-servlet-9.4.8.v20171121.jar'.
1514056296.482  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jetty-servlet-9.4.8.v20171121.jar'.
1514056296.497  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.servlet       info    BundleEvent INSTALLED
1514056296.498  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'jdisc_http_service' requires install from 'jetty-servlets-9.4.8.v20171121.jar'.
1514056296.498  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jetty-servlets-9.4.8.v20171121.jar'.
1514056296.506  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.servlets      info    BundleEvent INSTALLED
1514056296.507  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'jdisc_http_service' requires install from 'jetty-util-9.4.8.v20171121.jar'.
1514056296.508  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jetty-util-9.4.8.v20171121.jar'.
1514056296.521  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.util  info    BundleEvent INSTALLED
1514056296.522  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'jdisc_http_service' requires install from 'org.apache.aries.spifly.dynamic.bundle-1.0.8.jar'.
1514056296.523  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/org.apache.aries.spifly.dynamic.bundle-1.0.8.jar'.
1514056296.537  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.apache.aries.spifly.dynamic.bundle  info    BundleEvent INSTALLED
1514056296.537  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'jdisc_http_service' requires install from 'org.apache.aries.util-1.0.0.jar'.
1514056296.538  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/org.apache.aries.util-1.0.0.jar'.
1514056296.550  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.apache.aries.util   info    BundleEvent INSTALLED
1514056296.551  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'jdisc_http_service' requires install from 'component-jar-with-dependencies.jar'.
1514056296.552  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/component-jar-with-dependencies.jar'.
1514056296.563  vespa-7dc749d59c-b72rd  278     jdisc/configserver      component       info    BundleEvent INSTALLED
1514056296.564  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'vespaclient-container-plugin-jar-with-dependencies.jar'.
1514056296.564  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/vespaclient-container-plugin-jar-with-dependencies.jar'.
1514056296.601  vespa-7dc749d59c-b72rd  278     jdisc/configserver      vespaclient-container-plugin    info    BundleEvent INSTALLED
1514056296.602  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'simplemetrics-jar-with-dependencies.jar'.
1514056296.602  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/simplemetrics-jar-with-dependencies.jar'.
1514056296.609  vespa-7dc749d59c-b72rd  278     jdisc/configserver      simplemetrics   info    BundleEvent INSTALLED
1514056296.611  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'defaults-jar-with-dependencies.jar'.
1514056296.615  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/defaults-jar-with-dependencies.jar'.
1514056296.633  vespa-7dc749d59c-b72rd  278     jdisc/configserver      defaults        info    BundleEvent INSTALLED
1514056296.634  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'component-jar-with-dependencies.jar'.
1514056296.634  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle from 'file:/opt/vespa/lib/jars/component-jar-with-dependencies.jar' already installed.
1514056296.634  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'zkfacade-jar-with-dependencies.jar'.
1514056296.635  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/zkfacade-jar-with-dependencies.jar'.
1514056296.690  vespa-7dc749d59c-b72rd  278     jdisc/configserver      zkfacade        info    BundleEvent INSTALLED
1514056296.690  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'aopalliance-repackaged-2.5.0-b05.jar'.
1514056296.690  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/aopalliance-repackaged-2.5.0-b05.jar'.
1514056296.706  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.hk2.external.aopalliance-repackaged       info    BundleEvent INSTALLED
1514056296.707  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'hk2-api-2.5.0-b05.jar'.
1514056296.708  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/hk2-api-2.5.0-b05.jar'.
1514056296.733  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.hk2.api   info    BundleEvent INSTALLED
1514056296.733  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'hk2-locator-2.5.0-b05.jar'.
1514056296.734  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/hk2-locator-2.5.0-b05.jar'.
1514056296.759  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.hk2.locator       info    BundleEvent INSTALLED
1514056296.761  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'hk2-utils-2.5.0-b05.jar'.
1514056296.764  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/hk2-utils-2.5.0-b05.jar'.
1514056296.780  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.hk2.utils info    BundleEvent INSTALLED
1514056296.781  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'jackson-annotations-2.8.3.jar'.
1514056296.782  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jackson-annotations-2.8.3.jar'.
1514056296.790  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.core.jackson-annotations  info    BundleEvent INSTALLED
1514056296.790  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'jackson-core-2.8.3.jar'.
1514056296.791  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jackson-core-2.8.3.jar'.
1514056296.797  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.core.jackson-core info    BundleEvent INSTALLED
1514056296.798  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'jackson-databind-2.8.3.jar'.
1514056296.798  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jackson-databind-2.8.3.jar'.
1514056296.807  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.core.jackson-databind     info    BundleEvent INSTALLED
1514056296.807  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'jackson-datatype-jdk8-2.8.3.jar'.
1514056296.808  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jackson-datatype-jdk8-2.8.3.jar'.
1514056296.814  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.datatype.jackson-datatype-jdk8    info    BundleEvent INSTALLED
1514056296.816  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'jackson-datatype-jsr310-2.8.3.jar'.
1514056296.816  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jackson-datatype-jsr310-2.8.3.jar'.
1514056296.821  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.datatype.jackson-datatype-jsr310  info    BundleEvent INSTALLED
1514056296.822  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'jackson-jaxrs-base-2.8.3.jar'.
1514056296.822  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jackson-jaxrs-base-2.8.3.jar'.
1514056296.827  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.jaxrs.jackson-jaxrs-base  info    BundleEvent INSTALLED
1514056296.827  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'jackson-jaxrs-json-provider-2.8.3.jar'.
1514056296.828  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jackson-jaxrs-json-provider-2.8.3.jar'.
1514056296.833  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.jaxrs.jackson-jaxrs-json-provider info    BundleEvent INSTALLED
1514056296.834  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'jackson-module-jaxb-annotations-2.8.3.jar'.
1514056296.835  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jackson-module-jaxb-annotations-2.8.3.jar'.
1514056296.838  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.module.jackson-module-jaxb-annotations    info    BundleEvent INSTALLED
1514056296.840  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'javassist-3.20.0-GA.jar'.
1514056296.840  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/javassist-3.20.0-GA.jar'.
1514056296.843  vespa-7dc749d59c-b72rd  278     jdisc/configserver      javassist       info    BundleEvent INSTALLED
1514056296.843  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'javax.ws.rs-api-2.0.1.jar'.
1514056296.843  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/javax.ws.rs-api-2.0.1.jar'.
1514056296.850  vespa-7dc749d59c-b72rd  278     jdisc/configserver      javax.ws.rs-api info    BundleEvent INSTALLED
1514056296.850  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'jersey-client-2.23.2.jar'.
1514056296.851  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jersey-client-2.23.2.jar'.
1514056296.857  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.core.jersey-client info    BundleEvent INSTALLED
1514056296.857  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'jersey-common-2.23.2.jar'.
1514056296.857  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jersey-common-2.23.2.jar'.
1514056296.870  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.core.jersey-common info    BundleEvent INSTALLED
1514056296.871  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'jersey-container-servlet-2.23.2.jar'.
1514056296.871  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jersey-container-servlet-2.23.2.jar'.
1514056296.874  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.containers.jersey-container-servlet        info    BundleEvent INSTALLED
1514056296.875  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'jersey-container-servlet-core-2.23.2.jar'.
1514056296.875  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jersey-container-servlet-core-2.23.2.jar'.
1514056296.883  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.containers.jersey-container-servlet-core   info    BundleEvent INSTALLED
1514056296.883  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'jersey-entity-filtering-2.23.2.jar'.
1514056296.884  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jersey-entity-filtering-2.23.2.jar'.
1514056296.889  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.ext.jersey-entity-filtering        info    BundleEvent INSTALLED
1514056296.889  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'jersey-guava-2.23.2.jar'.
1514056296.890  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jersey-guava-2.23.2.jar'.
1514056296.901  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.bundles.repackaged.jersey-guava    info    BundleEvent INSTALLED
1514056296.901  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'jersey-media-jaxb-2.23.2.jar'.
1514056296.902  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jersey-media-jaxb-2.23.2.jar'.
1514056296.912  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.media.jersey-media-jaxb    info    BundleEvent INSTALLED
1514056296.912  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'jersey-media-json-jackson-2.23.2.jar'.
1514056296.913  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jersey-media-json-jackson-2.23.2.jar'.
1514056296.942  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.media.jersey-media-json-jackson    info    BundleEvent INSTALLED
1514056296.942  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'jersey-media-multipart-2.23.2.jar'.
1514056296.943  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jersey-media-multipart-2.23.2.jar'.
1514056296.952  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.media.jersey-media-multipart       info    BundleEvent INSTALLED
1514056296.953  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'mimepull-1.9.6.jar'.
1514056296.953  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/mimepull-1.9.6.jar'.
1514056296.959  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.jvnet.mimepull      info    BundleEvent INSTALLED
1514056296.960  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'jersey-server-2.23.2.jar'.
1514056296.960  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jersey-server-2.23.2.jar'.
1514056296.968864       vespa-7dc749d59c-b72rd  651/3287        config-sentinel runserver       event   starting/1 name="sbin/vespa-config-sentinel -c hosts/vespa-7dc749d59c-b72rd (pid 653)"
1514056296.978  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.core.jersey-server info    BundleEvent INSTALLED
1514056296.979  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'jersey-proxy-client-2.23.2.jar'.
1514056296.981  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/jersey-proxy-client-2.23.2.jar'.
1514056296.988  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.ext.jersey-proxy-client    info    BundleEvent INSTALLED
1514056296.988  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'osgi-resource-locator-1.0.1.jar'.
1514056296.989  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/osgi-resource-locator-1.0.1.jar'.
1514056296.998  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.hk2.osgi-resource-locator info    BundleEvent INSTALLED
1514056296.999  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'container-disc' requires install from 'validation-api-1.1.0.Final.jar'.
1514056296.999  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/validation-api-1.1.0.Final.jar'.
1514056297.007  vespa-7dc749d59c-b72rd  278     jdisc/configserver      javax.validation.api    info    BundleEvent INSTALLED
1514056297.007  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   OSGi bundle 'standalone-container' requires install from 'vespajlib.jar'.
1514056297.008  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    debug   Installing OSGi bundle from 'file:/opt/vespa/lib/jars/vespajlib.jar'.
1514056297.018  vespa-7dc749d59c-b72rd  278     jdisc/configserver      vespajlib       info    BundleEvent INSTALLED
1514056297.018  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.yahoo.jdisc.core.FelixFramework    info    OSGi bundle 'standalone-container' states that it requires privileged initialization, but privileges are not available. YMMV.
1514056297.072897       vespa-7dc749d59c-b72rd  653/32935       config-sentinel config-sentinel event   started/1 name="config-sentinel"
1514056297.362  vespa-7dc749d59c-b72rd  278     jdisc/configserver      config-bundle   info    BundleEvent RESOLVED
1514056297.362  vespa-7dc749d59c-b72rd  278     jdisc/configserver      component       info    BundleEvent RESOLVED
1514056297.362  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.http  info    BundleEvent RESOLVED
1514056297.362  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.apache.aries.util   info    BundleEvent RESOLVED
1514056297.362  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.util  info    BundleEvent RESOLVED
1514056297.362  vespa-7dc749d59c-b72rd  278     jdisc/configserver      container-search-and-docproc    info    BundleEvent RESOLVED
1514056297.362  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.apache.aries.spifly.dynamic.bundle  info    BundleEvent RESOLVED
1514056297.362  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.security      info    BundleEvent RESOLVED
1514056297.363  vespa-7dc749d59c-b72rd  278     jdisc/configserver      config-model    info    BundleEvent RESOLVED
1514056297.363  vespa-7dc749d59c-b72rd  278     jdisc/configserver      simplemetrics   info    BundleEvent RESOLVED
1514056297.363  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.core.jackson-core info    BundleEvent RESOLVED
1514056297.363  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.objectweb.asm.all.debug     info    BundleEvent RESOLVED
1514056297.363  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.jmx   info    BundleEvent RESOLVED
1514056297.363  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.server        info    BundleEvent RESOLVED
1514056297.364  vespa-7dc749d59c-b72rd  278     jdisc/configserver      container-disc  info    BundleEvent RESOLVED
1514056297.364  vespa-7dc749d59c-b72rd  278     jdisc/configserver      configdefinitions       info    BundleEvent RESOLVED
1514056297.364  vespa-7dc749d59c-b72rd  278     jdisc/configserver      standalone-container    info    BundleEvent RESOLVED
1514056297.364  vespa-7dc749d59c-b72rd  278     jdisc/configserver      config-provisioning     info    BundleEvent RESOLVED
1514056297.364  vespa-7dc749d59c-b72rd  278     jdisc/configserver      jdisc_http_service      info    BundleEvent RESOLVED
1514056297.364  vespa-7dc749d59c-b72rd  278     jdisc/configserver      config-model-api        info    BundleEvent RESOLVED
1514056297.364  vespa-7dc749d59c-b72rd  278     jdisc/configserver      javax.servlet-api       info    BundleEvent RESOLVED
1514056297.364  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.servlet       info    BundleEvent RESOLVED
1514056297.365  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.io    info    BundleEvent RESOLVED
1514056297.477  vespa-7dc749d59c-b72rd  527/12  configproxy     configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester       info    Connection to tcp/localhost:19070 failed or timed out, clients will keep existing config, will keep trying.
1514056298.153  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.google.inject.internal.util.$Stopwatch     debug   Module execution: 66ms
1514056298.164  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.google.inject.internal.util.$Stopwatch     debug   TypeListeners creation: 11ms
1514056298.183  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.google.inject.internal.util.$Stopwatch     debug   Scopes creation: 19ms
1514056298.184  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.google.inject.internal.util.$Stopwatch     debug   Converters creation: 1ms
1514056298.200  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.google.inject.internal.util.$Stopwatch     debug   Binding creation: 13ms
1514056298.202  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.google.inject.internal.util.$Stopwatch     debug   Private environment creation: 5ms
1514056298.203  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.google.inject.internal.util.$Stopwatch     debug   Injector construction: 0ms
1514056298.203  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.google.inject.internal.util.$Stopwatch     debug   Binding initialization: 1ms
1514056298.203  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.google.inject.internal.util.$Stopwatch     debug   Binding indexing: 0ms
1514056298.203  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.google.inject.internal.util.$Stopwatch     debug   Collecting injection requests: 0ms
1514056298.203  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.google.inject.internal.util.$Stopwatch     debug   Binding validation: 0ms
1514056298.203  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.google.inject.internal.util.$Stopwatch     debug   Static validation: 0ms
1514056298.204  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.google.inject.internal.util.$Stopwatch     debug   Instance member validation: 1ms
1514056298.205  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.google.inject.internal.util.$Stopwatch     debug   Provider verification: 1ms
1514056298.206  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.google.inject.internal.util.$Stopwatch     debug   Static member injection: 1ms
1514056298.207  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.google.inject.internal.util.$Stopwatch     debug   Instance injection: 1ms
1514056298.219  vespa-7dc749d59c-b72rd  278/1   jdisc/configserver      /com.google.inject.internal.util.$Stopwatch     debug   Preloading singletons: 12ms
1514056299.520  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.config.model.application.provider.FilesApplicationPackage   info    Adding user include dir 'config-models'
1514056300.200  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.config.model.application.provider.FilesApplicationPackage   info    Adding user include dir 'config-models'
1514056302.141  vespa-7dc749d59c-b72rd  278     jdisc/configserver      standalone-container    info    ServiceEvent REGISTERED
1514056302.142  vespa-7dc749d59c-b72rd  278     jdisc/configserver      standalone-container    info    BundleEvent STARTED
1514056302.142  vespa-7dc749d59c-b72rd  278     jdisc/configserver      configdefinitions       info    BundleEvent STARTED
1514056302.143  vespa-7dc749d59c-b72rd  278     jdisc/configserver      config-provisioning     info    BundleEvent STARTED
1514056302.143  vespa-7dc749d59c-b72rd  278     jdisc/configserver      config-bundle   info    BundleEvent STARTED
1514056302.143  vespa-7dc749d59c-b72rd  278     jdisc/configserver      config-model-api        info    BundleEvent STARTED
1514056302.143  vespa-7dc749d59c-b72rd  278     jdisc/configserver      config-model    info    BundleEvent STARTED
1514056302.146  vespa-7dc749d59c-b72rd  278     jdisc/configserver      container-disc  info    ServiceEvent REGISTERED
1514056302.146  vespa-7dc749d59c-b72rd  278     jdisc/configserver      container-disc  info    BundleEvent STARTED
1514056302.225  vespa-7dc749d59c-b72rd  278     jdisc/configserver      javax.validation.api    info    BundleEvent RESOLVED
1514056302.225  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.jvnet.mimepull      info    BundleEvent RESOLVED
1514056302.225  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.core.jersey-common info    BundleEvent RESOLVED
1514056302.226  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.hk2.external.aopalliance-repackaged       info    BundleEvent RESOLVED
1514056302.226  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.bundles.repackaged.jersey-guava    info    BundleEvent RESOLVED
1514056302.226  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.media.jersey-media-multipart       info    BundleEvent RESOLVED
1514056302.226  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.datatype.jackson-datatype-jsr310  info    BundleEvent RESOLVED
1514056302.226  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.containers.jersey-container-servlet-core   info    BundleEvent RESOLVED
1514056302.226  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.hk2.osgi-resource-locator info    BundleEvent RESOLVED
1514056302.226  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.module.jackson-module-jaxb-annotations    info    BundleEvent RESOLVED
1514056302.226  vespa-7dc749d59c-b72rd  278     jdisc/configserver      javax.ws.rs-api info    BundleEvent RESOLVED
1514056302.226  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.core.jersey-server info    BundleEvent RESOLVED
1514056302.226  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.hk2.utils info    BundleEvent RESOLVED
1514056302.226  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.jaxrs.jackson-jaxrs-json-provider info    BundleEvent RESOLVED
1514056302.227  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.core.jersey-client info    BundleEvent RESOLVED
1514056302.227  vespa-7dc749d59c-b72rd  278     jdisc/configserver      container-jersey2       info    BundleEvent RESOLVED
1514056302.227  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.datatype.jackson-datatype-jdk8    info    BundleEvent RESOLVED
1514056302.227  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.core.jackson-databind     info    BundleEvent RESOLVED
1514056302.228  vespa-7dc749d59c-b72rd  278     jdisc/configserver      javassist       info    BundleEvent RESOLVED
1514056302.228  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.core.jackson-annotations  info    BundleEvent RESOLVED
1514056302.228  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.jaxrs.jackson-jaxrs-base  info    BundleEvent RESOLVED
1514056302.228  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.hk2.locator       info    BundleEvent RESOLVED
1514056302.228  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.hk2.api   info    BundleEvent RESOLVED
1514056302.228  vespa-7dc749d59c-b72rd  278     jdisc/configserver      container-jersey2       info    BundleEvent STARTED
1514056302.230  vespa-7dc749d59c-b72rd  278     jdisc/configserver      container-search-and-docproc    info    BundleEvent STARTED
1514056302.241  vespa-7dc749d59c-b72rd  278     jdisc/configserver      docprocs        info    BundleEvent RESOLVED
1514056302.242  vespa-7dc749d59c-b72rd  278     jdisc/configserver      docprocs        info    BundleEvent STARTED
1514056302.243  vespa-7dc749d59c-b72rd  278     jdisc/configserver      jdisc_http_service      info    BundleEvent STARTED
1514056302.243  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.objectweb.asm.all.debug     info    BundleEvent STARTED
1514056302.244  vespa-7dc749d59c-b72rd  278     jdisc/configserver      javax.servlet-api       info    BundleEvent STARTED
1514056302.251  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.continuation  info    BundleEvent RESOLVED
1514056302.252  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.continuation  info    BundleEvent STARTED
1514056302.253  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.http  info    BundleEvent STARTED
1514056302.253  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.io    info    BundleEvent STARTED
1514056302.254  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.jmx   info    BundleEvent STARTED
1514056302.255  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.security      info    BundleEvent STARTED
1514056302.255  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.server        info    BundleEvent STARTED
1514056302.256  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.servlet       info    BundleEvent STARTED
1514056302.262  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.servlets      info    BundleEvent RESOLVED
1514056302.262  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.servlets      info    BundleEvent STARTED
1514056302.263  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.util  info    BundleEvent STARTED
1514056302.274  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.apache.aries.spifly.dynamic.bundle  info    ServiceEvent REGISTERED
1514056302.289  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.eclipse.jetty.http  info    ServiceEvent REGISTERED
1514056302.314  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.apache.aries.spifly.dynamic.bundle  info    BundleEvent STARTED
1514056302.315  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.apache.aries.util   info    BundleEvent STARTED
1514056302.317  vespa-7dc749d59c-b72rd  278     jdisc/configserver      component       info    BundleEvent STARTED
1514056302.331  vespa-7dc749d59c-b72rd  278     jdisc/configserver      vespaclient-container-plugin    info    BundleEvent RESOLVED
1514056302.332  vespa-7dc749d59c-b72rd  278     jdisc/configserver      vespaclient-container-plugin    info    BundleEvent STARTED
1514056302.333  vespa-7dc749d59c-b72rd  278     jdisc/configserver      simplemetrics   info    BundleEvent STARTED
1514056302.335  vespa-7dc749d59c-b72rd  278     jdisc/configserver      defaults        info    BundleEvent RESOLVED
1514056302.335  vespa-7dc749d59c-b72rd  278     jdisc/configserver      defaults        info    BundleEvent STARTED
1514056302.350  vespa-7dc749d59c-b72rd  278     jdisc/configserver      zkfacade        info    BundleEvent RESOLVED
1514056302.350  vespa-7dc749d59c-b72rd  278     jdisc/configserver      zkfacade        info    BundleEvent STARTED
1514056302.351  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.hk2.external.aopalliance-repackaged       info    BundleEvent STARTED
1514056302.351  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.hk2.api   info    BundleEvent STARTED
1514056302.352  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.hk2.locator       info    BundleEvent STARTED
1514056302.352  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.hk2.utils info    BundleEvent STARTED
1514056302.353  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.core.jackson-annotations  info    BundleEvent STARTED
1514056302.354  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.core.jackson-core info    BundleEvent STARTED
1514056302.355  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.core.jackson-databind     info    BundleEvent STARTED
1514056302.356  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.datatype.jackson-datatype-jdk8    info    BundleEvent STARTED
1514056302.362  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.datatype.jackson-datatype-jsr310  info    BundleEvent STARTED
1514056302.363  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.jaxrs.jackson-jaxrs-base  info    BundleEvent STARTED
1514056302.366  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.jaxrs.jackson-jaxrs-json-provider info    BundleEvent STARTED
1514056302.367  vespa-7dc749d59c-b72rd  278     jdisc/configserver      com.fasterxml.jackson.module.jackson-module-jaxb-annotations    info    BundleEvent STARTED
1514056302.367  vespa-7dc749d59c-b72rd  278     jdisc/configserver      javassist       info    BundleEvent STARTED
1514056302.368  vespa-7dc749d59c-b72rd  278     jdisc/configserver      javax.ws.rs-api info    BundleEvent STARTED
1514056302.370  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.core.jersey-client info    BundleEvent STARTED
1514056302.371  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.core.jersey-common info    BundleEvent STARTED
1514056302.379  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.containers.jersey-container-servlet        info    BundleEvent RESOLVED
1514056302.380  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.containers.jersey-container-servlet        info    BundleEvent STARTED
1514056302.382  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.containers.jersey-container-servlet-core   info    BundleEvent STARTED
1514056302.394  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.ext.jersey-entity-filtering        info    BundleEvent RESOLVED
1514056302.394  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.ext.jersey-entity-filtering        info    BundleEvent STARTED
1514056302.395  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.bundles.repackaged.jersey-guava    info    BundleEvent STARTED
1514056302.402  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.media.jersey-media-jaxb    info    BundleEvent RESOLVED
1514056302.403  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.media.jersey-media-jaxb    info    BundleEvent STARTED
1514056302.412  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.media.jersey-media-json-jackson    info    BundleEvent RESOLVED
1514056302.412  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.media.jersey-media-json-jackson    info    BundleEvent STARTED
1514056302.412  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.media.jersey-media-multipart       info    BundleEvent STARTED
1514056302.413  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.jvnet.mimepull      info    BundleEvent STARTED
1514056302.416  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.core.jersey-server info    BundleEvent STARTED
1514056302.425  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.ext.jersey-proxy-client    info    BundleEvent RESOLVED
1514056302.426  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.jersey.ext.jersey-proxy-client    info    BundleEvent STARTED
1514056302.497  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.glassfish.hk2.osgi-resource-locator info    BundleEvent STARTED
1514056302.501  vespa-7dc749d59c-b72rd  278     jdisc/configserver      javax.validation.api    info    BundleEvent STARTED
1514056302.507  vespa-7dc749d59c-b72rd  278     jdisc/configserver      vespajlib       info    BundleEvent RESOLVED
1514056302.507  vespa-7dc749d59c-b72rd  278     jdisc/configserver      vespajlib       info    BundleEvent STARTED
1514056302.572  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.config.model.application.provider.FilesApplicationPackage   info    Adding user include dir 'config-models'
1514056303.009  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.config.model.application.provider.FilesApplicationPackage   info    Adding user include dir 'config-models'
1514056304.100  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.container.core.config.HandlersConfigurerDi  info    Installing bundles from the latest application
1514056304.111  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.container.core.config.BundleLoader  info    Installing bundle with reference 'LocalFileDb:components/configserver.jar'
1514056304.131  vespa-7dc749d59c-b72rd  278     jdisc/configserver      configserver    info    BundleEvent INSTALLED
1514056304.131  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.container.core.config.BundleLoader  info    Installing bundle with reference 'LocalFileDb:components/config-model-fat.jar'
1514056304.361  vespa-7dc749d59c-b72rd  278     jdisc/configserver      config-model-fat        info    BundleEvent INSTALLED
1514056304.361  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.container.core.config.BundleLoader  info    Installing bundle with reference 'LocalFileDb:components/node-repository.jar'
1514056304.377  vespa-7dc749d59c-b72rd  278     jdisc/configserver      node-repository info    BundleEvent INSTALLED
1514056304.377  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.container.core.config.BundleLoader  info    Installing bundle with reference 'LocalFileDb:components/orchestrator.jar'
1514056304.382  vespa-7dc749d59c-b72rd  278     jdisc/configserver      orchestrator    info    BundleEvent INSTALLED
1514056304.382  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.container.core.config.BundleLoader  info    Installing bundle with reference 'LocalFileDb:components/application-model.jar'
1514056304.386  vespa-7dc749d59c-b72rd  278     jdisc/configserver      application-model       info    BundleEvent INSTALLED
1514056304.387  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.container.core.config.BundleLoader  info    Installing bundle with reference 'LocalFileDb:components/service-monitor.jar'
1514056304.391  vespa-7dc749d59c-b72rd  278     jdisc/configserver      service-monitor info    BundleEvent INSTALLED
1514056304.391  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.container.core.config.BundleLoader  info    Installing bundle with reference 'LocalFileDb:components/zkfacade.jar'
1514056304.402  vespa-7dc749d59c-b72rd  278     jdisc/configserver      configserver    info    BundleEvent RESOLVED
1514056304.403  vespa-7dc749d59c-b72rd  278     jdisc/configserver      configserver    info    BundleEvent STARTED
1514056304.410  vespa-7dc749d59c-b72rd  278     jdisc/configserver      config-model-fat        info    BundleEvent RESOLVED
1514056304.411  vespa-7dc749d59c-b72rd  278     jdisc/configserver      config-model-fat        info    BundleEvent STARTED
1514056304.422  vespa-7dc749d59c-b72rd  278     jdisc/configserver      orchestrator    info    BundleEvent RESOLVED
1514056304.423  vespa-7dc749d59c-b72rd  278     jdisc/configserver      application-model       info    BundleEvent RESOLVED
1514056304.423  vespa-7dc749d59c-b72rd  278     jdisc/configserver      service-monitor info    BundleEvent RESOLVED
1514056304.423  vespa-7dc749d59c-b72rd  278     jdisc/configserver      node-repository info    BundleEvent RESOLVED
1514056304.423  vespa-7dc749d59c-b72rd  278     jdisc/configserver      node-repository info    BundleEvent STARTED
1514056304.423  vespa-7dc749d59c-b72rd  278     jdisc/configserver      orchestrator    info    BundleEvent STARTED
1514056304.425  vespa-7dc749d59c-b72rd  278     jdisc/configserver      application-model       info    BundleEvent STARTED
1514056304.426  vespa-7dc749d59c-b72rd  278     jdisc/configserver      service-monitor info    BundleEvent STARTED
1514056304.426  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.container.core.config.BundleLoader  info    0 bundles were removed, and 7 bundles were installed.
1514056304.427  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.container.core.config.BundleLoader  info    Installed bundles: {[0]org.apache.felix.framework, [1]standalone-container, [2]configdefinitions, [3]config-provisioning, [4]config-bundle, [5]config-model-api, [6]config-model, [7]container-disc, [8]container-jersey2, [9]container-search-and-docproc, [10]docprocs, [11]jdisc_http_service, [12]org.objectweb.asm.all.debug, [13]javax.servlet-api, [14]org.eclipse.jetty.continuation, [15]org.eclipse.jetty.http, [16]org.eclipse.jetty.io, [17]org.eclipse.jetty.jmx, [18]org.eclipse.jetty.security, [19]org.eclipse.jetty.server, [20]org.eclipse.jetty.servlet, [21]org.eclipse.jetty.servlets, [22]org.eclipse.jetty.util, [23]org.apache.aries.spifly.dynamic.bundle, [24]org.apache.aries.util, [25]component, [26]vespaclient-container-plugin, [27]simplemetrics, [28]defaults, [29]zkfacade, [30]org.glassfish.hk2.external.aopalliance-repackaged, [31]org.glassfish.hk2.api, [32]org.glassfish.hk2.locator, [33]org.glassfish.hk2.utils, [34]com.fasterxml.jackson.core.jackson-annotations, [35]com.fasterxml.jackson.core.jackson-core, [36]com.fasterxml.jackson.core.jackson-databind, [37]com.fasterxml.jackson.datatype.jackson-datatype-jdk8, [38]com.fasterxml.jackson.datatype.jackson-datatype-jsr310, [39]com.fasterxml.jackson.jaxrs.jackson-jaxrs-base, [40]com.fasterxml.jackson.jaxrs.jackson-jaxrs-json-provider, [41]com.fasterxml.jackson.module.jackson-module-jaxb-annotations, [42]javassist, [43]javax.ws.rs-api, [44]org.glassfish.jersey.core.jersey-client, [45]org.glassfish.jersey.core.jersey-common, [46]org.glassfish.jersey.containers.jersey-container-servlet, [47]org.glassfish.jersey.containers.jersey-container-servlet-core, [48]org.glassfish.jersey.ext.jersey-entity-filtering, [49]org.glassfish.jersey.bundles.repackaged.jersey-guava, [50]org.glassfish.jersey.media.jersey-media-jaxb, [51]org.glassfish.jersey.media.jersey-media-json-jackson, [52]org.glassfish.jersey.media.jersey-media-multipart, [53]org.jvnet.mimepull, [54]org.glassfish.jersey.core.jersey-server, [55]org.glassfish.jersey.ext.jersey-proxy-client, [56]org.glassfish.hk2.osgi-resource-locator, [57]javax.validation.api, [58]vespajlib, [59]configserver, [60]config-model-fat, [61]node-repository, [62]orchestrator, [63]application-model, [64]service-monitor}
1514056304.433  vespa-7dc749d59c-b72rd  278     jdisc/configserver      org.apache.felix.framework      info    FrameworkEvent PACKAGES REFRESHED
1514056308.725  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.metrics.simple.MetricManager        config  setting up simple metrics gathering. reportPeriodSeconds=60, pointsToKeepPerMetric=100
1514056308.862  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.vespa.config.server.zookeeper.ConfigCurator config  Using jute max buffer size 52428800
1514056308.877  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.statistics.Counter  event   count/1 name=configserver.requests value=0
1514056308.878  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.statistics.Counter  event   count/1 name=configserver.failedRequests value=0
1514056308.880  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.statistics.Counter  event   count/1 name=procTime value=0
1514056308.910  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.config.model.application.provider.FilesApplicationPackage   info    Adding user include dir 'config-models'
1514056309.545  vespa-7dc749d59c-b72rd  278/44  configserver    Container.com.yahoo.vespa.config.server.zookeeper.ConfigCurator config  Using jute max buffer size 10485760
1514056309.649  vespa-7dc749d59c-b72rd  278/1   configserver    Container.org.glassfish.jersey.internal.ServiceFinder   config  Running in an OSGi environment
1514056309.920  vespa-7dc749d59c-b72rd  278/164 configserver    Container.com.yahoo.vespa.config.server.ConfigServerBootstrap   info    Configserver upgraded from 0.0.0 to 6.10.0. Redeploying all applications
1514056309.930  vespa-7dc749d59c-b72rd  278/164 configserver    Container.com.yahoo.vespa.config.server.ConfigServerBootstrap   info    All applications redeployed
1514056309.931  vespa-7dc749d59c-b72rd  278/164 configserver    Container.com.yahoo.container.jdisc.state.StateMonitor  info    Changing health status code from 'initializing' to 'up'
1514056309.931  vespa-7dc749d59c-b72rd  278/164 configserver    Container.com.yahoo.vespa.config.server.rpc.RpcServer   info    Rpc server listening on port 19070
1514056310.469  vespa-7dc749d59c-b72rd  278/1   configserver    Container.org.eclipse.jetty.util.log    info    Logging initialized @16718ms to org.eclipse.jetty.util.log.Slf4jLog
1514056312.588  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.jdisc.http.server.jetty.JettyHttpServer     info    Creating janitor executor with 1 threads
1514056312.753  vespa-7dc749d59c-b72rd  527/26  configproxy     configproxy.com.yahoo.vespa.config.proxy.DelayedResponseHandler warning Timed out (timeout 15000) getting config name=sentinel,namespace=cloud.config,configId=hosts/vespa-7dc749d59c-b72rd, will retry
1514056312.933  vespa-7dc749d59c-b72rd  278/1   configserver    Container.org.eclipse.jetty.server.Server       info    jetty-9.4.8.v20171121, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
1514056313.573  vespa-7dc749d59c-b72rd  278/1   configserver    Container.org.glassfish.jersey.server.ApplicationHandler        config  Initiating Jersey application, version Jersey: 2.23.2 2016-08-08 17:14:55...
1514056314.408  vespa-7dc749d59c-b72rd  527/18  configproxy     configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester       warning Request callback failed: APPLICATION_NOT_LOADED. Connection spec: tcp/localhost:19070, error message: Failed request (No application exists) from Connection { Socket[addr=/127.0.0.1,port=43092,localport=19070] }
1514056314.641  vespa-7dc749d59c-b72rd  278/1   configserver    Container.org.glassfish.jersey.process.internal.ExecutorProviders       config  Selected ExecutorServiceProvider implementation [org.glassfish.jersey.server.internal.process.ServerProcessingBinder$DefaultManagedAsyncExecutorProvider] to be used for injection of executor qualified by [org.glassfish.jersey.server.ManagedAsyncExecutor] annotation.
1514056314.653  vespa-7dc749d59c-b72rd  278/1   configserver    Container.org.glassfish.jersey.process.internal.ExecutorProviders       config  Selected ScheduledExecutorServiceProvider implementation [org.glassfish.jersey.server.internal.process.ServerProcessingBinder$DefaultBackgroundSchedulerProvider] to be used for injection of scheduler qualified by [org.glassfish.jersey.server.BackgroundScheduler] annotation.
1514056314.851  vespa-7dc749d59c-b72rd  278/1   configserver    Container.org.glassfish.jersey.server.ApplicationHandler        config  Jersey application initialized.\nRoot Resource Classes:\n  com.yahoo.vespa.orchestrator.resources.InstanceResource\n  com.yahoo.vespa.orchestrator.resources.ApplicationSuspensionResource\n  com.yahoo.vespa.orchestrator.resources.HostResource\n  com.yahoo.vespa.orchestrator.resources.HostSuspensionResource\nGlobal Reader Interceptors:\n   org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor\nGlobal Writer Interceptors:\n   org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor\n   org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor\nMessage Body Readers:\n   org.glassfish.jersey.media.multipart.internal.MultiPartReaderServerSide\n   com.fasterxml.jackson.jaxrs.json.JacksonJaxbJsonProvider\nMessage Body Writers:\n   org.glassfish.jersey.media.multipart.internal.MultiPartWriter\n   com.fasterxml.jackson.jaxrs.json.JacksonJaxbJsonProvider\n
1514056314.852  vespa-7dc749d59c-b72rd  278/1   configserver    Container.org.glassfish.jersey.internal.Errors  config  The following hints have been detected: HINT: A HTTP GET method, public void com.yahoo.vespa.orchestrator.resources.ApplicationSuspensionResource.getApplication(java.lang.String), returns a void type. It can be intentional and perfectly fine, but it is a little uncommon that GET method returns always "204 No Content".\n
1514056314.916  vespa-7dc749d59c-b72rd  278/1   configserver    Container.org.glassfish.jersey.server.ApplicationHandler        config  Initiating Jersey application, version Jersey: 2.23.2 2016-08-08 17:14:55...
1514056315.107  vespa-7dc749d59c-b72rd  278/1   configserver    Container.org.glassfish.jersey.process.internal.ExecutorProviders       config  Selected ExecutorServiceProvider implementation [org.glassfish.jersey.server.internal.process.ServerProcessingBinder$DefaultManagedAsyncExecutorProvider] to be used for injection of executor qualified by [org.glassfish.jersey.server.ManagedAsyncExecutor] annotation.
1514056315.111  vespa-7dc749d59c-b72rd  278/1   configserver    Container.org.glassfish.jersey.process.internal.ExecutorProviders       config  Selected ScheduledExecutorServiceProvider implementation [org.glassfish.jersey.server.internal.process.ServerProcessingBinder$DefaultBackgroundSchedulerProvider] to be used for injection of scheduler qualified by [org.glassfish.jersey.server.BackgroundScheduler] annotation.
1514056315.141  vespa-7dc749d59c-b72rd  278/1   configserver    Container.org.glassfish.jersey.server.ApplicationHandler        config  Jersey application initialized.\n\nGlobal Reader Interceptors:\n   org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor\nGlobal Writer Interceptors:\n   org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor\n   org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor\nMessage Body Readers:\n   org.glassfish.jersey.media.multipart.internal.MultiPartReaderServerSide\n   com.fasterxml.jackson.jaxrs.json.JacksonJaxbJsonProvider\nMessage Body Writers:\n   org.glassfish.jersey.media.multipart.internal.MultiPartWriter\n   com.fasterxml.jackson.jaxrs.json.JacksonJaxbJsonProvider\n
1514056315.177  vespa-7dc749d59c-b72rd  278/1   configserver    Container.org.glassfish.jersey.server.ApplicationHandler        config  Initiating Jersey application, version Jersey: 2.23.2 2016-08-08 17:14:55...
1514056315.396  vespa-7dc749d59c-b72rd  278/1   configserver    Container.org.glassfish.jersey.process.internal.ExecutorProviders       config  Selected ExecutorServiceProvider implementation [org.glassfish.jersey.server.internal.process.ServerProcessingBinder$DefaultManagedAsyncExecutorProvider] to be used for injection of executor qualified by [org.glassfish.jersey.server.ManagedAsyncExecutor] annotation.
1514056315.397  vespa-7dc749d59c-b72rd  278/1   configserver    Container.org.glassfish.jersey.process.internal.ExecutorProviders       config  Selected ScheduledExecutorServiceProvider implementation [org.glassfish.jersey.server.internal.process.ServerProcessingBinder$DefaultBackgroundSchedulerProvider] to be used for injection of scheduler qualified by [org.glassfish.jersey.server.BackgroundScheduler] annotation.
1514056315.458  vespa-7dc749d59c-b72rd  278/1   configserver    Container.org.glassfish.jersey.server.ApplicationHandler        config  Jersey application initialized.\nRoot Resource Classes:\n  com.yahoo.vespa.serviceview.StateResource\nGlobal Reader Interceptors:\n   org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor\nGlobal Writer Interceptors:\n   org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor\n   org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor\nMessage Body Readers:\n   org.glassfish.jersey.media.multipart.internal.MultiPartReaderServerSide\n   com.fasterxml.jackson.jaxrs.json.JacksonJaxbJsonProvider\nMessage Body Writers:\n   org.glassfish.jersey.media.multipart.internal.MultiPartWriter\n   com.fasterxml.jackson.jaxrs.json.JacksonJaxbJsonProvider\n
1514056315.493  vespa-7dc749d59c-b72rd  278/1   configserver    Container.org.glassfish.jersey.server.ApplicationHandler        config  Initiating Jersey application, version Jersey: 2.23.2 2016-08-08 17:14:55...
1514056315.652  vespa-7dc749d59c-b72rd  278/1   configserver    Container.org.glassfish.jersey.process.internal.ExecutorProviders       config  Selected ExecutorServiceProvider implementation [org.glassfish.jersey.server.internal.process.ServerProcessingBinder$DefaultManagedAsyncExecutorProvider] to be used for injection of executor qualified by [org.glassfish.jersey.server.ManagedAsyncExecutor] annotation.
1514056315.653  vespa-7dc749d59c-b72rd  278/1   configserver    Container.org.glassfish.jersey.process.internal.ExecutorProviders       config  Selected ScheduledExecutorServiceProvider implementation [org.glassfish.jersey.server.internal.process.ServerProcessingBinder$DefaultBackgroundSchedulerProvider] to be used for injection of scheduler qualified by [org.glassfish.jersey.server.BackgroundScheduler] annotation.
1514056315.672  vespa-7dc749d59c-b72rd  278/1   configserver    Container.org.glassfish.jersey.server.ApplicationHandler        config  Jersey application initialized.\nRoot Resource Classes:\n  com.yahoo.vespa.config.server.restapi.impl.StatusResource\nGlobal Reader Interceptors:\n   org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor\nGlobal Writer Interceptors:\n   org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor\n   org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor\nMessage Body Readers:\n   org.glassfish.jersey.media.multipart.internal.MultiPartReaderServerSide\n   com.fasterxml.jackson.jaxrs.json.JacksonJaxbJsonProvider\nMessage Body Writers:\n   org.glassfish.jersey.media.multipart.internal.MultiPartWriter\n   com.fasterxml.jackson.jaxrs.json.JacksonJaxbJsonProvider\n
1514056315.678  vespa-7dc749d59c-b72rd  278/1   configserver    Container.org.eclipse.jetty.server.handler.ContextHandler       info    Started o.e.j.s.ServletContextHandler@6020cd42{/,null,AVAILABLE}
1514056315.681  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.jdisc.http.server.jetty.JDiscServerConnector        info    Using channel set by activator: sun.nio.ch.ServerSocketChannelImpl[/0:0:0:0:0:0:0:0:19071]
1514056315.724  vespa-7dc749d59c-b72rd  278/1   configserver    Container.org.eclipse.jetty.server.AbstractConnector    info    Started configserver@71ad9b74{HTTP/1.1,[http/1.1]}{0.0.0.0:19071}
1514056315.726  vespa-7dc749d59c-b72rd  278/1   configserver    Container.org.eclipse.jetty.server.Server       info    Started @21979ms
1514056315.726  vespa-7dc749d59c-b72rd  278/1   configserver    Container.com.yahoo.container.jdisc.ConfiguredApplication       info    Switching to the latest deployed set of configurations and components. Application switch number: 0
1514056317.093179       vespa-7dc749d59c-b72rd  653/3548        config-sentinel config-sentinel.config.frt.frtconnection        warning Connection to tcp/localhost:19090 failed or timed out
1514056317.093333       vespa-7dc749d59c-b72rd  653/3548        config-sentinel config-sentinel.config.frt.frtconnection        warning FRT Connection tcp/localhost:19090 suspended until 2017-12-23 19:12:07 GMT
1514056317.093375       vespa-7dc749d59c-b72rd  653/3548        config-sentinel config-sentinel.config.frt.frtconfigagent       info    Error response or no response from config server (key: name=sentinel,namespace=cloud.config,configId=hosts/vespa-7dc749d59c-b72rd) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1514056320.000  vespa-7dc749d59c-b72rd  278/35  configserver    Container.com.yahoo.statistics.Counter  event   count/1 name=configserver.requests value=0
1514056320.001  vespa-7dc749d59c-b72rd  278/35  configserver    Container.com.yahoo.statistics.Counter  event   count/1 name=configserver.failedRequests value=0
1514056320.001  vespa-7dc749d59c-b72rd  278/35  configserver    Container.com.yahoo.statistics.Counter  event   count/1 name=procTime value=0
1514056341.442  vespa-7dc749d59c-b72rd  527/12  configproxy     configproxy.com.yahoo.vespa.config.proxy.RpcConfigSourceClient  info    Subscribe for 'name=sentinel,namespace=cloud.config,configId=hosts/vespa-7dc749d59c-b72rd,0944a8c189a502c0e2fe1930114897b7' failed, closing subscriber
1514056353.107723       vespa-7dc749d59c-b72rd  653/3548        config-sentinel config-sentinel.config.frt.frtconnection        warning Connection to tcp/localhost:19090 failed or timed out
1514056353.107827       vespa-7dc749d59c-b72rd  653/3548        config-sentinel config-sentinel.config.frt.frtconnection        warning FRT Connection tcp/localhost:19090 suspended until 2017-12-23 19:12:53 GMT
1514056353.107854       vespa-7dc749d59c-b72rd  653/3548        config-sentinel config-sentinel.config.frt.frtconfigagent       info    Error response or no response from config server (key: name=sentinel,namespace=cloud.config,configId=hosts/vespa-7dc749d59c-b72rd) (errcode=103, validresponse:0), trying again in 7000 milliseconds
1514056366.752  vespa-7dc749d59c-b72rd  527/26  configproxy     configproxy.com.yahoo.vespa.config.proxy.DelayedResponseHandler warning Timed out (timeout 25000) getting config name=sentinel,namespace=cloud.config,configId=hosts/vespa-7dc749d59c-b72rd, will retry
1514056380.000  vespa-7dc749d59c-b72rd  278/35  configserver    Container.com.yahoo.statistics.Counter  event   count/1 name=procTime value=0
1514056380.001  vespa-7dc749d59c-b72rd  278/35  configserver    Container.com.yahoo.statistics.Counter  event   count/1 name=configserver.requests value=0
1514056380.001  vespa-7dc749d59c-b72rd  278/35  configserver    Container.com.yahoo.statistics.Counter  event   count/1 name=configserver.failedRequests value=0
1514056380.143  vespa-7dc749d59c-b72rd  527/18  configproxy     configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester       warning Request callback failed: APPLICATION_NOT_LOADED. Connection spec: tcp/localhost:19070, error message: Failed request (No application exists) from Connection { Socket[addr=/127.0.0.1,port=43092,localport=19070] }
1514056385.446  vespa-7dc749d59c-b72rd  527/12  configproxy     configproxy.com.yahoo.vespa.config.proxy.RpcConfigSourceClient  info    Subscribe for 'name=sentinel,namespace=cloud.config,configId=hosts/vespa-7dc749d59c-b72rd,0944a8c189a502c0e2fe1930114897b7' failed, closing subscriber
1514056390.131514       vespa-7dc749d59c-b72rd  653/3548        config-sentinel config-sentinel.config.frt.frtconnection        warning Connection to tcp/localhost:19090 failed or timed out
1514056390.131638       vespa-7dc749d59c-b72rd  653/3548        config-sentinel config-sentinel.config.frt.frtconnection        warning FRT Connection tcp/localhost:19090 suspended until 2017-12-23 19:13:40 GMT
1514056390.131667       vespa-7dc749d59c-b72rd  653/3548        config-sentinel config-sentinel.config.frt.frtconfigagent       info    Error response or no response from config server (key: name=sentinel,namespace=cloud.config,configId=hosts/vespa-7dc749d59c-b72rd) (errcode=103, validresponse:0), trying again in 8000 milliseconds
1514056410.752  vespa-7dc749d59c-b72rd  527/26  configproxy     configproxy.com.yahoo.vespa.config.proxy.DelayedResponseHandler warning Timed out (timeout 25000) getting config name=sentinel,namespace=cloud.config,configId=hosts/vespa-7dc749d59c-b72rd, will retry
1514056428.149751       vespa-7dc749d59c-b72rd  653/3548        config-sentinel config-sentinel.config.frt.frtconnection        warning Connection to tcp/localhost:19090 failed or timed out
1514056428.149860       vespa-7dc749d59c-b72rd  653/3548        config-sentinel config-sentinel.config.frt.frtconnection        warning FRT Connection tcp/localhost:19090 suspended until 2017-12-23 19:14:28 GMT
1514056428.149880       vespa-7dc749d59c-b72rd  653/3548        config-sentinel config-sentinel.config.frt.frtconfigagent       info    Error response or no response from config server (key: name=sentinel,namespace=cloud.config,configId=hosts/vespa-7dc749d59c-b72rd) (errcode=103, validresponse:0), trying again in 9000 milliseconds
1514056429.452  vespa-7dc749d59c-b72rd  527/12  configproxy     configproxy.com.yahoo.vespa.config.proxy.RpcConfigSourceClient  info    Subscribe for 'name=sentinel,namespace=cloud.config,configId=hosts/vespa-7dc749d59c-b72rd,0944a8c189a502c0e2fe1930114897b7' failed, closing subscriber
1514056440.000  vespa-7dc749d59c-b72rd  278/35  configserver    Container.com.yahoo.statistics.Counter  event   count/1 name=configserver.failedRequests value=0
1514056440.001  vespa-7dc749d59c-b72rd  278/35  configserver    Container.com.yahoo.statistics.Counter  event   count/1 name=procTime value=0
1514056440.001  vespa-7dc749d59c-b72rd  278/35  configserver    Container.com.yahoo.statistics.Counter  event   count/1 name=configserver.requests value=0
1514056444.113  vespa-7dc749d59c-b72rd  527/18  configproxy     configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester       warning Request callback failed: APPLICATION_NOT_LOADED. Connection spec: tcp/localhost:19070, error message: Failed request (No application exists) from Connection { Socket[addr=/127.0.0.1,port=43092,localport=19070] }
1514056454.752  vespa-7dc749d59c-b72rd  527/26  configproxy     configproxy.com.yahoo.vespa.config.proxy.DelayedResponseHandler warning Timed out (timeout 25000) getting config name=sentinel,namespace=cloud.config,configId=hosts/vespa-7dc749d59c-b72rd, will retry
1514056467.168358       vespa-7dc749d59c-b72rd  653/3548        config-sentinel config-sentinel.config.frt.frtconnection        warning Connection to tcp/localhost:19090 failed or timed out
1514056467.168459       vespa-7dc749d59c-b72rd  653/3548        config-sentinel config-sentinel.config.frt.frtconnection        warning FRT Connection tcp/localhost:19090 suspended until 2017-12-23 19:15:17 GMT
1514056467.168481       vespa-7dc749d59c-b72rd  653/3548        config-sentinel config-sentinel.config.frt.frtconfigagent       info    Error response or no response from config server (key: name=sentinel,namespace=cloud.config,configId=hosts/vespa-7dc749d59c-b72rd) (errcode=103, validresponse:0), trying again in 10000 milliseconds
1514056473.462  vespa-7dc749d59c-b72rd  527/12  configproxy     configproxy.com.yahoo.vespa.config.proxy.RpcConfigSourceClient  info    Subscribe for 'name=sentinel,namespace=cloud.config,configId=hosts/vespa-7dc749d59c-b72rd,0944a8c189a502c0e2fe1930114897b7' failed, closing subscriber
1514056477.079951       vespa-7dc749d59c-b72rd  653/32935       config-sentinel config-sentinel warning Timout getting config, please check your setup. Will exit and restart: Timed out while subscribing to 'cloud.config.sentinel', configid 'hosts/vespa-7dc749d59c-b72rd'
1514056477.080282       vespa-7dc749d59c-b72rd  653/32935       config-sentinel config-sentinel event   stopping/1 name="config-sentinel" why="ConfigTimeoutException: Timed out while subscribing to 'cloud.config.sentinel', configid 'hosts/vespa-7dc749d59c-b72rd'\nBacktrace:\n  /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigTimeoutException::ConfigTimeoutException(vespalib::stringref const&, vespalib::stringref const&, int)+0x15) [0x7f87c9004255]\n  /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigManager::subscribe(config::ConfigKey const&, unsigned long)+0x786) [0x7f87c8ff4a56]\n  /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigSubscriptionSet::subscribe(config::ConfigKey const&, unsigned long)+0x4b) [0x7f87c9007afb]\n  sbin/vespa-config-sentinel(std::unique_ptr<config::ConfigHandle<cloud::config::internal::InternalSentinelType const>, std::default_delete<config::ConfigHandle<cloud::config::internal::InternalSentinelType const> > > config::ConfigSubscriber::subscribe<cloud::config::internal::InternalSentinelType const>(std::string const&, unsigned long)+0xca) [0x40ca6a]\n  sbin/vespa-config-sentinel(config::sentinel::ConfigHandler::subscribe(std::string const&, unsigned long)+0x25) [0x409805]\n  sbin/vespa-config-sentinel(main+0x139) [0x405fd9]\n  /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f87c64c9c05]\n  sbin/vespa-config-sentinel() [0x40670a]"
1514056477.081157       vespa-7dc749d59c-b72rd  651/3287        config-sentinel runserver       event   stopped/1 name="sbin/vespa-config-sentinel -c hosts/vespa-7dc749d59c-b72rd (pid 653)" pid=653 exitcode=1
1514056477.081202       vespa-7dc749d59c-b72rd  651/3287        config-sentinel runserver       info    will restart in 0 seconds
1514056477.081369       vespa-7dc749d59c-b72rd  651/3287        config-sentinel runserver       event   starting/1 name="sbin/vespa-config-sentinel -c hosts/vespa-7dc749d59c-b72rd (pid 866)"
1514056477.091588       vespa-7dc749d59c-b72rd  866/22163       config-sentinel config-sentinel event   started/1 name="config-sentinel"
1514056497.110773       vespa-7dc749d59c-b72rd  866/58311       config-sentinel config-sentinel.config.frt.frtconnection        warning Connection to tcp/localhost:19090 failed or timed out
1514056497.110942       vespa-7dc749d59c-b72rd  866/58311       config-sentinel config-sentinel.config.frt.frtconnection        warning FRT Connection tcp/localhost:19090 suspended until 2017-12-23 19:15:07 GMT
1514056497.110983       vespa-7dc749d59c-b72rd  866/58311       config-sentinel config-sentinel.config.frt.frtconfigagent       info    Error response or no response from config server (key: name=sentinel,namespace=cloud
.config,configId=hosts/vespa-7dc749d59c-b72rd) (errcode=103, validresponse:0), trying again in 6000 milliseconds

netstat

root@vespa-7dc749d59c-b72rd vespa]# netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 localhost:43864         localhost:19090         ESTABLISHED
tcp6    5060      0 localhost:19090         localhost:43864         ESTABLISHED
tcp6       0      0 vespa-7dc749d5:eforward vespa-7dc749d59c-:60462 ESTABLISHED
tcp6       0      0 vespa-7dc749d59c-:60462 vespa-7dc749d5:eforward ESTABLISHED
tcp6       0      0 localhost:43092         localhost:19070         ESTABLISHED
tcp6       0      0 localhost:19070         localhost:43092         ESTABLISHED

@lundin
Copy link
Author

lundin commented Dec 23, 2017

Closing. Configserver is working if exec into the container and using gce ingress rules with default backend.

@lundin lundin closed this as completed Dec 23, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@lundin