Knative services are slow to become ready #1206

nicolaferraro · 2020-01-15T16:35:59Z

This is an old problem that sometimes comes back: knative/serving#5802

I wonder if it's a configuration issue from our side, because it does not happen e.g. with other Knative quickstarts

kameshsampath · 2020-01-15T16:39:44Z

@bbrowning @markusthoemmes - do you have any clue ?

kameshsampath · 2020-01-15T16:43:57Z

This is a service that became ready after several attempts or long waiting

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  annotations:
    serving.knative.dev/creator: system:serviceaccount:chapter-7:camel-k-operator
    serving.knative.dev/lastModifier: system:serviceaccount:chapter-7:camel-k-operator
  creationTimestamp: "2020-01-15T16:18:00Z"
  generation: 1
  labels:
    camel.apache.org/generation: "1"
    camel.apache.org/integration: echoer
  name: echoer
  namespace: chapter-7
  ownerReferences:
  - apiVersion: camel.apache.org/v1
    blockOwnerDeletion: true
    controller: true
    kind: Integration
    name: echoer
    uid: 994805f5-37b2-11ea-88ab-08002762facf
  resourceVersion: "40303"
  selfLink: /apis/serving.knative.dev/v1/namespaces/chapter-7/services/echoer
  uid: 996b3a32-37b2-11ea-88ab-08002762facf
spec:
  template:
    metadata:
      annotations:
        alpha.image.policy.openshift.io/resolve-names: '*'
      creationTimestamp: null
      labels:
        camel.apache.org/generation: "1"
        camel.apache.org/integration: echoer
    spec:
      containerConcurrency: 0
      containers:
      - env:
        - name: CAMEL_K_DIGEST
          value: vbKLnxIfkpcy8WZQBEvcnpE5-gb6bvPbgZSOkOhS36ZA
        - name: CAMEL_K_ROUTES
          value: file:/etc/camel/sources/i-source-000/echoer.yaml?language=yaml
        - name: CAMEL_K_CONF
          value: /etc/camel/conf/application.properties
        - name: CAMEL_K_CONF_D
          value: /etc/camel/conf.d
        - name: CAMEL_KNATIVE_CONFIGURATION
          value: '{"services":[{"type":"endpoint","name":"echoer","host":"0.0.0.0","port":8080,"metadata":{"camel.endpoint.kind":"source","knative.apiVersion":"serving.knative.dev/v1","knative.kind":"Service","service.path":"/"}}]}'
        - name: CAMEL_K_VERSION
          value: 1.0.0-RC1
        - name: CAMEL_K_RUNTIME_VERSION
          value: 1.0.9
        - name: CAMEL_VERSION
          value: 3.0.0
        - name: NAMESPACE
          value: chapter-7
        - name: JAVA_MAIN_CLASS
          value: org.apache.camel.k.main.Application
        - name: AB_JOLOKIA_OFF
          value: "true"
        - name: AB_PROMETHEUS_OFF
          value: "true"
        - name: JAVA_CLASSPATH
          value: ./resources:/etc/camel/conf:/etc/camel/resources:/etc/camel/sources/i-source-000:dependencies/com.fasterxml.jackson.core.jackson-annotations-2.10.1.jar:dependencies/com.fasterxml.jackson.core.jackson-core-2.10.1.jar:dependencies/com.fasterxml.jackson.core.jackson-databind-2.10.1.jar:dependencies/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.1.jar:dependencies/com.fasterxml.jackson.datatype.jackson-datatype-jdk8-2.10.1.jar:dependencies/com.github.ben-manes.caffeine.caffeine-2.8.0.jar:dependencies/com.sun.xml.bind.jaxb-core-2.3.0.1.jar:dependencies/com.sun.xml.bind.jaxb-impl-2.3.0.jar:dependencies/commons-io.commons-io-2.6.jar:dependencies/io.netty.netty-buffer-4.1.42.Final.jar:dependencies/io.netty.netty-codec-4.1.42.Final.jar:dependencies/io.netty.netty-codec-dns-4.1.42.Final.jar:dependencies/io.netty.netty-codec-http-4.1.42.Final.jar:dependencies/io.netty.netty-codec-http2-4.1.42.Final.jar:dependencies/io.netty.netty-codec-socks-4.1.42.Final.jar:dependencies/io.netty.netty-common-4.1.42.Final.jar:dependencies/io.netty.netty-handler-4.1.42.Final.jar:dependencies/io.netty.netty-handler-proxy-4.1.42.Final.jar:dependencies/io.netty.netty-resolver-4.1.42.Final.jar:dependencies/io.netty.netty-resolver-dns-4.1.42.Final.jar:dependencies/io.netty.netty-transport-4.1.42.Final.jar:dependencies/io.vertx.vertx-auth-common-3.8.3.jar:dependencies/io.vertx.vertx-bridge-common-3.8.3.jar:dependencies/io.vertx.vertx-core-3.8.3.jar:dependencies/io.vertx.vertx-web-3.8.3.jar:dependencies/io.vertx.vertx-web-client-3.8.3.jar:dependencies/io.vertx.vertx-web-common-3.8.3.jar:dependencies/javax.xml.bind.jaxb-api-2.3.0.jar:dependencies/org.apache.camel.camel-api-3.0.0.jar:dependencies/org.apache.camel.camel-base-3.0.0.jar:dependencies/org.apache.camel.camel-bean-3.0.0.jar:dependencies/org.apache.camel.camel-caffeine-lrucache-3.0.0.jar:dependencies/org.apache.camel.camel-cloud-3.0.0.jar:dependencies/org.apache.camel.camel-core-engine-3.0.0.jar:dependencies/org.apache.camel.camel-jaxp-3.0.0.jar:dependencies/org.apache.camel.camel-log-3.0.0.jar:dependencies/org.apache.camel.camel-main-3.0.0.jar:dependencies/org.apache.camel.camel-management-api-3.0.0.jar:dependencies/org.apache.camel.camel-support-3.0.0.jar:dependencies/org.apache.camel.camel-util-3.0.0.jar:dependencies/org.apache.camel.camel-util-json-3.0.0.jar:dependencies/org.apache.camel.k.camel-k-loader-yaml-1.0.9.jar:dependencies/org.apache.camel.k.camel-k-loader-yaml-common-1.0.9.jar:dependencies/org.apache.camel.k.camel-k-runtime-core-1.0.9.jar:dependencies/org.apache.camel.k.camel-k-runtime-main-1.0.9.jar:dependencies/org.apache.camel.k.camel-knative-1.0.9.jar:dependencies/org.apache.camel.k.camel-knative-api-1.0.9.jar:dependencies/org.apache.camel.k.camel-knative-http-1.0.9.jar:dependencies/org.apache.camel.spi-annotations-3.0.0.jar:dependencies/org.apache.commons.commons-collections4-4.4.jar:dependencies/org.apache.commons.commons-lang3-3.9.jar:dependencies/org.apache.logging.log4j.log4j-api-2.13.0.jar:dependencies/org.apache.logging.log4j.log4j-core-2.13.0.jar:dependencies/org.apache.logging.log4j.log4j-slf4j-impl-2.13.0.jar:dependencies/org.slf4j.slf4j-api-1.7.30.jar:dependencies/org.yaml.snakeyaml-1.24.jar
        image: 10.111.197.108/chapter-7/camel-k-kit-bofj39ei1aal47gim830:36212
        name: integration
        readinessProbe:
          successThreshold: 1
          tcpSocket:
            port: 0
        resources: {}
        volumeMounts:
        - mountPath: /etc/camel/sources/i-source-000
          name: i-source-000
          readOnly: true
        - mountPath: /etc/camel/conf
          name: integration-properties
          readOnly: true
      timeoutSeconds: 300
      volumes:
      - configMap:
          items:
          - key: content
            path: echoer.yaml
          name: echoer-source-000
        name: i-source-000
      - configMap:
          items:
          - key: application.properties
            path: application.properties
          name: echoer-properties
        name: integration-properties
  traffic:
  - latestRevision: true
    percent: 100
status:
  address:
    url: http://echoer.chapter-7.svc.cluster.local
  conditions:
  - lastTransitionTime: "2020-01-15T16:20:39Z"
    status: "True"
    type: ConfigurationsReady
  - lastTransitionTime: "2020-01-15T16:20:40Z"
    status: "True"
    type: Ready
  - lastTransitionTime: "2020-01-15T16:20:40Z"
    status: "True"
    type: RoutesReady
  latestCreatedRevisionName: echoer-xz9ns
  latestReadyRevisionName: echoer-xz9ns
  observedGeneration: 1
  traffic:
  - latestRevision: true
    percent: 100
    revisionName: echoer-xz9ns
  url: http://echoer.chapter-7.example.com```

kameshsampath · 2020-01-15T17:13:51Z

This is another service that just created which is waiting for revision to be come ready

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  annotations:
    serving.knative.dev/creator: system:serviceaccount:chapter-7:camel-k-operator
    serving.knative.dev/lastModifier: system:serviceaccount:chapter-7:camel-k-operator
  creationTimestamp: "2020-01-15T17:06:18Z"
  generation: 1
  labels:
    camel.apache.org/generation: "1"
    camel.apache.org/integration: fruits-producer
  name: fruits-producer
  namespace: chapter-7
  ownerReferences:
  - apiVersion: camel.apache.org/v1
    blockOwnerDeletion: true
    controller: true
    kind: Integration
    name: fruits-producer
    uid: 64c2aa1d-37b7-11ea-88ab-08002762facf
  resourceVersion: "44500"
  selfLink: /apis/serving.knative.dev/v1/namespaces/chapter-7/services/fruits-producer
  uid: 58b90755-37b9-11ea-88ab-08002762facf
spec:
  template:
    metadata:
      annotations:
        alpha.image.policy.openshift.io/resolve-names: '*'
      creationTimestamp: null
      labels:
        camel.apache.org/generation: "1"
        camel.apache.org/integration: fruits-producer
    spec:
      containerConcurrency: 0
      containers:
      - env:
        - name: CAMEL_K_DIGEST
          value: vjMN8Zj3sW6W0fGvp-EEIaPP220wEGxA4theJo0CjDZQ
        - name: CAMEL_K_ROUTES
          value: file:/etc/camel/sources/i-source-000/fruits-producer.yaml?language=yaml
        - name: CAMEL_K_CONF
          value: /etc/camel/conf/application.properties
        - name: CAMEL_K_CONF_D
          value: /etc/camel/conf.d
        - name: CAMEL_KNATIVE_CONFIGURATION
          value: '{"services":[{"type":"endpoint","name":"fruits-producer","host":"0.0.0.0","port":8080,"metadata":{"camel.endpoint.kind":"source","knative.apiVersion":"serving.knative.dev/v1","knative.kind":"Service","service.path":"/"}}]}'
        - name: CAMEL_K_VERSION
          value: 1.0.0-RC1
        - name: CAMEL_K_RUNTIME_VERSION
          value: 1.0.9
        - name: CAMEL_VERSION
          value: 3.0.0
        - name: NAMESPACE
          value: chapter-7
        - name: JAVA_MAIN_CLASS
          value: org.apache.camel.k.main.Application
        - name: AB_JOLOKIA_OFF
          value: "true"
        - name: AB_PROMETHEUS_OFF
          value: "true"
        - name: JAVA_CLASSPATH
          value: ./resources:/etc/camel/conf:/etc/camel/resources:/etc/camel/sources/i-source-000:dependencies/com.fasterxml.jackson.core.jackson-annotations-2.10.1.jar:dependencies/com.fasterxml.jackson.core.jackson-core-2.10.1.jar:dependencies/com.fasterxml.jackson.core.jackson-databind-2.10.1.jar:dependencies/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.1.jar:dependencies/com.fasterxml.jackson.datatype.jackson-datatype-jdk8-2.10.1.jar:dependencies/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.10.1.jar:dependencies/com.github.ben-manes.caffeine.caffeine-2.8.0.jar:dependencies/com.github.luben.zstd-jni-1.4.0-1.jar:dependencies/com.jayway.jsonpath.json-path-2.4.0.jar:dependencies/com.sun.activation.javax.activation-1.2.0.jar:dependencies/com.sun.xml.bind.jaxb-core-2.3.0.1.jar:dependencies/com.sun.xml.bind.jaxb-impl-2.3.0.jar:dependencies/commons-codec.commons-codec-1.11.jar:dependencies/commons-io.commons-io-2.6.jar:dependencies/commons-logging.commons-logging-1.2.jar:dependencies/io.netty.netty-buffer-4.1.42.Final.jar:dependencies/io.netty.netty-codec-4.1.42.Final.jar:dependencies/io.netty.netty-codec-dns-4.1.42.Final.jar:dependencies/io.netty.netty-codec-http-4.1.42.Final.jar:dependencies/io.netty.netty-codec-http2-4.1.42.Final.jar:dependencies/io.netty.netty-codec-socks-4.1.42.Final.jar:dependencies/io.netty.netty-common-4.1.42.Final.jar:dependencies/io.netty.netty-handler-4.1.42.Final.jar:dependencies/io.netty.netty-handler-proxy-4.1.42.Final.jar:dependencies/io.netty.netty-resolver-4.1.42.Final.jar:dependencies/io.netty.netty-resolver-dns-4.1.42.Final.jar:dependencies/io.netty.netty-transport-4.1.42.Final.jar:dependencies/io.vertx.vertx-auth-common-3.8.3.jar:dependencies/io.vertx.vertx-bridge-common-3.8.3.jar:dependencies/io.vertx.vertx-core-3.8.3.jar:dependencies/io.vertx.vertx-web-3.8.3.jar:dependencies/io.vertx.vertx-web-client-3.8.3.jar:dependencies/io.vertx.vertx-web-common-3.8.3.jar:dependencies/jakarta.activation.jakarta.activation-api-1.2.1.jar:dependencies/jakarta.xml.bind.jakarta.xml.bind-api-2.3.2.jar:dependencies/javax.servlet.javax.servlet-api-3.1.0.jar:dependencies/javax.xml.bind.jaxb-api-2.3.0.jar:dependencies/net.minidev.accessors-smart-1.2.jar:dependencies/net.minidev.json-smart-2.3.jar:dependencies/net.minidev.json-smart-action-2.3.jar:dependencies/org.apache.camel.camel-api-3.0.0.jar:dependencies/org.apache.camel.camel-attachments-3.0.0.jar:dependencies/org.apache.camel.camel-base-3.0.0.jar:dependencies/org.apache.camel.camel-caffeine-lrucache-3.0.0.jar:dependencies/org.apache.camel.camel-cloud-3.0.0.jar:dependencies/org.apache.camel.camel-core-engine-3.0.0.jar:dependencies/org.apache.camel.camel-file-3.0.0.jar:dependencies/org.apache.camel.camel-http-3.0.0.jar:dependencies/org.apache.camel.camel-http-common-3.0.0.jar:dependencies/org.apache.camel.camel-jackson-3.0.0.jar:dependencies/org.apache.camel.camel-jaxp-3.0.0.jar:dependencies/org.apache.camel.camel-jsonpath-3.0.0.jar:dependencies/org.apache.camel.camel-kafka-3.0.0.jar:dependencies/org.apache.camel.camel-log-3.0.0.jar:dependencies/org.apache.camel.camel-main-3.0.0.jar:dependencies/org.apache.camel.camel-management-api-3.0.0.jar:dependencies/org.apache.camel.camel-support-3.0.0.jar:dependencies/org.apache.camel.camel-util-3.0.0.jar:dependencies/org.apache.camel.camel-util-json-3.0.0.jar:dependencies/org.apache.camel.k.camel-k-loader-yaml-1.0.9.jar:dependencies/org.apache.camel.k.camel-k-loader-yaml-common-1.0.9.jar:dependencies/org.apache.camel.k.camel-k-runtime-core-1.0.9.jar:dependencies/org.apache.camel.k.camel-k-runtime-main-1.0.9.jar:dependencies/org.apache.camel.k.camel-knative-1.0.9.jar:dependencies/org.apache.camel.k.camel-knative-api-1.0.9.jar:dependencies/org.apache.camel.k.camel-knative-http-1.0.9.jar:dependencies/org.apache.camel.spi-annotations-3.0.0.jar:dependencies/org.apache.commons.commons-collections4-4.4.jar:dependencies/org.apache.commons.commons-lang3-3.9.jar:dependencies/org.apache.httpcomponents.httpclient-4.5.10.jar:dependencies/org.apache.httpcomponents.httpcore-4.4.12.jar:dependencies/org.apache.kafka.kafka-clients-2.3.1.jar:dependencies/org.apache.logging.log4j.log4j-api-2.13.0.jar:dependencies/org.apache.logging.log4j.log4j-core-2.13.0.jar:dependencies/org.apache.logging.log4j.log4j-slf4j-impl-2.13.0.jar:dependencies/org.lz4.lz4-java-1.6.0.jar:dependencies/org.ow2.asm.asm-5.0.4.jar:dependencies/org.slf4j.slf4j-api-1.7.30.jar:dependencies/org.xerial.snappy.snappy-java-1.1.7.3.jar:dependencies/org.yaml.snakeyaml-1.24.jar
        image: 10.111.197.108/chapter-7/camel-k-kit-bofk6gui1aal47gim840:43338
        name: integration
        readinessProbe:
          successThreshold: 1
          tcpSocket:
            port: 0
        resources: {}
        volumeMounts:
        - mountPath: /etc/camel/sources/i-source-000
          name: i-source-000
          readOnly: true
        - mountPath: /etc/camel/conf
          name: integration-properties
          readOnly: true
      timeoutSeconds: 300
      volumes:
      - configMap:
          items:
          - key: content
            path: fruits-producer.yaml
          name: fruits-producer-source-000
        name: i-source-000
      - configMap:
          items:
          - key: application.properties
            path: application.properties
          name: fruits-producer-properties
        name: integration-properties
  traffic:
  - latestRevision: true
    percent: 100
status:
  conditions:
  - lastTransitionTime: "2020-01-15T17:06:18Z"
    status: Unknown
    type: ConfigurationsReady
  - lastTransitionTime: "2020-01-15T17:06:18Z"
    message: Configuration "fruits-producer" is waiting for a Revision to become ready.
    reason: RevisionMissing
    status: Unknown
    type: Ready
  - lastTransitionTime: "2020-01-15T17:06:18Z"
    message: Configuration "fruits-producer" is waiting for a Revision to become ready.
    reason: RevisionMissing
    status: Unknown
    type: RoutesReady
  latestCreatedRevisionName: fruits-producer-rh8w5
  observedGeneration: 1
  url: http://fruits-producer.chapter-7.example.com

markusthoemmes · 2020-01-15T17:29:54Z

Can you inspect the pods launched themselves? How long do they take to come up and how long does the application container need to come up and be ready specifically?

kameshsampath · 2020-01-16T03:44:01Z

The time to build the app image and deploy the pod usually takes a bit of time (this is a camel-k optimisation issue), but even after I see the ksvc deployment pods come up the ksvc reports "RevisionMissing" and its hung there for more than 20 mins.

the ksvc pods also terminates after the grace period but still the "RevisionMissing" status is reported.
But the service is never reachable as its not ready

nicolaferraro · 2020-02-11T15:48:50Z

I think I've isolated the issue.

This kservice is fast to start:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  labels:
    camel.apache.org/generation: "1"
    camel.apache.org/integration: fast-printer
  name: fast-printer
  namespace: default
spec:
  template:
    metadata:
      labels:
        camel.apache.org/generation: "1"
        camel.apache.org/integration: fast-printer
    spec:
      containerConcurrency: 0
      containers:
      - args:
        - org.apache.camel.k.main.Application
        command:
        - java
        image: camelk/camel-k-kit-bp1c6ngo4v80887bt9h0:9311
        name: integration
        readinessProbe:
          successThreshold: 1
          tcpSocket:
            port: 0
        workingDir: /deployments
      timeoutSeconds: 300
  traffic:
  - latestRevision: true
    percent: 100

This one is slow:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  labels:
    camel.apache.org/generation: "1"
    camel.apache.org/integration: fast-printer
  name: fast-printer
  namespace: default
spec:
  template:
    metadata:
      labels:
        camel.apache.org/generation: "1"
        camel.apache.org/integration: fast-printer
    spec:
      containerConcurrency: 0
      containers:
      - args:
        - org.apache.camel.k.main.Application
        command:
        - java
        image: 10.107.75.139/default/camel-k-kit-bp1c6ngo4v80887bt9h0:9311
        name: integration
        readinessProbe:
          successThreshold: 1
          tcpSocket:
            port: 0
        workingDir: /deployments
      timeoutSeconds: 300
  traffic:
  - latestRevision: true
    percent: 100

The only difference is that (the fast) one uses an image from docker.io, while the other uses the Minikube internal registry 10.107.75.139.
The serving controller tries to do something with that registry before starting the ksvc and remains stale.

My best guess is that the serving controller tries to contact it on https port 443, but the service never replies on that port (in fact curl reaches a timeout) and the controller remains waiting forever.

Does this sound familiar to you @markusthoemmes ?

markusthoemmes · 2020-02-11T15:56:51Z

Not necessarily familiar, but certainly sounds like something that could cause this. You can doublecheck if it works if you use the internal registry but skip it's ip for tag resolution in the config.

See https://github.com/knative/serving/blob/master/config/core/configmaps/deployment.yaml#L45

nicolaferraro · 2020-02-11T16:04:43Z

Yes, it is faster. Also if I provide the sha in the image id it starts immediately.

I remember someone telling me that registries referenced via ip addresses were considered insecure by default, but it seems not the case..

kameshsampath · 2020-02-12T11:17:52Z

@nicolaferraro - great to find the cause, I tried to make sure the IP is not used for internal registry by https://github.com/kameshsampath/minikube-helpers/tree/master/registry, I am due to submit a patch to minikube with it.

nicolaferraro · 2020-02-20T23:07:53Z

Fixed by #1259

nicolaferraro added this to the 1.0.0-RC2 milestone Jan 15, 2020

nicolaferraro added the area/knative Related to Knative label Jan 15, 2020

nicolaferraro closed this as completed Feb 20, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Knative services are slow to become ready #1206

Knative services are slow to become ready #1206

nicolaferraro commented Jan 15, 2020

kameshsampath commented Jan 15, 2020 •

edited

Loading

kameshsampath commented Jan 15, 2020 •

edited

Loading

kameshsampath commented Jan 15, 2020

markusthoemmes commented Jan 15, 2020

kameshsampath commented Jan 16, 2020

nicolaferraro commented Feb 11, 2020

markusthoemmes commented Feb 11, 2020

nicolaferraro commented Feb 11, 2020

kameshsampath commented Feb 12, 2020

nicolaferraro commented Feb 20, 2020

Knative services are slow to become ready #1206

Knative services are slow to become ready #1206

Comments

nicolaferraro commented Jan 15, 2020

kameshsampath commented Jan 15, 2020 • edited Loading

kameshsampath commented Jan 15, 2020 • edited Loading

kameshsampath commented Jan 15, 2020

markusthoemmes commented Jan 15, 2020

kameshsampath commented Jan 16, 2020

nicolaferraro commented Feb 11, 2020

markusthoemmes commented Feb 11, 2020

nicolaferraro commented Feb 11, 2020

kameshsampath commented Feb 12, 2020

nicolaferraro commented Feb 20, 2020

kameshsampath commented Jan 15, 2020 •

edited

Loading

kameshsampath commented Jan 15, 2020 •

edited

Loading