knative channel consumer stuck in running state #1222

weimeilin79 · 2020-01-22T16:57:17Z

Problem when starting a consumer for a inmemeorychannel, the integration get stuck in the running state. And I found this error in the knative eventing controller pod

{"level":"info","ts":"2020-01-22T15:53:30.034Z",
"logger":"controller.subscription-controller.event-broadcaster",
"caller":"record/event.go:258"
,"msg":"Event(v1.ObjectReference{Kind:\"Subscription\", Namespace:\"anewusecase\", Name:\"bloodpressure-test-reply\", UID:\"494a6d21-3d2f-11ea-946d-0ad1811f81cd\", APIVersion:\"messaging.knative.dev/v1alpha1\", ResourceVersion:\"680328\", FieldPath:\"\"}): 
type: 'Warning' reason: 'SubscriberResolveFailed' Failed to resolve spec.subscriber: address not set for &ObjectReference{Kind:Service,Namespace:anewusecase

Here are the details:
Channel:

oc get inmemorychannels.messaging.knative.dev bloodpressure -o yaml
apiVersion: messaging.knative.dev/v1alpha1
kind: InMemoryChannel
metadata:
  annotations:
    messaging.knative.dev/creator: opentlc-mgr
    messaging.knative.dev/lastModifier: opentlc-mgr
  creationTimestamp: 2020-01-22T15:02:41Z
  generation: 1
  name: bloodpressure
  namespace: anewusecase
  resourceVersion: "644692"
  selfLink: /apis/messaging.knative.dev/v1alpha1/namespaces/anewusecase/inmemorychannels/bloodpressure
  uid: 3ce5512a-3d28-11ea-946d-0ad1811f81cd
spec: {}
status:
  address:
    hostname: bloodpressure-kn-channel.anewusecase.svc.cluster.local
    url: http://bloodpressure-kn-channel.anewusecase.svc.cluster.local
  conditions:
  - lastTransitionTime: 2020-01-22T15:02:41Z
    status: "True"
    type: Addressable
  - lastTransitionTime: 2020-01-22T15:02:41Z
    status: "True"
    type: ChannelServiceReady
  - lastTransitionTime: 2020-01-22T15:02:41Z
    status: "True"
    type: DispatcherReady
  - lastTransitionTime: 2020-01-22T15:02:41Z
    status: "True"
    type: EndpointsReady
  - lastTransitionTime: 2020-01-22T15:02:41Z
    status: "True"
    type: Ready
  - lastTransitionTime: 2020-01-22T15:02:41Z
    status: "True"
    type: ServiceReady

kservice of the consumer :

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  annotations:
    serving.knative.dev/creator: system:serviceaccount:openshift-operators:camel-k-operator
    serving.knative.dev/lastModifier: system:serviceaccount:openshift-operators:camel-k-operator
  creationTimestamp: 2020-01-22T16:49:15Z
  generation: 1
  labels:
    camel.apache.org/generation: "1"
    camel.apache.org/integration: test-reply
  name: test-reply
  namespace: anewusecase
  ownerReferences:
  - apiVersion: camel.apache.org/v1
    blockOwnerDeletion: true
    controller: true
    kind: Integration
    name: test-reply
    uid: 1ad8668c-3d37-11ea-9b99-0205f72f8fa9
  resourceVersion: "718851"
  selfLink: /apis/serving.knative.dev/v1/namespaces/anewusecase/services/test-reply
  uid: 1fa7ee74-3d37-11ea-946d-0ad1811f81cd
spec:
  template:
    metadata:
      annotations:
        alpha.image.policy.openshift.io/resolve-names: '*'
      creationTimestamp: null
      labels:
        camel.apache.org/generation: "1"
        camel.apache.org/integration: test-reply
    spec:
      containerConcurrency: 0
      containers:
      - env:
        - name: CAMEL_K_DIGEST
          value: vHviqCqC5aNO_nGoOm3HZBo9Jdr_lle0LEXBjt5a-6C4
        - name: CAMEL_K_ROUTES
          value: file:/etc/camel/sources/i-source-000/TestReply.groovy?language=groovy
        - name: CAMEL_K_CONF
          value: /etc/camel/conf/application.properties
        - name: CAMEL_K_CONF_D
          value: /etc/camel/conf.d
        - name: CAMEL_KNATIVE_CONFIGURATION
          value: '{"services":[{"type":"channel","name":"bloodpressure","host":"0.0.0.0","port":8080,"metadata":{"camel.endpoint.kind":"source","knative.apiVersion":"messaging.knative.dev/v1alpha1","knative.kind":"InMemoryChannel","service.path":"/"}}]}'
        - name: CAMEL_K_VERSION
          value: 1.0.0-RC1
        - name: CAMEL_K_RUNTIME_VERSION
          value: 1.0.9
        - name: CAMEL_VERSION
          value: 3.0.0
        - name: NAMESPACE
          value: anewusecase
        - name: JAVA_MAIN_CLASS
          value: org.apache.camel.k.main.Application
        - name: AB_JOLOKIA_OFF
          value: "true"
        - name: AB_PROMETHEUS_OFF
          value: "true"
        - name: JAVA_CLASSPATH
          value: ./resources:/etc/camel/conf:/etc/camel/resources:/etc/camel/sources/i-source-000:dependencies/com.fasterxml.jackson.core.jackson-annotations-2.10.1.jar:dependencies/com.fasterxml.jackson.core.jackson-core-2.10.1.jar:dependencies/com.fasterxml.jackson.core.jackson-databind-2.10.1.jar:dependencies/com.fasterxml.jackson.datatype.jackson-datatype-jdk8-2.10.1.jar:dependencies/com.github.ben-manes.caffeine.caffeine-2.8.0.jar:dependencies/com.sun.xml.bind.jaxb-core-2.3.0.1.jar:dependencies/com.sun.xml.bind.jaxb-impl-2.3.0.jar:dependencies/commons-io.commons-io-2.6.jar:dependencies/io.netty.netty-buffer-4.1.42.Final.jar:dependencies/io.netty.netty-codec-4.1.42.Final.jar:dependencies/io.netty.netty-codec-dns-4.1.42.Final.jar:dependencies/io.netty.netty-codec-http-4.1.42.Final.jar:dependencies/io.netty.netty-codec-http2-4.1.42.Final.jar:dependencies/io.netty.netty-codec-socks-4.1.42.Final.jar:dependencies/io.netty.netty-common-4.1.42.Final.jar:dependencies/io.netty.netty-handler-4.1.42.Final.jar:dependencies/io.netty.netty-handler-proxy-4.1.42.Final.jar:dependencies/io.netty.netty-resolver-4.1.42.Final.jar:dependencies/io.netty.netty-resolver-dns-4.1.42.Final.jar:dependencies/io.netty.netty-transport-4.1.42.Final.jar:dependencies/io.vertx.vertx-auth-common-3.8.3.jar:dependencies/io.vertx.vertx-bridge-common-3.8.3.jar:dependencies/io.vertx.vertx-core-3.8.3.jar:dependencies/io.vertx.vertx-web-3.8.3.jar:dependencies/io.vertx.vertx-web-client-3.8.3.jar:dependencies/io.vertx.vertx-web-common-3.8.3.jar:dependencies/javax.xml.bind.jaxb-api-2.3.0.jar:dependencies/org.apache.camel.camel-api-3.0.0.jar:dependencies/org.apache.camel.camel-base-3.0.0.jar:dependencies/org.apache.camel.camel-caffeine-lrucache-3.0.0.jar:dependencies/org.apache.camel.camel-cloud-3.0.0.jar:dependencies/org.apache.camel.camel-core-engine-3.0.0.jar:dependencies/org.apache.camel.camel-endpointdsl-3.0.0.jar:dependencies/org.apache.camel.camel-groovy-3.0.0.jar:dependencies/org.apache.camel.camel-jaxp-3.0.0.jar:dependencies/org.apache.camel.camel-main-3.0.0.jar:dependencies/org.apache.camel.camel-management-api-3.0.0.jar:dependencies/org.apache.camel.camel-support-3.0.0.jar:dependencies/org.apache.camel.camel-util-3.0.0.jar:dependencies/org.apache.camel.camel-util-json-3.0.0.jar:dependencies/org.apache.camel.k.camel-k-loader-groovy-1.0.9.jar:dependencies/org.apache.camel.k.camel-k-runtime-core-1.0.9.jar:dependencies/org.apache.camel.k.camel-k-runtime-main-1.0.9.jar:dependencies/org.apache.camel.k.camel-knative-1.0.9.jar:dependencies/org.apache.camel.k.camel-knative-api-1.0.9.jar:dependencies/org.apache.camel.k.camel-knative-http-1.0.9.jar:dependencies/org.apache.camel.spi-annotations-3.0.0.jar:dependencies/org.apache.commons.commons-collections4-4.4.jar:dependencies/org.apache.commons.commons-lang3-3.9.jar:dependencies/org.apache.logging.log4j.log4j-api-2.13.0.jar:dependencies/org.apache.logging.log4j.log4j-core-2.13.0.jar:dependencies/org.apache.logging.log4j.log4j-slf4j-impl-2.13.0.jar:dependencies/org.codehaus.groovy.groovy-2.5.8.jar:dependencies/org.slf4j.slf4j-api-1.7.30.jar
        image: image-registry.openshift-image-registry.svc:5000/anewusecase/camel-k-kit-bok6uok6kp33jl2vtg9g:678755
        name: integration
        readinessProbe:
          successThreshold: 1
          tcpSocket:
            port: 0
        resources: {}
        volumeMounts:
        - mountPath: /etc/camel/sources/i-source-000
          name: i-source-000
          readOnly: true
        - mountPath: /etc/camel/conf
          name: integration-properties
          readOnly: true
      timeoutSeconds: 300
      volumes:
      - configMap:
          items:
          - key: content
            path: TestReply.groovy
          name: test-reply-source-000
        name: i-source-000
      - configMap:
          items:
          - key: application.properties
            path: application.properties
          name: test-reply-properties
        name: integration-properties
  traffic:
  - latestRevision: true
    percent: 100
status:
  conditions:
  - lastTransitionTime: 2020-01-22T16:49:15Z
    message: 'Revision "test-reply-lb2bf" failed with message: Unable to fetch image
      "image-registry.openshift-image-registry.svc:5000/anewusecase/camel-k-kit-bok6uok6kp33jl2vtg9g:678755":
      Get https://image-registry.openshift-image-registry.svc:5000/v2/: x509: certificate
      signed by unknown authority.'
    reason: RevisionFailed
    status: "False"
    type: ConfigurationsReady
  - lastTransitionTime: 2020-01-22T16:49:15Z
    message: Configuration "test-reply" does not have any ready Revision.
    reason: RevisionMissing
    status: "False"
    type: Ready
  - lastTransitionTime: 2020-01-22T16:49:15Z
    message: Configuration "test-reply" does not have any ready Revision.
    reason: RevisionMissing
    status: "False"
    type: RoutesReady
  latestCreatedRevisionName: test-reply-lb2bf
  observedGeneration: 1
  url: http://test-reply.anewusecase.apps.cluster-camelk-df1a.camelk-df1a.example.opentlc.com

Subscription

apiVersion: messaging.knative.dev/v1alpha1
kind: Subscription
metadata:
  annotations:
    messaging.knative.dev/creator: system:serviceaccount:openshift-operators:camel-k-operator
    messaging.knative.dev/lastModifier: system:serviceaccount:openshift-operators:camel-k-operator
  creationTimestamp: 2020-01-22T16:49:15Z
  generation: 1
  labels:
    camel.apache.org/generation: "1"
    camel.apache.org/integration: test-reply
  name: bloodpressure-test-reply
  namespace: anewusecase
  ownerReferences:
  - apiVersion: camel.apache.org/v1
    blockOwnerDeletion: true
    controller: true
    kind: Integration
    name: test-reply
    uid: 1ad8668c-3d37-11ea-9b99-0205f72f8fa9
  resourceVersion: "718809"
  selfLink: /apis/messaging.knative.dev/v1alpha1/namespaces/anewusecase/subscriptions/bloodpressure-test-reply
  uid: 1f965029-3d37-11ea-946d-0ad1811f81cd
spec:
  channel:
    apiVersion: messaging.knative.dev/v1alpha1
    kind: InMemoryChannel
    name: bloodpressure
  subscriber:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: test-reply
status:
  conditions:
  - lastTransitionTime: 2020-01-22T16:49:15Z
    status: Unknown
    type: AddedToChannel
  - lastTransitionTime: 2020-01-22T16:49:15Z
    status: Unknown
    type: ChannelReady
  - lastTransitionTime: 2020-01-22T16:49:15Z
    message: 'Failed to resolve spec.subscriber: address not set for &ObjectReference{Kind:Service,Namespace:anewusecase,Name:test-reply,UID:,APIVersion:serving.knative.dev/v1,ResourceVersion:,FieldPath:,}'
    reason: SubscriberResolveFailed
    status: "False"
    type: Ready
  - lastTransitionTime: 2020-01-22T16:49:15Z
    message: 'Failed to resolve spec.subscriber: address not set for &ObjectReference{Kind:Service,Namespace:anewusecase,Name:test-reply,UID:,APIVersion:serving.knative.dev/v1,ResourceVersion:,FieldPath:,}'
    reason: SubscriberResolveFailed
    status: "False"
    type: Resolved
  physicalSubscription: {}

The text was updated successfully, but these errors were encountered:

weimeilin79 · 2020-01-22T19:24:36Z

The problem was cause by knative serving controller needs access the container registry to resolve image tags when a revision is created. But with self-signed certs, it's going to have a problem with handshake.

message: 'Revision "test-reply-lb2bf" failed with message: Unable to fetch image
      "image-registry.openshift-image-registry.svc:5000/anewusecase/camel-k-kit-bok6uok6kp33jl2vtg9g:678755":
      Get https://image-registry.openshift-image-registry.svc:5000/v2/: x509: certificate
      signed by unknown authority.'

We need to manually skip it.
please update the configmap by executing following command. And manually restart the serving controller.

oc patch cm config-deployment -p '{"data":{"registriesSkippingTagResolving":"image-registry.openshift-image-registry.svc:5000"}}' -n knative-serving

And we are good to go..

nicolaferraro · 2020-02-11T22:31:51Z

We can avoid the patch to the configmap.

… resolution

weimeilin79 closed this as completed Jan 22, 2020

nicolaferraro reopened this Feb 11, 2020

nicolaferraro added this to the 1.0.0-RC2 milestone Feb 11, 2020

nicolaferraro added a commit to nicolaferraro/camel-k that referenced this issue Feb 11, 2020

fix apache#1222: use image sha in s2i build to avoid knative registry…

b8ee6ff

… resolution

astefanutti closed this as completed in 18c188a Feb 12, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

knative channel consumer stuck in running state #1222

knative channel consumer stuck in running state #1222

weimeilin79 commented Jan 22, 2020

weimeilin79 commented Jan 22, 2020

nicolaferraro commented Feb 11, 2020

knative channel consumer stuck in running state #1222

knative channel consumer stuck in running state #1222

Comments

weimeilin79 commented Jan 22, 2020

weimeilin79 commented Jan 22, 2020

nicolaferraro commented Feb 11, 2020