Uncaught exception in REST call to /admin/loggers/ #4105

ruchirvaninasdaq · 2020-12-14T23:07:32Z

Describe the bug
I am usinng kafka strimzi 0.20.0 on EKS 1.15. We have custom kafka-connector (using apache camel connector) deployed . I followed instruction from here (https://strimzi.io/blog/2020/01/27/deploying-debezium-with-kafkaconnector-resource/).

I am seeing follwing errors in my connector logs:

Dec 14, 2020 9:24:14 PM org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper toResponse
SEVERE: Uncaught exception in REST call to /admin/loggers/
javax.ws.rs.NotFoundException: HTTP 404 Not Found
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:250)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:679)
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:392)
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:365)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:318)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
        at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:505)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804)
        at java.lang.Thread.run(Thread.java:748)

To Reproduce
Steps to reproduce the behavior:

Run kafka strimzi 0.20.0
I am using following yaml (connector yaml)

apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaConnect
metadata:
  name: kafka-connect-cluster
  labels:
    app: kafka-connect-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 2.5.0
  replicas: 1
  resources:
    limits:
      cpu: 6500m
      memory: 8Gi
    requests:
      cpu: 2000m
      memory: 4Gi
  bootstrapServers: kafka-cluster-kafka-bootstrap.mdic-strimzi-pro:9093
  image: {{ .Values.image }}
  tls:
    trustedCertificates:
      - secretName: kafka-cluster-cluster-ca-cert
        certificate: ca.crt
  authentication:
    type: oauth
    clientId: {client-id}
    clientSecret:
      key: secret
      secretName: {{secretname}}
    tokenEndpointUri: {{ .Values.keycloak }}
  config:
    group.id: s3-connector
    offset.storage.topic: KAFKA-CONNECT-CLUSTER-OFFSET
    config.storage.topic: KAFKA-CONNECT-CLUSTER-CONFIG
    status.storage.topic: KAFKA-CONNECT-CLUSTER-STATUS
    key.converter: org.apache.kafka.connect.storage.StringConverter
    value.converter: org.apache.camel.kafkaconnector.converters.S3ObjectConverter

This is my docker file

FROM {internal registry}
#Copy in code base
COPY . .

# Build
RUN mvn -B \
     -s ci_settings.xml \
     clean install -Dmaven.test.skip=true

### Build Images ###
## Coping the plugins ##
FROM strimzi/kafka:latest-kafka-2.3.0 as core
USER root:root


COPY --from=builder /core/target/kafka-connector-0.0.1-SNAPSHOT-package/share/java/kafka-connector/ /opt/kafka/plugins/

See errors in logs

SEVERE: Uncaught exception in REST call to /admin/loggers/
javax.ws.rs.NotFoundException: HTTP 404 Not Found
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:250)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:679)
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:392)
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:365)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:318)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
        at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:505)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804)
        at java.lang.Thread.run(Thread.java:748)

Expected behavior

When I was running this strimzi 0.19.0 , I didnt see any issues. We just upgraded to strimzi 0.20.0.
If I upgrade my docker file for connector to use strimzi 2.5.0 , this error disappears
strimzi/kafka:latest-kafka-2.5.0 as core

The text was updated successfully, but these errors were encountered:

scholzj · 2020-12-15T00:07:50Z

Strimzi 0.20.0 can work only with the container images form strimzi 0.20.0. By using strimzi/kafka:latest-kafka-2.3.0 as your bae image, you are using some very old image from some master build long time ago. This image might not support all features required by Strimzi 0.20 and that might cause the errors you have seen. So if oyu use Stirmzi 0.20, you should use one of the images from Strimzi 0.20:

strimzi/kafka:0.20.0-kafka-2.5.0
strimzi/kafka:0.20.0-kafka-2.5.1
strimzi/kafka:0.20.0-kafka-2.6.0

ruchirvaninasdaq · 2020-12-15T14:24:10Z

Thanks for the information.

Is there any documentation page/ section within documentation which I can follow to keep track of backward compatibility of strimzi-container versions?
Does strimzi 0.20 supports kafka consumer clients 2.3.0?
-- I have seen that it works in my system, but is there anything I should be aware of strimzi version to kafka version compatibility?

scholzj · 2020-12-15T14:37:12Z

Is there any documentation page/ section within documentation which I can follow to keep track of backward compatibility of strimzi-container versions?

TBH, I'm not exactly sure what is documented where. But the rule is simple - operator version X always works with container images version X. Any other combination is not tested and might or might not work depending on the changes in Kafka, in Strimzi etc. List of what Kafka versions are supported in each Strimzi versions is here: https://strimzi.io/downloads/.

@PaulRMellor @laidan6000 Maybe we should make this a bit more clear about the Strimzi versions of the containers?

Does strimzi 0.20 supports kafka consumer clients 2.3.0?
-- I have seen that it works in my system, but is there anything I should be aware of strimzi version to kafka version compatibility?

The Kafka protocol is implemented as backwards compatible in the clients and brokers. So you can use older client with newer broker or newer client with older broker without any major issues (at least unless you get too far back into the 0.x versions etc.).

There are some consequences of it -> such as that the broker might need to down-convert the messages to the older protocol format which might cost some performance etc. But it works in general.

ruchirvaninasdaq · 2020-12-15T14:59:54Z

Thank you!!

ruchirvaninasdaq added the bug label Dec 14, 2020

scholzj added question and removed bug labels Dec 15, 2020

ruchirvaninasdaq closed this as completed Dec 15, 2020

ruchirvaninasdaq mentioned this issue Dec 15, 2020

Getting some error logs for camel.connector 3.1.0 with strimzi 0.20.0 update apache/camel-kafka-connector#766

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uncaught exception in REST call to /admin/loggers/ #4105

Uncaught exception in REST call to /admin/loggers/ #4105

ruchirvaninasdaq commented Dec 14, 2020 •

edited

scholzj commented Dec 15, 2020

ruchirvaninasdaq commented Dec 15, 2020 •

edited

scholzj commented Dec 15, 2020

ruchirvaninasdaq commented Dec 15, 2020

Uncaught exception in REST call to /admin/loggers/ #4105

Uncaught exception in REST call to /admin/loggers/ #4105

Comments

ruchirvaninasdaq commented Dec 14, 2020 • edited

scholzj commented Dec 15, 2020

ruchirvaninasdaq commented Dec 15, 2020 • edited

scholzj commented Dec 15, 2020

ruchirvaninasdaq commented Dec 15, 2020

ruchirvaninasdaq commented Dec 14, 2020 •

edited

ruchirvaninasdaq commented Dec 15, 2020 •

edited