Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Question] Expose Kafka Cluster to external Kafka Client using Openshift Routes #3285

Closed
AJCandfield opened this issue Jul 6, 2020 · 10 comments
Labels

Comments

@AJCandfield
Copy link

Hi,

We're trying to expose the Kafka Cluster to an external client. We followed your guide but when we try to query for the topics present on the cluster we get the following error:

[root@ip-172-25-16-7 bin]# ./kafka-topics.sh --list --bootstrap-server my-kafka-cluster-kafka-bootstrap-strimzi-test.apps.ocp.<domain>:443
Error while executing topic command : org.apache.kafka.common.errors.TimeoutException: Call(callName=listTopics, deadlineMs=1594028537598) timed out at 1594028537599 after 1 attempt(s)
[2020-07-06 09:42:17,603] ERROR java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listTopics, deadlineMs=1594028537598) timed out at 1594028537599 after 1 attempt(s)
        at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
        at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
        at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
        at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
        at kafka.admin.TopicCommand$AdminClientTopicService.getTopics(TopicCommand.scala:333)
        at kafka.admin.TopicCommand$AdminClientTopicService.listTopics(TopicCommand.scala:252)
        at kafka.admin.TopicCommand$.main(TopicCommand.scala:66)
        at kafka.admin.TopicCommand.main(TopicCommand.scala)
Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listTopics, deadlineMs=1594028537598) timed out at 1594028537599 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
 (kafka.admin.TopicCommand$)

I'll summarize the steps we took:

  • We edited the Custom Resource Definition adding the fields we found on the guide.
  • We noticed a new route was created listening on port 443 which redirects to a new service listening on port 9094

Have we missed something?

Thanks for your support,

Alexander

@scholzj
Copy link
Member

scholzj commented Jul 6, 2020

First thing which comes to my mind as missing is the TLS configuration for the kafka-topics.sh. As described in the blog post, the routes are always TLS. So you need to enable / configure TLS in the clients.

@AJCandfield
Copy link
Author

AJCandfield commented Jul 8, 2020

Hi Scholzj!

Thanks for your reply. We had a look at the kafka-topics.sh bash script and it doesn't let us specify a certficate.

We also followed the instructions in the post but we didn't manage to establish a TLS pass-through connection from the external client to the brokers.

I attach the commands we ran and the console output:

$ oc extract secret/my-kafka-cluster-cluster-ca-cert --keys=ca.crt --to=- > ca.crt

$ keytool -import -trustcacerts -alias root -file ca.crt -keystore truststore.jks -storepass password -noprompt

$ kafka_2.12-2.5.0/bin/kafka-console-producer.sh --broker-list my-kafka-cluster-kafka-bootstrap-strimzi-test.apps.ocp.<domain>:443 --producer-property security.protocol=SSL --producer-property ssl.truststore.password=password --producer-property ssl.truststore.location=./truststore.jks --topic atesttopic

>[2020-07-08 14:44:34,332] ERROR [Producer clientId=console-producer] Connection to node -1 (my-kafka-cluster-kafka-bootstrap-strimzi-test.apps.ocp.<domain>/<Public IP>:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2020-07-08 14:44:34,333] WARN [Producer clientId=console-producer] Bootstrap broker my-kafka-cluster-kafka-bootstrap-strimzi-test.apps.ocp.<domain>:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

I also attach the section of the CRD of the Kafka resource:

...
  kafka:
    listeners:
      external:
        type: route
      plain: {}
      tls: {}
...

This is from the server.config config map:

...
##########
# External listener
##########
listener.name.external-9094.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
listener.name.external-9094.ssl.keystore.password=${CERTS_STORE_PASSWORD}
listener.name.external-9094.ssl.keystore.type=PKCS12

##########
# Common listener configuration
##########
listeners=REPLICATION-9091://0.0.0.0:9091,PLAIN-9092://0.0.0.0:9092,TLS-9093://0.0.0.0:9093,EXTERNAL-9094://0.0.0.0:9094
advertised.listeners=REPLICATION-9091://my-kafka-cluster-kafka-${STRIMZI_BROKER_ID}.my-kafka-cluster-kafka-brokers.strimzi-test.svc:9091,PLAIN-9092://my-kafka-cluster-kafka-${STRIMZI_BROKER_ID}.my-kafka-cluster-kafka-brokers.strimzi-test.svc:9092,TLS-9093://my-kafka-cluster-kafka-${STRIMZI_BROKER_ID}.my-kafka-cluster-kafka-brokers.strimzi-test.svc:9093,EXTERNAL-9094://${STRIMZI_EXTERNAL_9094_ADVERTISED_HOSTNAME}:${STRIMZI_EXTERNAL_9094_ADVERTISED_PORT}
listener.security.protocol.map=REPLICATION-9091:SSL,PLAIN-9092:PLAINTEXT,TLS-9093:SSL,EXTERNAL-9094:SSL
inter.broker.listener.name=REPLICATION-9091
sasl.enabled.mechanisms=
ssl.secure.random.implementation=SHA1PRNG
ssl.endpoint.identification.algorithm=HTTPS
...

Let us know if you spot anything mis-configured.

Thanks a lot,

Alexander

@scholzj
Copy link
Member

scholzj commented Jul 8, 2020

Thanks for your reply. We had a look at the kafka-topics.sh bash script and it doesn't let us specify a certficate.

I do not have it around to check right now. But I guess there should be at least some option to specify a config file where you can specify it.

Te producer configuration looks good to me. Not sure why it does not work. Can you try to run it with the -Djavax.net.debug=ssl options (passed for example through KAFKA_OPTS env var)? That should help us debug the TLS issue.

@AJCandfield
Copy link
Author

AJCandfield commented Jul 8, 2020

Hi again,

Here's the output we got:

System property jdk.tls.client.cipherSuites is set to 'null'
System property jdk.tls.server.cipherSuites is set to 'null'
Ignoring disabled cipher suite: TLS_DH_anon_WITH_AES_256_CBC_SHA
Ignoring disabled cipher suite: TLS_DH_anon_WITH_AES_256_CBC_SHA256
Ignoring disabled cipher suite: TLS_ECDHE_RSA_WITH_NULL_SHA
Ignoring disabled cipher suite: SSL_RSA_WITH_DES_CBC_SHA
Ignoring disabled cipher suite: SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA
Ignoring disabled cipher suite: TLS_KRB5_WITH_DES_CBC_MD5
Ignoring disabled cipher suite: TLS_ECDH_RSA_WITH_NULL_SHA
Ignoring disabled cipher suite: SSL_DH_anon_EXPORT_WITH_RC4_40_MD5
Ignoring disabled cipher suite: SSL_DH_anon_WITH_DES_CBC_SHA
Ignoring disabled cipher suite: TLS_DH_anon_WITH_AES_128_CBC_SHA
Ignoring disabled cipher suite: TLS_KRB5_WITH_3DES_EDE_CBC_SHA
Ignoring disabled cipher suite: TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
Ignoring disabled cipher suite: TLS_KRB5_WITH_DES_CBC_SHA
Ignoring disabled cipher suite: TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5
Ignoring disabled cipher suite: TLS_ECDHE_ECDSA_WITH_RC4_128_SHA
Ignoring disabled cipher suite: SSL_DHE_RSA_WITH_DES_CBC_SHA
Ignoring disabled cipher suite: TLS_KRB5_WITH_3DES_EDE_CBC_MD5
Ignoring disabled cipher suite: SSL_DH_anon_WITH_RC4_128_MD5
Ignoring disabled cipher suite: TLS_ECDHE_ECDSA_WITH_NULL_SHA
Ignoring disabled cipher suite: SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA
Ignoring disabled cipher suite: TLS_RSA_WITH_NULL_SHA256
Ignoring disabled cipher suite: TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA
Ignoring disabled cipher suite: SSL_DH_anon_WITH_3DES_EDE_CBC_SHA
Ignoring disabled cipher suite: TLS_ECDH_anon_WITH_NULL_SHA
Ignoring disabled cipher suite: SSL_RSA_WITH_3DES_EDE_CBC_SHA
Ignoring disabled cipher suite: SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA
Ignoring disabled cipher suite: TLS_ECDH_anon_WITH_RC4_128_SHA
Ignoring disabled cipher suite: SSL_DHE_DSS_WITH_DES_CBC_SHA
Ignoring disabled cipher suite: TLS_KRB5_EXPORT_WITH_RC4_40_SHA
Ignoring disabled cipher suite: SSL_RSA_EXPORT_WITH_DES40_CBC_SHA
Ignoring disabled cipher suite: TLS_KRB5_WITH_RC4_128_SHA
Ignoring disabled cipher suite: TLS_ECDH_anon_WITH_AES_256_CBC_SHA
Ignoring disabled cipher suite: SSL_RSA_EXPORT_WITH_RC4_40_MD5
Ignoring disabled cipher suite: TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA
Ignoring disabled cipher suite: TLS_KRB5_EXPORT_WITH_RC4_40_MD5
Ignoring disabled cipher suite: TLS_ECDH_anon_WITH_AES_128_CBC_SHA
Ignoring disabled cipher suite: TLS_ECDH_ECDSA_WITH_RC4_128_SHA
Ignoring disabled cipher suite: TLS_KRB5_WITH_RC4_128_MD5
Ignoring disabled cipher suite: TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA
Ignoring disabled cipher suite: TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA
Ignoring disabled cipher suite: SSL_RSA_WITH_RC4_128_SHA
Ignoring disabled cipher suite: TLS_ECDH_ECDSA_WITH_NULL_SHA
Ignoring disabled cipher suite: TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA
Ignoring disabled cipher suite: TLS_ECDH_RSA_WITH_RC4_128_SHA
Ignoring disabled cipher suite: SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA
Ignoring disabled cipher suite: SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
Ignoring disabled cipher suite: SSL_RSA_WITH_NULL_SHA
Ignoring disabled cipher suite: TLS_ECDHE_RSA_WITH_RC4_128_SHA
Ignoring disabled cipher suite: SSL_RSA_WITH_RC4_128_MD5
Ignoring disabled cipher suite: TLS_DH_anon_WITH_AES_128_CBC_SHA256
Ignoring disabled cipher suite: SSL_RSA_WITH_NULL_MD5
Ignoring disabled cipher suite: TLS_DH_anon_WITH_AES_128_GCM_SHA256
Ignoring disabled cipher suite: TLS_DH_anon_WITH_AES_256_GCM_SHA384
Ignoring disabled cipher suite: TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
Ignoring disabled cipher suite: SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA
Ignoring disabled cipher suite: TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA
Ignoring disabled cipher suite: SSL_RSA_WITH_3DES_EDE_CBC_SHA
Ignoring disabled cipher suite: TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA
Ignoring disabled cipher suite: TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA
Ignoring disabled cipher suite: SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
Ignoring disabled cipher suite: TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
Ignoring disabled cipher suite: SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA
Ignoring disabled cipher suite: TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA
Ignoring disabled cipher suite: SSL_RSA_WITH_3DES_EDE_CBC_SHA
Ignoring disabled cipher suite: TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA
Ignoring disabled cipher suite: TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA
Ignoring disabled cipher suite: SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
adding as trusted cert:
  Subject: CN=cluster-ca v0, O=io.strimzi
  Issuer:  CN=cluster-ca v0, O=io.strimzi
  Algorithm: RSA; Serial number: 0xf480b40f3c3cda9d
  Valid from Mon Apr 06 13:01:36 UTC 2020 until Tue Apr 06 13:01:36 UTC 2021

trigger seeding of SecureRandom
done seeding SecureRandom
>Using SSLEngineImpl.
%% Initialized:  [Session-1, SSL_NULL_WITH_NULL_NULL]
Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
%% No cached client session
update handshake state: client_hello[1]
upcoming handshake states: server_hello[2]
*** ClientHello, TLSv1.2
RandomCookie:  GMT: 1594156516 bytes = { 241, 177, 203, 169, 227, 183, 65, 53, 195, 235, 195, 107, 39, 137, 209, 142, 148, 215, 254, 52, 212, 15, 12, 231, 85, 33, 163, 175 }
Session ID:  {}
Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods:  { 0 }
Extension elliptic_curves, curve names: {secp256r1, secp384r1, secp521r1}
Extension ec_point_formats, formats: [uncompressed]
Extension signature_algorithms, signature_algorithms: SHA512withECDSA, SHA512withRSA, SHA384withECDSA, SHA384withRSA, SHA256withECDSA, SHA256withRSA, SHA256withDSA, SHA224withECDSA, SHA224withRSA, SHA224withDSA, SHA1withECDSA, SHA1withRSA, SHA1withDSA
Extension extended_master_secret
Extension server_name, server_name: [type=host_name (0), value=my-kafka-cluster-kafka-bootstrap-strimzi-test.apps.ocp.]
***
kafka-producer-network-thread | console-producer, WRITE: TLSv1.2 Handshake, length = 265
kafka-producer-network-thread | console-producer, READ: TLSv1.2 Handshake, length = 91
check handshake state: server_hello[2]
*** ServerHello, TLSv1.2
RandomCookie:  GMT: -1118611906 bytes = { 67, 120, 11, 101, 181, 243, 46, 77, 64, 2, 244, 204, 74, 186, 70, 30, 117, 169, 28, 212, 125, 100, 148, 20, 142, 119, 147, 69 }
Session ID:  {162, 209, 95, 198, 121, 19, 244, 116, 227, 106, 69, 247, 10, 33, 115, 34, 105, 2, 160, 160, 103, 219, 161, 160, 153, 180, 68, 169, 58, 209, 212, 84}
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
Compression Method: 0
Extension server_name, server_name:
Extension ec_point_formats, formats: [uncompressed]
Extension renegotiation_info, renegotiated_connection: <empty>
***
%% Initialized:  [Session-2, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
** TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
update handshake state: server_hello[2]
upcoming handshake states: server certificate[11]
upcoming handshake states: server_key_exchange[12](optional)
upcoming handshake states: certificate_request[13](optional)
upcoming handshake states: server_hello_done[14]
upcoming handshake states: client certificate[11](optional)
upcoming handshake states: client_key_exchange[16]
upcoming handshake states: certificate_verify[15](optional)
upcoming handshake states: client change_cipher_spec[-1]
upcoming handshake states: client finished[20]
upcoming handshake states: server change_cipher_spec[-1]
upcoming handshake states: server finished[20]
kafka-producer-network-thread | console-producer, READ: TLSv1.2 Handshake, length = 4851
check handshake state: certificate[11]
update handshake state: certificate[11]
upcoming handshake states: server_key_exchange[12](optional)
upcoming handshake states: certificate_request[13](optional)
upcoming handshake states: server_hello_done[14]
upcoming handshake states: client certificate[11](optional)
upcoming handshake states: client_key_exchange[16]
upcoming handshake states: certificate_verify[15](optional)
upcoming handshake states: client change_cipher_spec[-1]
upcoming handshake states: client finished[20]
upcoming handshake states: server change_cipher_spec[-1]
upcoming handshake states: server finished[20]
*** Certificate chain

@scholzj
Copy link
Member

scholzj commented Jul 8, 2020

Is that all you got? The Certificate Chain on the last line should be followed by the certificate used by the remote side. If this is all you got, I think there might be some issue in your OpenShift Router setup or on the network between you and the router.

@AJCandfield
Copy link
Author

AJCandfield commented Jul 8, 2020

My bad, I thought the rest was useless. Here's the rest:

chain [0] = [
[
  Version: V3
  Subject: CN=*.apps.ocp.
  Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11

  Key:  Sun RSA public key, 2048 bits
  params: null
  modulus: 23150421808524910145022434926775672345567078028661400876243972432765770969049353398507945560032273213737375067650824314247395872497220243495893340456964437979848049805512423470254771027103256127262201501741000999592385039894168196500290017280599882531977040912869529306481171583972360640356768077300522157166490321004826959686157774742982747284808735234571286442274868711363741415971474717388451929847852418515236542307002094739120841597982348302861177650035333135129778075045285336003160943829517790683603915787923328524035360707996973794221187003942935742438120644081282521078210471597805231331844872465502183406397
  public exponent: 65537
  Validity: [From: Fri Nov 29 00:00:00 UTC 2019,
               To: Tue Dec 29 12:00:00 UTC 2020]
  Issuer: CN=Amazon, OU=Server CA 1B, O=Amazon, C=US
  SerialNumber: [    0e92a238 69dda0ff f9c2a104 b42afa2e]

Certificate Extensions: 10
[1]: ObjectId: 1.3.6.1.4.1.11129.2.4.2 Criticality=false
Extension unknown: DER encoded OCTET string =
0000: 04 81 F6 04 81 F3 00 F1   00 77 00 A4 B9 09 90 B4  .........w......
0010: 18 58 14 87 BB 13 A2 CC   67 70 0A 3C 35 98 04 F9  .X......gp.<5...
0020: 1B DF B8 E3 77 CD 0E C8   0D DC 10 00 00 01 6E B7  ....w.........n.
0030: 88 7A 97 00 00 04 03 00   48 30 46 02 21 00 99 D5  .z......H0F.!...
0040: 9A 24 78 F1 B4 03 DC C9   E9 8C 4C 59 E3 AB F5 11  .$x.......LY....
0050: F0 8E CC FE 41 0D 2A D1   84 50 1D D2 06 E4 02 21  ....A.*..P.....!
0060: 00 C2 19 22 A3 54 48 E0   FD 5C 21 72 EA FF E8 73  ...".TH..\!r...s
0070: F0 19 54 FE 69 AA 60 EA   47 12 71 CA D0 E7 68 B9  ..T.i.`.G.q...h.
0080: 20 00 76 00 87 75 BF E7   59 7C F8 8C 43 99 5F BD   .v..u..Y...C._.
0090: F3 6E FF 56 8D 47 56 36   FF 4A B5 60 C1 B4 EA FF  .n.V.GV6.J.`....
00A0: 5E A0 83 0F 00 00 01 6E   B7 88 7B 2A 00 00 04 03  ^......n...*....
00B0: 00 47 30 45 02 20 55 04   24 D4 11 CD 63 73 06 99  .G0E. U.$...cs..
00C0: EB B6 36 16 D7 5B FA 82   1C B6 DD 92 0C 92 83 F2  ..6..[..........
00D0: AC AC F8 0B A1 44 02 21   00 93 C3 B2 59 C2 95 B1  .....D.!....Y...
00E0: 44 46 6E 84 66 CE 3C 38   56 7C 4F 27 78 7C B8 8E  DFn.f.<8V.O'x...
00F0: C5 18 A7 D9 06 D4 4B 64   A0                       ......Kd.


[2]: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false
AuthorityInfoAccess [
  [
   accessMethod: ocsp
   accessLocation: URIName: http://ocsp.sca1b.amazontrust.com
,
   accessMethod: caIssuers
   accessLocation: URIName: http://crt.sca1b.amazontrust.com/sca1b.crt
]
]

[3]: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 59 A4 66 06 52 A0 7B 95   92 3C A3 94 07 27 96 74  Y.f.R....<...'.t
0010: 5B F9 3D D0                                        [.=.
]
]

[4]: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:false
  PathLen: undefined
]

[5]: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
  [DistributionPoint:
     [URIName: http://crl.sca1b.amazontrust.com/sca1b.crl]
]]

[6]: ObjectId: 2.5.29.32 Criticality=false
CertificatePolicies [
  [CertificatePolicyId: [2.16.840.1.114412.1.2]
[]  ]
  [CertificatePolicyId: [2.23.140.1.2.1]
[]  ]
]

[7]: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  serverAuth
  clientAuth
]

[8]: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Key_Encipherment
]

[9]: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  DNSName: *.apps.
]

[10]: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: EB 42 DD 20 9F 2C D4 A5   2A 72 92 57 AF 74 19 04  .B. .,..*r.W.t..
0010: 61 2E 76 89                                        a.v.
]
]

]
  Algorithm: [SHA256withRSA]
  Signature:
0000: 88 A1 58 0D B3 F5 F3 24   18 8F 00 E5 62 BE 81 EC  ..X....$....b...
0010: AD 55 9F 13 21 B9 C4 12   5E C1 7E 9D 48 4B E6 01  .U..!...^...HK..
0020: 9C BE 61 AB 31 35 6B 85   67 8E C1 E6 0B 2A 3A B4  ..a.15k.g....*:.
0030: C7 E3 A6 3A 33 B4 83 F9   B1 5B DB 85 01 FB 3E BF  ...:3....[....>.
0040: 39 0A 3D 1B AC 1C B6 FD   30 66 F4 3B 04 4D D8 E2  9.=.....0f.;.M..
0050: 3B 1F CD D2 CB 27 CD F8   DF 38 16 2C AD 8E D7 92  ;....'...8.,....
0060: F7 A4 16 36 76 31 A7 75   FD AC 32 CB C8 23 AE 2F  ...6v1.u..2..#./
0070: EE 74 7F 2E 8D 31 92 18   75 9E 54 E2 C2 55 37 57  .t...1..u.T..U7W
0080: DB B4 6A 21 FB 1A 7F E8   DB 56 94 D2 64 25 00 31  ..j!.....V..d%.1
0090: E6 11 8C 91 14 B0 0E AA   92 C8 0B 9C 55 A1 39 D8  ............U.9.
00A0: 4E A6 5D 2F 05 F8 D4 75   85 68 67 5A 67 6D 2F DC  N.]/...u.hgZgm/.
00B0: A1 96 B7 23 D1 6A 72 B3   AE 49 46 27 BF 0A 01 A6  ...#.jr..IF'....
00C0: 50 37 43 BE 16 E2 27 3C   AD 20 A5 2F 76 2A 8B E2  P7C...'<. ./v*..
00D0: 50 1A 32 19 98 DC F6 AF   BB 88 AA FB 83 44 4C E3  P.2..........DL.
00E0: AB 95 8E 46 5F 67 E9 7C   4F 93 89 B2 F4 35 58 CE  ...F_g..O....5X.
00F0: 05 1F 15 76 F5 6B E5 49   EB 82 49 A4 D0 EE 6C DE  ...v.k.I..I...l.

]
chain [1] = [
[
  Version: V3
  Subject: CN=Amazon, OU=Server CA 1B, O=Amazon, C=US
  Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11

  Key:  Sun RSA public key, 2048 bits
  params: null
  modulus: 24528737555851895213919827617064808536856788789868126310716752303420041319710819680867697306230985630039655096548324364189962675576756038921107965025585889330528490649228935527969954506874750514159926943451238689552458142167021149788529783891257271028002485075630471793111207960868638365698705018555597520367289025831586046483446904825820575805338475813865444295353094097022678376192149453480223428943386514159000527368947588174705227657134217583008630047462959260157651883088072156905420231950318110240318878613016990846576820326568049365612395397183597930457965295993595011597251067348997341253617591444999389873893
  public exponent: 65537
  Validity: [From: Thu Oct 22 00:00:00 UTC 2015,
               To: Sun Oct 19 00:00:00 UTC 2025]
  Issuer: CN=Amazon Root CA 1, O=Amazon, C=US
  SerialNumber: [    067f9457 8587e8ac 77deb253 325bbc99 8b560d]

Certificate Extensions: 7
[1]: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false
AuthorityInfoAccess [
  [
   accessMethod: ocsp
   accessLocation: URIName: http://ocsp.rootca1.amazontrust.com
,
   accessMethod: caIssuers
   accessLocation: URIName: http://crt.rootca1.amazontrust.com/rootca1.cer
]
]

[2]: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 84 18 CC 85 34 EC BC 0C   94 94 2E 08 59 9C C7 B2  ....4.......Y...
0010: 10 4E 0A 08                                        .N..
]
]

[3]: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen:0
]

[4]: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
  [DistributionPoint:
     [URIName: http://crl.rootca1.amazontrust.com/rootca1.crl]
]]

[5]: ObjectId: 2.5.29.32 Criticality=false
CertificatePolicies [
  [CertificatePolicyId: [2.23.140.1.2.1]
[]  ]
]

[6]: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Key_CertSign
  Crl_Sign
]

[7]: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 59 A4 66 06 52 A0 7B 95   92 3C A3 94 07 27 96 74  Y.f.R....<...'.t
0010: 5B F9 3D D0                                        [.=.
]
]

]
  Algorithm: [SHA256withRSA]
  Signature:
0000: 85 92 BE 35 BB 79 CF A3   81 42 1C E4 E3 63 73 53  ...5.y...B...csS
0010: 39 52 35 E7 D1 AD FD AE   99 8A AC 89 12 2F BB E7  9R5........../..
0020: 6F 9A D5 4E 72 EA 20 30   61 F9 97 B2 CD A5 27 02  o..Nr. 0a.....'.
0030: 45 A8 CA 76 3E 98 4A 83   9E B6 E6 45 E0 F2 43 F6  E..v>.J....E..C.
0040: 08 DE 6D E8 6E DB 31 07   13 F0 2F 31 0D 93 6D 61  ..m.n.1.../1..ma
0050: 37 7B 58 F0 FC 51 98 91   28 02 4F 05 76 B7 D3 F0  7.X..Q..(.O.v...
0060: 1B C2 E6 5E D0 66 85 11   0F 2E 81 C6 10 81 29 FE  ...^.f........).
0070: 20 60 48 F3 F2 F0 84 13   53 65 35 15 11 6B 82 51   `H.....Se5..k.Q
0080: 40 55 57 5F 18 B5 B0 22   3E AD F2 5E A3 01 E3 C3  @UW_...">..^....
0090: B3 F9 CB 41 5A E6 52 91   BB E4 36 87 4F 2D A9 A4  ...AZ.R...6.O-..
00A0: 07 68 35 BA 94 72 CD 0E   EA 0E 7D 57 F2 79 FC 37  .h5..r.....W.y.7
00B0: C5 7B 60 9E B2 EB C0 2D   90 77 0D 49 10 27 A5 38  ..`....-.w.I.'.8
00C0: AD C4 12 A3 B4 A3 C8 48   B3 15 0B 1E E2 E2 19 DC  .......H........
00D0: C4 76 52 C8 BC 8A 41 78   70 D9 6D 97 B3 4A 8B 78  .vR...Axp.m..J.x
00E0: 2D 5E B4 0F A3 4C 60 CA   E1 47 CB 78 2D 12 17 B1  -^...L`..G.x-...
00F0: 52 8B CA 39 2C BD B5 2F   C2 33 02 96 AB DA 94 7F  R..9,../.3......

]
chain [2] = [
[
  Version: V3
  Subject: CN=Amazon Root CA 1, O=Amazon, C=US
  Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11

  Key:  Sun RSA public key, 2048 bits
  params: null
  modulus: 22529839904807742196558773392430766620630713202204326167346456925862066285712069978308045976033918808540171076811098215136401323342247576789054764683787147408289170989302937775178809187827657352584557953877946352196797789035355954596527030584944622221752357105572088106020206921431118198373122638305846252087992561841631797199384157902018140720267433956687491591657652730221337591680012205319549572614035105482287002884850178224609018864719685310905426619874727796905080238179726224664042154200651710137931048812546957419686875805576245376866031854569863410951649630469236463991472642618512857920826701027482532358669
  public exponent: 65537
  Validity: [From: Mon May 25 12:00:00 UTC 2015,
               To: Thu Dec 31 01:00:00 UTC 2037]
  Issuer: CN=Starfield Services Root Certificate Authority - G2, O="Starfield Technologies, Inc.", L=Scottsdale, ST=Arizona, C=US
  SerialNumber: [    067f944a 2a27cdf3 fac2ae2b 01f908ee b9c4c6]

Certificate Extensions: 7
[1]: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false
AuthorityInfoAccess [
  [
   accessMethod: ocsp
   accessLocation: URIName: http://ocsp.rootg2.amazontrust.com
,
   accessMethod: caIssuers
   accessLocation: URIName: http://crt.rootg2.amazontrust.com/rootg2.cer
]
]

[2]: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 9C 5F 00 DF AA 01 D7 30   2B 38 88 A2 B8 6D 4A 9C  ._.....0+8...mJ.
0010: F2 11 91 83                                        ....
]
]

[3]: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen:2147483647
]

[4]: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
  [DistributionPoint:
     [URIName: http://crl.rootg2.amazontrust.com/rootg2.crl]
]]

[5]: ObjectId: 2.5.29.32 Criticality=false
CertificatePolicies [
  [CertificatePolicyId: [2.5.29.32.0]
[]  ]
]

[6]: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Key_CertSign
  Crl_Sign
]

[7]: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 84 18 CC 85 34 EC BC 0C   94 94 2E 08 59 9C C7 B2  ....4.......Y...
0010: 10 4E 0A 08                                        .N..
]
]

]
  Algorithm: [SHA256withRSA]
  Signature:
0000: 62 37 42 5C BC 10 B5 3E   8B 2C E9 0C 9B 6C 45 E2  b7B\...>.,...lE.
0010: 07 00 7A F9 C5 58 0B B9   08 8C 3E ED B3 25 3C B5  ..z..X....>..%<.
0020: 6F 50 E4 CD 35 6A A7 93   34 96 32 21 A9 48 44 AB  oP..5j..4.2!.HD.
0030: 9C ED 3D B4 AA 73 6D E4   7F 16 80 89 6C CF 28 03  ..=..sm.....l.(.
0040: 18 83 47 79 A3 10 7E 30   5B AC 3B B0 60 E0 77 D4  ..Gy...0[.;.`.w.
0050: 08 A6 E1 1D 7C 5E C0 BB   F9 9A 7B 22 9D A7 00 09  .....^....."....
0060: 7E AC 46 17 83 DC 9C 26   57 99 30 39 62 96 8F ED  ..F....&W.09b...
0070: DA DE AA C5 CC 1B 3E CA   43 68 6C 57 16 BC D5 0E  ......>.ChlW....
0080: 20 2E FE FF C2 6A 5D 2E   A0 4A 6D 14 58 87 94 E6   ....j]..Jm.X...
0090: 39 31 5F 7C 73 CB 90 88   6A 84 11 96 27 A6 ED D9  91_.s...j...'...
00A0: 81 46 A6 7E A3 72 00 0A   52 3E 83 88 07 63 77 89  .F...r..R>...cw.
00B0: 69 17 0F 39 85 D2 AB 08   45 4D D0 51 3A FD 5D 5D  i..9....EM.Q:.]]
00C0: 37 64 4C 7E 30 B2 55 24   42 9D 36 B0 5D 9C 17 81  7dL.0.U$B.6.]...
00D0: 61 F1 CA F9 10 02 24 AB   EB 0D 74 91 8D 7B 45 29  a.....$...t...E)
00E0: 50 39 88 B2 A6 89 35 25   1E 14 6A 47 23 31 2F 5C  P9....5%..jG#1/\
00F0: 9A FA AD 9A 0E 62 51 A4   2A A9 C4 F9 34 9D 21 18  .....bQ.*...4.!.

]
chain [3] = [
[
  Version: V3
  Subject: CN=Starfield Services Root Certificate Authority - G2, O="Starfield Technologies, Inc.", L=Scottsdale, ST=Arizona, C=US
  Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11

  Key:  Sun RSA public key, 2048 bits
  params: null
  modulus: 26894789576491863019171445242018370132029525033879210664513024255165308689836081694724912552986436241602345929261854187816625921774943728567119070351838976265193901442169339571326613928339955106648223197498035701437846440970934704192382084561469274550003268570741310868032789070264835003681318445644941362885752628282968349509706358865971392279088395067847314610178969555804359319567178098112935181143559364150874524817692694181296058297355335204675211145990489303168553611700020424738364579606192390834705213026692659672388567853246354560726855054573503174641583891075106464210711468427779853334564691648681991700229
  public exponent: 65537
  Validity: [From: Wed Sep 02 00:00:00 UTC 2009,
               To: Wed Jun 28 17:39:16 UTC 2034]
  Issuer: OU=Starfield Class 2 Certification Authority, O="Starfield Technologies, Inc.", C=US
  SerialNumber: [    a70e4a4c 3482b77f]

Certificate Extensions: 7
[1]: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false
AuthorityInfoAccess [
  [
   accessMethod: ocsp
   accessLocation: URIName: http://o.ss2.us/
,
   accessMethod: caIssuers
   accessLocation: URIName: http://x.ss2.us/x.cer
]
]

[2]: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: BF 5F B7 D1 CE DD 1F 86   F4 5B 55 AC DC D7 10 C2  ._.......[U.....
0010: 0E A9 88 E7                                        ....
]
]

[3]: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen:2147483647
]

[4]: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
  [DistributionPoint:
     [URIName: http://s.ss2.us/r.crl]
]]

[5]: ObjectId: 2.5.29.32 Criticality=false
CertificatePolicies [
  [CertificatePolicyId: [2.5.29.32.0]
[]  ]
]

[6]: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Key_CertSign
  Crl_Sign
]

[7]: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 9C 5F 00 DF AA 01 D7 30   2B 38 88 A2 B8 6D 4A 9C  ._.....0+8...mJ.
0010: F2 11 91 83                                        ....
]
]

]
  Algorithm: [SHA256withRSA]
  Signature:
0000: 23 1D E3 8A 57 CA 7D E9   17 79 4C F1 1E 55 FD CC  #...W....yL..U..
0010: 53 6E 3E 47 0F DF C6 55   F2 B2 04 36 ED 80 1F 53  Sn>G...U...6...S
0020: C4 5D 34 28 6B BE C7 55   FC 67 EA CB 3F 7F 90 B2  .]4(k..U.g..?...
0030: 33 CD 1B 58 10 82 02 F8   F8 2F F5 13 60 D4 05 CE  3..X...../..`...
0040: F1 81 08 C1 DD A7 75 97   4F 18 B9 6D DE F7 93 91  ......u.O..m....
0050: 08 BA 7E 40 2C ED C1 EA   BB 76 9E 33 06 77 1D 0D  ...@,....v.3.w..
0060: 08 7F 53 DD 1B 64 AB 82   27 F1 69 D5 4D 5E AE F4  ..S..d..'.i.M^..
0070: A1 C3 75 A7 58 44 2D F2   3C 70 98 AC BA 69 B6 95  ..u.XD-.<p...i..
0080: 77 7F 0F 31 5E 2C FC A0   87 3A 47 69 F0 79 5F F4  w..1^,...:Gi.y_.
0090: 14 54 A4 95 5E 11 78 12   60 27 CE 9F C2 77 FF 23  .T..^.x.`'...w.#
00A0: 53 77 5D BA FF EA 59 E7   DB CF AF 92 96 EF 24 9A  Sw]...Y.......$.
00B0: 35 10 7A 9C 91 C6 0E 7D   99 F6 3F 19 DF F5 72 54  5.z.......?...rT
00C0: E1 15 A9 07 59 7B 83 BF   52 2E 46 8C B2 00 64 76  ....Y...R.F...dv
00D0: 1C 48 D3 D8 79 E8 6E 56   CC AE 2C 03 90 D7 19 38  .H..y.nV..,....8
00E0: 99 E4 CA 09 19 5B FF 07   96 B0 A8 7F 34 49 DF 56  .....[......4I.V
00F0: A9 F7 B0 5F ED 33 ED 8C   47 B7 30 03 5D F4 03 8C  ..._.3..G.0.]...

]
***
kafka-producer-network-thread | console-producer, fatal error: 46: General SSLEngine problem
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
%% Invalidated:  [Session-1, SSL_NULL_WITH_NULL_NULL]
%% Invalidated:  [Session-2, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
kafka-producer-network-thread | console-producer, SEND TLSv1.2 ALERT:  fatal, description = certificate_unknown
kafka-producer-network-thread | console-producer, WRITE: TLSv1.2 Alert, length = 2
kafka-producer-network-thread | console-producer, fatal: engine already closed.  Rethrowing javax.net.ssl.SSLHandshakeException: General SSLEngine problem
kafka-producer-network-thread | console-producer, called closeOutbound()
kafka-producer-network-thread | console-producer, closeOutboundInternal()
kafka-producer-network-thread | console-producer, called closeInbound()
kafka-producer-network-thread | console-producer, fatal: engine already closed.  Rethrowing javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
kafka-producer-network-thread | console-producer, called closeOutbound()
kafka-producer-network-thread | console-producer, closeOutboundInternal()
[2020-07-08 15:31:48,732] ERROR [Producer clientId=console-producer] Connection to node -1 (my-kafka-cluster-kafka-bootstrap-strimzi-test.apps.ocp.:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2020-07-08 15:31:48,733] WARN [Producer clientId=console-producer] Bootstrap broker my-kafka-cluster-kafka-bootstrap-strimzi-test.apps.ocp.:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
Using SSLEngineImpl.

and then it repeats itself.

Thanks!!!

@scholzj
Copy link
Member

scholzj commented Jul 8, 2020

So, I guess this suggests that there is some issue with the TLS Passthrough in your OCP Router or you use wrong address to connect. Is the address my-kafka-cluster-kafka-bootstrap-strimzi-test.apps.ocp.<domain> the same which you can see in the Kafka custom resource in the .status section? If it is, it has to be in the OCP Router. Do you use the original HAProxy router? I know from some past issues that for example the F5 router did not support TLS passthrough.

@AJCandfield
Copy link
Author

Hi,

Yes, the route is correct and we're using the original router.

@scholzj
Copy link
Member

scholzj commented Jul 9, 2020

Hmm ... maybe there is some configuration to disable / enable TLS Passthrough in the Router. But I have no idea I'm afraid, it always worked out of the box for me.

@scholzj
Copy link
Member

scholzj commented Jul 29, 2020

Do you have anything more @AJCandfield what we can help with? Or can we close this?

@scholzj scholzj closed this as completed Aug 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants