Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move to use okhttp4 due to a bug on IPV6 based cluster #2632

Open
ppatierno opened this issue Nov 25, 2020 · 27 comments
Open

Move to use okhttp4 due to a bug on IPV6 based cluster #2632

ppatierno opened this issue Nov 25, 2020 · 27 comments

Comments

@ppatierno
Copy link
Contributor

The current version of fabric8 Kubernetes client is using okhttp3 library which is affected by bug on hostname verification when connecting to the Kubernetes server API in IPV6 based cluster.
The same bug affects even the kubernetes-client Java library: kubernetes-client/java#1289
This issue was fixed in okhttp4 as you can see from this already merged PR:
square/okhttp#5889

We are facing this while using fabric8 in the Strimzi operator. There is a workaround as described here strimzi/strimzi-kafka-operator#4002 but it's just a workaround.
Is there any plan to update the fabric8 client to the okhttp4 library version?

@rohanKanojia
Copy link
Member

Isn't okhttp4 based on kotlin? Not sure if it can break any backward compatibility. @manusa @oscerd @iocanel @metacosm WDYT??

@oscerd
Copy link
Member

oscerd commented Nov 26, 2020

Yes, it is. It's a completely different library from 3.x

@ppatierno
Copy link
Contributor Author

So what's the solution you envisage? Back porting the fix about IPV6 from okhttp4 to okhttp3 and then updating fabric8? Right now fabric8 is actually bugged with IPV6 clusters due to this problem in okhttp3 :-(

@oscerd
Copy link
Member

oscerd commented Dec 1, 2020

Bumping to 4.x is something that require effort and time. I'm not sure they will backport the fix on 3.x anyway

@ppatierno
Copy link
Contributor Author

Well,, that could be a contribution to backport on okhttp3 if they won't do that. Otherwise, you should list this as a well-known bug on fabric8.

@manusa
Copy link
Member

manusa commented Dec 9, 2020

Bumping OkHTTP version to 4.x will not only require a lot of work in the client, but some work on the Kubernetes Client Quarkus extension too (I think).

I agree that we should eventually upgrade to 4.x, new issues are bound to happen in the future. But I'm not sure if the timing is right at the moment, since we need to deliver 5.0 ASAP.

Maybe for the time being contributing a backport of the fix to OkHttp 3.x is more cost-effective WDYT?

@metacosm
Copy link
Collaborator

metacosm commented Dec 9, 2020

Maybe for the time being contributing a backport of the fix to OkHttp 3.x is more cost-effective WDYT?

Are we confident the backport would be accepted, though? Or are we talking about maintaining a 3.x fork until the time when we decide to switch to 4?

@manusa
Copy link
Member

manusa commented Dec 10, 2020

My suggestion was just (for now) try to see if the back-port is accepted in the OkHttp project, since it's only a few lines (square/okhttp#5889 - haven't checked branch compatibility though).
Maintaining an OkHttp is not what I was implying. That would probably be a much greater effort than upgrading the dependency in the client.

@yschimke
Copy link

OkHttp 4.x is binary backwards compatible with OkHttp 3.x. The only breaking experience would be if you have kotlin clients building as it's not source compatible. Full description here https://square.github.io/okhttp/upgrading_to_okhttp_4/

It is not a whole new library, clients can upgrade and downgrade (and do, like in Retrofit) since the public API is maintained.

3.14.x is no longer maintained, so you should upgrade to 4.x unless you wanted legacy support with 3.12.12 (Java 7, Android < 5). See https://square.github.io/okhttp/security/

@maxandersen
Copy link

okhttp drags in kotlin runtime,right ? that brings its own set of challenges.

@yschimke
Copy link

I'm not aware of any actual runtime. It uses a kotlin stdlib library, but nothing like a JVM runtime.

https://mvnrepository.com/artifact/org.jetbrains.kotlin/kotlin-stdlib-jdk8/1.4.21-2

This is around 20kb.

@titisan
Copy link

titisan commented Feb 24, 2021

Any progress with this issue?
We are facing the same problem in a IPv6 Kubernetes cluster.

@stale
Copy link

stale bot commented May 25, 2021

This issue has been automatically marked as stale because it has not had any activity since 90 days. It will be closed if no further activity occurs within 7 days. Thank you for your contributions!

@maxandersen
Copy link

Still relevant

@Fabian-K
Copy link
Contributor

Hi,

is it already set in stone that fabric8 client replaces okhttp 3 with vert.x client? What are the main advantages from moving to vert.x client?

Sometimes I see transitive dependencies of okhttp as a motivation. I´m wondering where that comes from because when comparing the dependencies of io.vertx:vertx-web-client:4.0.3 and com.squareup.okhttp3:okhttp:4.9.1 the dependency trees look like this:

\--- com.squareup.okhttp3:okhttp:4.9.1
     +--- com.squareup.okio:okio:2.8.0
     |    +--- org.jetbrains.kotlin:kotlin-stdlib:1.4.0 -> 1.4.10
     |    |    +--- org.jetbrains.kotlin:kotlin-stdlib-common:1.4.10
     |    |    \--- org.jetbrains:annotations:13.0
     |    \--- org.jetbrains.kotlin:kotlin-stdlib-common:1.4.0 -> 1.4.10
     \--- org.jetbrains.kotlin:kotlin-stdlib:1.4.10 (*)
\--- io.vertx:vertx-web-client:4.0.3
     +--- io.vertx:vertx-web-common:4.0.3
     |    \--- io.vertx:vertx-core:4.0.3
     |         +--- io.netty:netty-common:4.1.60.Final
     |         +--- io.netty:netty-buffer:4.1.60.Final
     |         |    \--- io.netty:netty-common:4.1.60.Final
     |         +--- io.netty:netty-transport:4.1.60.Final
     |         |    +--- io.netty:netty-common:4.1.60.Final
     |         |    +--- io.netty:netty-buffer:4.1.60.Final (*)
     |         |    \--- io.netty:netty-resolver:4.1.60.Final
     |         |         \--- io.netty:netty-common:4.1.60.Final
     |         +--- io.netty:netty-handler:4.1.60.Final
     |         |    +--- io.netty:netty-common:4.1.60.Final
     |         |    +--- io.netty:netty-resolver:4.1.60.Final (*)
     |         |    +--- io.netty:netty-buffer:4.1.60.Final (*)
     |         |    +--- io.netty:netty-transport:4.1.60.Final (*)
     |         |    \--- io.netty:netty-codec:4.1.60.Final
     |         |         +--- io.netty:netty-common:4.1.60.Final
     |         |         +--- io.netty:netty-buffer:4.1.60.Final (*)
     |         |         \--- io.netty:netty-transport:4.1.60.Final (*)
     |         +--- io.netty:netty-handler-proxy:4.1.60.Final
     |         |    +--- io.netty:netty-common:4.1.60.Final
     |         |    +--- io.netty:netty-buffer:4.1.60.Final (*)
     |         |    +--- io.netty:netty-transport:4.1.60.Final (*)
     |         |    +--- io.netty:netty-codec:4.1.60.Final (*)
     |         |    +--- io.netty:netty-codec-socks:4.1.60.Final
     |         |    |    +--- io.netty:netty-common:4.1.60.Final
     |         |    |    +--- io.netty:netty-buffer:4.1.60.Final (*)
     |         |    |    +--- io.netty:netty-transport:4.1.60.Final (*)
     |         |    |    \--- io.netty:netty-codec:4.1.60.Final (*)
     |         |    \--- io.netty:netty-codec-http:4.1.60.Final
     |         |         +--- io.netty:netty-common:4.1.60.Final
     |         |         +--- io.netty:netty-buffer:4.1.60.Final (*)
     |         |         +--- io.netty:netty-transport:4.1.60.Final (*)
     |         |         +--- io.netty:netty-codec:4.1.60.Final (*)
     |         |         \--- io.netty:netty-handler:4.1.60.Final (*)
     |         +--- io.netty:netty-codec-http:4.1.60.Final (*)
     |         +--- io.netty:netty-codec-http2:4.1.60.Final
     |         |    +--- io.netty:netty-common:4.1.60.Final
     |         |    +--- io.netty:netty-buffer:4.1.60.Final (*)
     |         |    +--- io.netty:netty-transport:4.1.60.Final (*)
     |         |    +--- io.netty:netty-codec:4.1.60.Final (*)
     |         |    +--- io.netty:netty-handler:4.1.60.Final (*)
     |         |    \--- io.netty:netty-codec-http:4.1.60.Final (*)
     |         +--- io.netty:netty-resolver:4.1.60.Final (*)
     |         +--- io.netty:netty-resolver-dns:4.1.60.Final
     |         |    +--- io.netty:netty-common:4.1.60.Final
     |         |    +--- io.netty:netty-buffer:4.1.60.Final (*)
     |         |    +--- io.netty:netty-resolver:4.1.60.Final (*)
     |         |    +--- io.netty:netty-transport:4.1.60.Final (*)
     |         |    +--- io.netty:netty-codec:4.1.60.Final (*)
     |         |    +--- io.netty:netty-codec-dns:4.1.60.Final
     |         |    |    +--- io.netty:netty-common:4.1.60.Final
     |         |    |    +--- io.netty:netty-buffer:4.1.60.Final (*)
     |         |    |    +--- io.netty:netty-transport:4.1.60.Final (*)
     |         |    |    \--- io.netty:netty-codec:4.1.60.Final (*)
     |         |    \--- io.netty:netty-handler:4.1.60.Final (*)
     |         \--- com.fasterxml.jackson.core:jackson-core:2.11.3
     +--- io.vertx:vertx-auth-common:4.0.3
     |    \--- io.vertx:vertx-core:4.0.3 (*)
     \--- io.vertx:vertx-core:4.0.3 (*)

Also from a size point of view, its ~2.6 MB vs ~5.3 MB (results of building a fat jar with just okhttp or vert.x).

I agree that the project should replace okhttp 3 rather sooner than later. And if you prefer vert.x over okhttp, that's perfectly fine. Just interested in the reasons for it :)

@maxandersen
Copy link

One issue is having a dependency that requires a full kotlin build to rebuild from source. (Afaik)

Other is that by using vert.x we can share and optimize http stacks with frameworks like vert.x itself and quarkus.

Optimally the http client layer would be pluggable but not sure how feasible that is.

@yschimke
Copy link

What do you mean about full Kotlin build? And from source? OkHttp itself is near 100% Kotlin. But it's just a java jar when you use it as a dependency.

@rohanKanojia
Copy link
Member

rohanKanojia commented Jul 14, 2021

@ppatierno: Hello, I'm looking into this issue. I reproduced your issue by using KubernetesClient with a kind cluster with ipv6 configuration specified in kind docs[0]. Are you able to resolve your issue by excluding okhttp3 dependencies introduced by kubernetes-client jar and adding direct okhttp 4 dependencies?

I tested this on a simple reproducer project[1] on a ipv6 kind cluster and it seems to be working fine for me.

    <properties>
        <fabric8.version>5.5.0</fabric8.version>
        <okhttp.version>4.9.0</okhttp.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>io.fabric8</groupId>
            <artifactId>kubernetes-client</artifactId>
            <version>${fabric8.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>com.squareup.okhttp3</groupId>
                    <artifactId>okhttp</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>com.squareup.okhttp3</groupId>
                    <artifactId>logging-interceptor</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>com.squareup.okhttp3</groupId>
            <artifactId>okhttp</artifactId>
            <version>${okhttp.version}</version>
        </dependency>
        <dependency>
            <groupId>com.squareup.okhttp3</groupId>
            <artifactId>logging-interceptor</artifactId>
            <version>${okhttp.version}</version>
        </dependency>
    </dependencies>

I also checked to upgrade Okhttp in KubernetesClient to v4.9.0:

I was able to get that reproducer project working on Kind ipv6 cluster with 5.6-SNAPSHOT(based on my branch). But I need to test it further to see if it's completely safe to upgrade.

I tested this with Quarkus KubernetesClient Extension as well, I'm able to get it working on a Kubernetes ipv6 cluster with the above exclusions. You can see the full project here.

[0] https://kind.sigs.k8s.io/docs/user/configuration/#ip-family
[1] https://github.com/rohankanojia-forks/fabric8-okhttp-ipv6-k8s-cluster-bug-reproducer

@ovolynets
Copy link

ovolynets commented Aug 4, 2021

Hello community!

I would also be interested in either upgrading okhttp or the migration to Vert.x - whatever is decided upon. Reason: CVE-2021-0341 reported by NexusIQ in okhttp in versions prior to 4.9.1, see square/okhttp#6724 for details.

The upgrade of okhttp to 4.9.1 or the most recent 5.0.0-alpha.2 would be what fixes the CVE but if you decide to move to Vert.x this would also solve the problem.

Who can make the decision or either ways to proceed with eliminating the CVE rather than implementing a workaround (e.g. as was suggested earlier for a different problem #2632 (comment))?

Thanks

UPD: 4.9.1 version of okhttp does NOT contain yet the necessary fix, only 5.0.0-alpha.2 does.

@yschimke
Copy link

yschimke commented Aug 4, 2021

@ovolynets the CVE isn't exploitable unless you take the HostnameVerifier from OkHttp and use it for other purposes. Unless you are doing something like that which would be really strange, it isn't exploitable at all.

@zalejus
Copy link

zalejus commented May 13, 2022

Hi everyone,

I'm very interested if/when it will be possible to get a Kubernetes plugin with proper IPv6 support. Currently most of k8s clusters are built on VPC using IPv6, and I'm not exception here :) I'm trying to implement jenkins with all necessary connectivity with kubernetes, and additionally configuring everything as a code. When I add Kubernetes Credentials Provider plugin (which is based on Kubernetes Client API), I got the following error:

2022-05-13 11:05:42.248+0000 [id=40]	INFO	c.c.j.p.k.KubernetesCredentialProvider#startWatchingForSecrets: retrieving secrets with selector: jenkins.io/credentials-type, LabelSelector(matchExpressions=[], matchLabels=null, additionalProperties={})
2022-05-13 11:05:42.306+0000 [id=40]	SEVERE	c.c.j.p.k.KubernetesCredentialProvider#startWatchingForSecrets: Failed to initialise k8s secret provider, secrets from Kubernetes will not be available
javax.net.ssl.SSLPeerUnverifiedException: Hostname fdc0:95c4:2fc::1 not verified:
    certificate: sha256/g9NzNCPT5ppq2NamTlMGTfLlXaQLGTUz9PdAhZC9rSQ=
    DN: CN=kube-apiserver
    subjectAltNames: [fdc0:95c4:2fc:0:0:0:0:1, XXXX:XXXX:XXXX:XXXX:0:0:0:YYYY, 10.0.56.117, fd131eabd68e9bb05d6353da5a27a257.gr7.eu-central-1.eks.amazonaws.com, ip-10-0-56-117.eu-central-1.compute.internal, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local]
	at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:334)
	at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:284)
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:169)
	at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:258)
	at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
	at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
	at okhttp3.RealCall.execute(RealCall.java:93)
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl.send(OkHttpClientImpl.java:138)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.retryWithExponentialBackoff(OperationSupport.java:574)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:553)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:518)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:502)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:133)
Caused: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [list]  for kind: [Secret]  with name: [null]  in namespace: [azalejski]  failed.
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:130)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:140)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:415)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:404)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:83)
	at com.cloudbees.jenkins.plugins.kubernetes_credentials_provider.KubernetesCredentialProvider.startWatchingForSecrets(KubernetesCredentialProvider.java:116)
	at com.cloudbees.jenkins.plugins.kubernetes_credentials_provider.KubernetesCredentialProvider$1.doRun(KubernetesCredentialProvider.java:161)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:92)
	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
2022-05-13 11:05:42.307+0000 [id=40]	INFO	c.c.j.p.k.KubernetesCredentialProvider#reconnectLater: Attempting to reconnect Kubernetes client in 5 mins

The most important thing here is IPv6 address [fdc0:95c4:2fc::1], which probably can't be successfully verified.
I found temporary solution for this topic setting environment variable KUBERNETES_DISABLE_HOSTNAME_VERIFICATION=true in jenkins container, but in my opinion the whole host verification mechanism should be implemented with IPv6 working properly.

Best regards,
Arek

@shawkins
Copy link
Contributor

shawkins commented Dec 7, 2022

Removed the never stale - it doesn't seem like an okhttp4 based option is in the cards to be distributed with the core project. It's easy enough for a user to create one on their own, and especially with 6.3 the jetty and jdk clients are viable alternatives. Vertx should come along in whatever follows 6.3.

@manusa
Copy link
Member

manusa commented Dec 8, 2022

Well, one of the options I was considering is to (finally) upgrade to OkHttp 4 once the Vert.x client is out and Quarkus is no longer conditioned by it.
In any case, that should be tackled before this issue becomes stale again :)

@manusa manusa added this to the 6.x milestone Dec 8, 2022
@stale
Copy link

stale bot commented Mar 8, 2023

This issue has been automatically marked as stale because it has not had any activity since 90 days. It will be closed if no further activity occurs within 7 days. Thank you for your contributions!

@tomdw
Copy link

tomdw commented Nov 27, 2023

Are there still planes to upgrade to okhttp4? the 3.x version depends on an older junit dependency from kubernetes-server-mock through mockwebserver of okhttp3. And this gives CVE vulnerability warning in the IDE.

@maxandersen
Copy link

@manusa Kubernetes client does not rely on okhttp anymore - it's uses vertx.

Mockserver using okhttp is unfortunate. Not sure what the cure is here. For now you can manually override the version.

@manusa
Copy link
Member

manusa commented Nov 27, 2023

Hi @maxandersen

@manusa Kubernetes client does not rely on okhttp anymore - it's uses vertx.

I'm not sure if your intention was to tag me here or you wanted to reply to @tomdw instead.

Fabric8 Kubernetes Client relies on OkHttp for its kubernetes-httpclient-okhttp HttpClient implementation and the Kubernetes Mock Web Server.

@tomdw, as Max said, you should be able to override the OkHttp version for MockWebServer by forcing the dependency to a newer OkHttp version like documented here.

Mockserver using okhttp is unfortunate. Not sure what the cure is here.

Our kubernetes-server-mock and mockwebserver modules rely on OkHttp's mockwebserver as its backend.
One of the options I was considering was to replace this with a Vertx or a Jetty server. However, this will require additional work since right now everything is tightly coupled to OkHttp. (discussed in #5613)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests