Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crypto/tls: extension (5) error with TLS 1.3 with Go server and Java client #35722

Closed
adobley opened this issue Nov 21, 2019 · 14 comments
Closed
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@adobley
Copy link

adobley commented Nov 21, 2019

What version of Go are you using (go version)?

$ go version
go version go1.13.4 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.13.4/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.13.4/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/h7/7dckdqy514j9bjr451bjx8rr0000gn/T/go-build975453702=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

We have a Go server that uses TLS1.3. We are able to successfully connect to it with a Go client.

We have a Java application running on OpenJDK 11.0.5_10. When connecting to the server we get an exception: javax.net.ssl.SSLHandshakeException: extension (5) should not be presented in certificate_request.

When we disable TLS1.3 on either side it works. For example, setting GODEBUG to have tls13=0 or using an older version of Java that does not support TLS1.3.

Has anyone else seen this problem?
Is there a way to change the extensions presented?

Unfortunately, we haven't been able to reproduce this on a fully local environment due to java-localhost-cert problems.

What did you expect to see?

A successful TLS handshake and the http request to complete.

What did you see instead?

We see an exception in the Java client javax.net.ssl.SSLHandshakeException: extension (5) should not be presented in certificate_request and an error reported on the Go server remote error: tls: unsupported extension

For redundancy, our main questions here are:
Has anyone else seen this problem?
Is there a way to change the extensions presented?

cc/ @ameowlia

@odeke-em
Copy link
Member

odeke-em commented Nov 21, 2019

Hello there @adobley, thank you for this bug report and welcome to the Go project!

I just saw this issue right before bed and spun up for you something in Go and Java, to hopefully complete your bug report and fully diagnose the problem at https://github.com/odeke-em/bugs/tree/master/golang/35722 whose structure will look like this

$ tree .
.
├── cert.pem
├── jettyclient
│   ├── pom.xml
│   └── src
│       └── main
│           └── java
│               └── org
│                   └── golang
│                       └── bugs
│                           └── b35722
│                               └── Client.java
├── key.pem
└── main.go

and with the files inlined below:

main.go -- contains the server

package main

import (
	"fmt"
	"net/http"
	"time"
)

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/hello", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "%s", time.Now())
	})
	addr := ":8888"
	println("Serving on " + addr)
	if err := http.ListenAndServeTLS(addr, "cert.pem", "key.pem", mux); err != nil {
		panic(err)
	}
}

jettyclient/src/main/java/org/golang/bugs/b35722/Client.java

package org.golang.bugs.b35722;

import java.nio.charset.StandardCharsets;
import org.eclipse.jetty.client.HttpClient;
import org.eclipse.jetty.client.HttpRequest;
import org.eclipse.jetty.util.ssl.SslContextFactory;
import org.eclipse.jetty.client.api.ContentResponse;

public class Client {
    public static void main(String[] args) {
        SslContextFactory insecureSkipVerifyFactory = new SslContextFactory.Client(true);
        HttpClient httpClient = new HttpClient(insecureSkipVerifyFactory);

        try {
            httpClient.start();
        } catch (Exception e) {
            System.err.println("Failed to start the HTTP client " + e);
            return;
        }

        String url = "https://localhost:8888/hello";
        if (args.length > 0 && !args[0].isEmpty())
            url = args[0];

        HttpRequest req = (HttpRequest) httpClient.newRequest(url);
        try {
            ContentResponse res = req.send();
            String payload = new String(res.getContent(), StandardCharsets.UTF_8);
            System.out.println("Response:\n" + payload);
        } catch (Exception e) {
            System.err.println("Exception instead of response " + e);
        }
    }
}

jettyclient/pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>org.golang.bugs</groupId>
    <artifactId>bug-35722</artifactId>
    <packaging>jar</packaging>
    <version>0.0.1</version>
    <name>gobugs</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <!-- Change this to the Jetty version that you would like to use -->
        <jetty.version>9.4.18.v20190429</jetty.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-client</artifactId>
            <version>${jetty.version}</version>
        </dependency>
    </dependencies>

    <build>
        <extensions>
            <extension>
                <groupId>kr.motd.maven</groupId>
                <artifactId>os-maven-plugin</artifactId>
                <version>1.5.0.Final</version>
            </extension>
        </extensions>

        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <version>3.7.0</version>
                    <configuration>
                        <source>1.8</source>
                        <target>1.8</target>
                    </configuration>
                </plugin>
            </plugins>
        </pluginManagement>

        <plugins>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>appassembler-maven-plugin</artifactId>
                <version>1.10</version>
                <configuration>
                    <programs>
                        <program>
                            <id>SQLApp</id>
                            <mainClass>org.golang.bugs.b35722.Client</mainClass>
                        </program>
                    </programs>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

Running it

Run the server first

$ go run main.go 
Serving on :8888

Run the Jetty client

$ mvn install && mvn exec:java -Dexec.mainClass=org.golang.bugs.b35722.Client

and that produces for the client

2019-11-21 02:00:18.524:INFO::org.golang.bugs.b35722.Client.main(): Logging initialized @1836ms to org.eclipse.jetty.util.log.StdErrLog
2019-11-21 02:00:18.672:WARN:oejusS.config:org.golang.bugs.b35722.Client.main(): Trusting all certificates configured for Client@7215a16f[provider=null,keyStore=null,trustStore=null]
2019-11-21 02:00:18.673:WARN:oejusS.config:org.golang.bugs.b35722.Client.main(): No Client EndPointIdentificationAlgorithm configured for Client@7215a16f[provider=null,keyStore=null,trustStore=null]
Response:
2019-11-21 02:00:18.836025 -0500 EST m=+1072.942000100

and from above you'll see that with Jetty we are able to get a successful response (by skipping verification of certificates) so perhaps you might have to make a custom TrustManager, but anyways the repros I've provided above should perhaps help seed you with an MVP to complete this bug report and diagnosis.

I am off to sleep but I shall kindly ping @FiloSottile @katiehockman @agl to take a look or be ware of this report.

@odeke-em odeke-em changed the title extension (5) error with TLS 1.3 with golang server and java client crypto/tls: extension (5) error with TLS 1.3 with Go server and Java client Nov 21, 2019
@odeke-em odeke-em added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 21, 2019
@FiloSottile
Copy link
Contributor

Extension 5 is status_request. Starting in TLS 1.3 status_request can be sent as an empty extension in CertificateRequest to request the client send an OCSP response.

  A server MAY request that a client present an OCSP response with its
   certificate by sending an empty "status_request" extension in its
   CertificateRequest message.  If the client opts to send an OCSP
   response, the body of its "status_request" extension MUST be a
   CertificateStatus structure as defined in [RFC6066].

CertificateRequest extensions are new in TLS 1.3, and OpenSSL also got this wrong and broke down talking to Go (#34040). It looks like OpenJDK 11 is rejecting a spec-compliant behavior on our side.

@davidben have you seen this before?

@FiloSottile FiloSottile modified the milestones: Backlog, Go1.14 Nov 21, 2019
@davidben
Copy link
Contributor

I haven't, though we don't send status_request in CertificateRequest. I agree with you that rejecting the extension in CertificateRequest is non-compliant. Although it's unclear to me what 'empty "status_request" extension' in RFC 8446 is trying to say, since a "semantically empty" status_request extension isn't quite empty.
https://tools.ietf.org/html/rfc6066#section-8

I suspect this is a mistake and the RFC shouldn't have said "empty" there. What to do about it now, I'm not sure. :-(

Do you know if OpenJDK 11 is rejecting the extension because it is not a valid CertificateStatusRequest, or because it's present at all? The latter is definitely wrong, but the former is unclear.

@Lekensteyn
Copy link
Contributor

Lekensteyn commented Nov 22, 2019

The change that adds "empty status_request" was done in tlswg/tls13-spec#910, but I can't exactly remember the reason for doing it. The PR lacks a discussion too. My guess is that the change was done based on a misreading or observed implementation behavior.

When the CH includes status_request with CertificateStatusRequest, the server should reply with a SH status_request extension to acknowledge its support, and send a CertificateStatus handshake message later. The SH extension carries no further meaning and is therefore empty.

In TLS 1.3 the server can also send a status_request extension in the CR extensions. Unlike the earlier server-side status_request extension, this one functions like the CH extension and should therefore not be empty.

It should probably be filed as errata, allowing non-empty extensions again, but I am afraid it will break implementations that strictly follow the spec. Sorry for this mess.

References:

@davidben
Copy link
Contributor

I suppose, worst case, we can always give up on status_request in CertificateRequest and allocate status_request_oops_lets_try_that_again. :-)

@FiloSottile
Copy link
Contributor

Yeah, this is unfortunate. It definitely makes sense for it to contain a CertificateStatusRequest, but I think the language of the spec at this point is pretty clearly defining it to be zero length.

Wireshark also expects it to be empty, and I am strictly enforcing it in Go, so that ship might have sailed unless we want to leave behind a big chunk of Go versions.

Anyway, I really can't read Java, but it looks to me like OpenJDK simply doesn't know status_request is allowed in CertificateRequest. https://github.com/openjdk/jdk/blob/92d5dcf3335179ff68b761aa9e2cc83bf74d47fe/src/java.base/share/classes/sun/security/ssl/CertStatusExtension.java

@ianlancetaylor
Copy link
Contributor

@FiloSottile Is there anything we should do here for 1.14?

@martinuy
Copy link

martinuy commented Dec 16, 2019

I've raised this issue in OpenJDK's upstream [1], and has been acknowledged [2].

BTW, I completely agree with @Lekensteyn in the empty status_request comment.

--
[1] - https://mail.openjdk.java.net/pipermail/security-dev/2019-December/021036.html
[2] - https://mail.openjdk.java.net/pipermail/security-dev/2019-December/021037.html

@FiloSottile
Copy link
Contributor

@FiloSottile Is there anything we should do here for 1.14?

If the WG decides to issue an errata we should probably relax our client implementation to allow non-empty extensions, to mitigate the compatibility mess. It's a very very safe change. However, I'm mildly opposed to it still, because even if we backport it there will be many Go clients out there that are intolerant to the "rich" extension, because we followed the spec as it was written initially (even if I agree it would make sense for it not to be empty).

@nmiyake
Copy link
Contributor

nmiyake commented Dec 17, 2019

I've run into this issue as well, and have been able to create a minimal repro by creating a simple Go server built using Go 1.13.* with tlsConfig.ClientAuth set to anything besides tls.NoClientCert and connecting to it using a Java 11 client (using native java.net.http.HttpClient) using OpenJDK 11.0.5_10.

This is a pretty major issue for us -- we have many Go servers that require client auth, and building them with Go 1.13 causes all clients that use Java 11 to fail.

Based on ticket, it looks like our current options are:

  • Force TLS 1.2 on the Go side by setting tls13=0 (but this will only work for Go 1.13, as this option is being removed in Go 1.14), or by setting tlsConfig.MaxVersion = tls.VersionTLS12 in the product code itself
  • Wait for resolution on the Java side and then wait for that to be rolled out across all clients

If there are any other modifications/configuration changes I can test locally please let me know.

@jnimeh
Copy link

jnimeh commented Dec 19, 2019

Hello folks, thanks for bringing this issue to our attention. I've got the issue filed and a fix that I've tested against a Go TLS server. Once I get some jtreg test code cooked up that can replicate the server-side behavior to drive the bug it'll go out for review.

Just to be clear, this will only resolve the issue of the JSSE client barfing on the status_request extension in the CR message so the handshake can complete. This won't implement client-side OCSP stapling. That'll have to come later as a new feature. I haven't filed the OpenJDK RFE for that yet, but I will.

In case anyone wishes to keep tabs on the issue from the OpenJDK side of things:
https://bugs.openjdk.java.net/browse/JDK-8236039

@kifj
Copy link

kifj commented Jan 6, 2020

For the documentation, if someone else has the same issue: As a workaround it is possible to disable TLS 1.3 on the JAVA client by setting -Djdk.tls.client.protocols=TLSv1.2

@jnimeh
Copy link

jnimeh commented Jan 6, 2020

FYI: Fix for JSSE applied last night to the mainline repository:
https://hg.openjdk.java.net/jdk/jdk/rev/b9d1ce20dd4b

@odeke-em
Copy link
Member

Thanks for the patience everyone! @jnimeh mailed out a fix that landed in the Java Secure Socket Extensions 3 weeks ago. I don't this issue is a blocker anymore for Go1.14 or perhaps should even be closed. @FiloSottile would you like to do the closing honors?

@odeke-em odeke-em modified the milestones: Go1.14, Unplanned Jan 27, 2020
teyckmans pushed a commit to kuberig-io/kuberig that referenced this issue Feb 17, 2020
- improvements to init environment for oc command line
- use namespace of kubectl config as default namespace for the initialized environment.
- in case the kubectl config does not contain a CA to trust, try to retrieve it with an SSL handshake.
- limit TLS to v1.2 - see golang/go#35722 for more information.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests