jetty:9-jre8 SSL no longer works, FIN sent in reply to CLIENT_HELLO

[joakime]

Issue by draeath
Friday Mar 22, 2019 at 18:24 GMT
Originally opened as appropriate/docker-jetty#104

---

Specific image ID: 1dc9280cc083

Jetty starts, and the contexts appear to come up. However, all attempts to access them via SSL fail as follows. Yes, the port is exposed and not firewalled etc - this happens even via localhost.

EDIT: confirmed. 8u181-b13-1~deb9u1 works, 8u181-b13-2~deb9u1 BREAKS, 8u212-b01-1~deb9u1 works again (caveat: have to add an alpn-impl pointing at alpn-boot-8.1.13.v20181017)

draeath@ginnungagap:~/scratch$ curl --insecure --ciphers ALL -I -v https://REDACTED:44420/cas
*   Trying REDACTED...
* TCP_NODELAY set
* Connected to REDACTED (REDACTED) port 44420 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to REDACTED:44420 
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
curl: (35) Unknown SSL protocol error in connection to REDACTED:44420

Note if I leave the ciphers parameter off, the defaults fail in the same way.

Packet capture shows:

-> SYN
<- SYN,ACK
-> ACK
-> TLSv1 Client Hello
<- ACK
<- FIN, ACK
-> FIN, ACK
<- ACK

There is no log, STDOUT, or STDERR emissions from Jetty when this occurs.

---

Digging around, I discovered the following. I have an older build of this software that's working fine. I performed a docker container export of both the old build and the new one (which fails), and did a recursive diff between. Ignoring binary differences, I found the following differences, and only the following differences, presumably from upstream openjdk:8-jre

• curl, libcurl3 packages upgraded from 7.52.1-5+deb9u6 to 7.52.1-5+deb9u7
• openjdk-8-jre, openjdk-8-jre-headless packages upgraded from 8u181-b13-1~deb9u1 to 8u181-b13-2~deb9u1

Debian package changelogs for these show the curl change seems unrelated:

curl (7.52.1-5+deb9u7) stretch-security; urgency=high

  * Fix NTLM password overflow via integer overflow as per CVE-2018-14618
    https://curl.haxx.se/docs/CVE-2018-14618.html

 -- Alessandro Ghedini <ghedo@debian.org>  Mon, 03 Sep 2018 23:50:29 +0100

However the java changelog seems particularly relevant:

openjdk-8 (8u181-b13-2) unstable; urgency=high

  [ Tiago Stürmer Daitx ]
  * Apply patches from 8u191-b12 security update.
    - CVE-2018-3136, S8194534: Manifest better support.
    - CVE-2018-3139, S8196902: Better HTTP Redirection.
    - CVE-2018-3149, S8199177: Enhance JNDI lookups.
    - CVE-2018-3169, S8199226: Improve field accesses.
    - CVE-2018-3180, S8202613: Improve TLS connections stability.
    - CVE-2018-3183, S8202936: Improve script engine support.
    - CVE-2018-3214, S8205361: Better RIFF reading support.
    - CVE-2018-3211: Unspecified vulnerability in the Serviceability component.
    - S8195868: Address Internet Addresses.
    - S8195874: Improve jar specification adherence.
    - S8201756: Improve cipher inputs.
    - S8203654: Improve cypher state updates.
    - S8204497: Better formatting of decimals.
  * debian/patches/jdk-freetypeScaler-crash.diff: removed as this patch causes 
    a memory leak; upstream fixed it in openjdk-7, albeit in a different way.
    Closes: #910672.

  [ Matthias Klose ]
  * Bump standards version.

 -- Matthias Klose <doko@ubuntu.com>  Sun, 21 Oct 2018 12:23:32 +0200

Specifically this update touches several areas around TLS/SSL.

Now, for some of my local info for context.

---

Dockerfile:

# ---- runtime ---- #
FROM jetty:9-jre8 AS runtime

USER root
# mountpoint for runtime volume - easy persistent logs
RUN mkdir -pv /mnt/jetty-logs && chown -v jetty:jetty /mnt/jetty-logs

USER jetty
ENV JAVA_OPTIONS "-Xmx512m -Xms512m"
RUN java -jar ${JETTY_HOME}/start.jar --add-to-start=https,http2 --approve-all-licenses
RUN rm -v ${JETTY_BASE}/start.d/http.ini
ADD --chown=jetty:jetty cas-redirect.war /var/lib/jetty/webapps/ROOT.war
COPY --chown=jetty:jetty cas.war /var/lib/jetty/webapps/cas.war
ADD --chown=jetty:jetty etc/cas /etc/cas
ADD --chown=jetty:jetty keystore /var/lib/jetty/etc/keystore
ADD --chown=jetty:jetty ssl.ini /var/lib/jetty/start.d/ssl.ini

Referenced ssl.ini content:

jetty.sslContext.keyStorePassword=REDACTED
jetty.sslContext.keyManagerPassword=REDACTED

• 'cas.war' is a build of https://apereo.github.io/cas/5.3.x/index.html but this issue happens even if this context is omitted
• 'cas-redirect.war' is a custom app, super simple (just takes anything from it's own context and 302's it to the /cas context) and is installed as the root context. I will reply with a comment shortly with this code, to show that it's not doing anything fancy.

keystore is valid with one private key and public cert pair:

draeath@ginnungagap:~/scratch$ keytool -list -keystore ./keystore
Enter keystore password: REDACTED
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

REDACTED, Aug 10, 2018, PrivateKeyEntry, 
Certificate fingerprint (SHA-256): REDACTED

[joakime]

Comment by draeath
Friday Mar 22, 2019 at 18:28 GMT

---

For completeness, this is the cas-redirect.war code. It does nothing fancy whatsoever. This problem exists even if this is the only context, and it's put down as ROOT.war.

Done this to rule out the war files I'm trying to use as the source of this trouble.

---

pom.xml:

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>REDACTED</groupId>
  <artifactId>cas-redirect</artifactId>
  <version>1.0</version>
  <packaging>war</packaging>

  <name>cas-redirect</name>
  <url>REDACTED</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
    <isExecutable>false</isExecutable>
  </properties>

  <dependencies>
    <dependency>
      <groupId>javax.servlet</groupId>
      <artifactId>javax.servlet-api</artifactId>
      <version>3.1.0</version>
      <scope>provided</scope>
    </dependency>
  </dependencies>

  <build>
    <finalName>cas-redirect</finalName>
    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-war-plugin</artifactId>
          <version>2.6</version>
        </plugin>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-compiler-plugin</artifactId>
          <version>3.3</version>
        </plugin>
      </plugins>
    </pluginManagement>
  </build>
</project>

---

src/main/java/REDACTED/cas/Redirect.java:

package REDACTED.cas;

import java.io.IOException;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.util.logging.Logger;
import java.util.logging.Level;

public class Redirect extends HttpServlet {
  protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
    final Logger logger = Logger.getLogger(Redirect.class.getName());

    String forwardedForHeader = request.getHeader("X-FORWARDED-FOR");
    String clientAddress = (forwardedForHeader != null) ? request.getRemoteAddr() + " (X-FORWARDED-FOR: " + forwardedForHeader + ")" : request.getRemoteAddr();
    logger.log(Level.WARNING,"redirecting out-of-context request from " + clientAddress + " for " + request.getRequestURL());

    StringBuffer replyURI = new StringBuffer("/cas");
    StringBuffer requestedURI = new StringBuffer(request.getRequestURI());
    String[] requestedURIparts = requestedURI.toString().split("/");
    boolean hasQueryString = new Boolean(request.getQueryString() != null);

    if (requestedURIparts.length > 2) {
      for (int i = 2; i < requestedURIparts.length; i++) {
        replyURI.append("/" + requestedURIparts[i]);
      }
    }

    if (hasQueryString) {
      replyURI.append('?' + request.getQueryString());
    }

    response.sendRedirect(replyURI.toString());
  }
}

---

src/main/webapp/WEB-INF/web.xml:

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd" metadata-complete="true" version="3.0">
   <display-name>CAS Redirect</display-name>
   <servlet>
     <servlet-name>redirect</servlet-name>
     <servlet-class>REDACTED.cas.Redirect</servlet-class>
   </servlet>
   <servlet-mapping>
     <servlet-name>redirect</servlet-name>
     <url-pattern>/*</url-pattern>
   </servlet-mapping>
</web-app>

[joakime]

Comment by draeath
Friday Mar 22, 2019 at 19:35 GMT

---

I'm going to replicate the jetty Dockerfile build using an older openjdk Dockerfile build, to validate.

[joakime]

Comment by draeath
Friday Mar 22, 2019 at 20:30 GMT

---

Couldn't roll back, debian's security repo doesn't have that old package. Can only go forward to 8u212

[joakime]

Comment by draeath
Friday Mar 22, 2019 at 20:49 GMT

---

So, I have no idea why, but the syntax for the GPG command needed to be slightly changed. Namely I needed to provide the key IDs differently, instead of just the fingerprint as a string.

diff --git a/9.4-jre8/Dockerfile b/9.4-jre8/Dockerfile
index ab80fce..dbc8286 100644
--- a/9.4-jre8/Dockerfile
+++ b/9.4-jre8/Dockerfile
@@ -14,21 +14,21 @@ ENV JETTY_TGZ_URL https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-home/$J
 # GPG Keys are personal keys of Jetty committers (see https://github.com/eclipse/jetty.project/blob/0607c0e66e44b9c12a62b85551da3a0edce0281e/KEYS.txt)
 ENV JETTY_GPG_KEYS \
        # Jan Bartel      <janb@mortbay.com>
-       AED5EE6C45D0FE8D5D1B164F27DED4BF6216DB8F \
+       0x27DED4BF6216DB8F \
        # Jesse McConnell <jesse.mcconnell@gmail.com>
-       2A684B57436A81FA8706B53C61C3351A438A3B7D \
+       0x61C3351A438A3B7D \
        # Joakim Erdfelt  <joakim.erdfelt@gmail.com>
-       5989BAF76217B843D66BE55B2D0E1FB8FE4B68B4 \
+       0x2D0E1FB8FE4B68B4 \
        # Joakim Erdfelt  <joakim@apache.org>
-       B59B67FD7904984367F931800818D9D68FB67BAC \
+       0x0818D9D68FB67BAC \
        # Joakim Erdfelt  <joakim@erdfelt.com>
-       BFBB21C246D7776836287A48A04E0C74ABB35FEA \
+       0xA04E0C74ABB35FEA \
        # Simone Bordet   <simone.bordet@gmail.com>
-       8B096546B1A8F02656B15D3B1677D141BCF3584D \
+       0x1677D141BCF3584D \
        # Greg Wilkins    <gregw@webtide.com>
-       FBA2B18D238AB852DF95745C76157BDF03D0DCD6 \
+       0x76157BDF03D0DCD6 \
        # Greg Wilkins    <gregw@webtide.com>
-       5C9579B3DB2E506429319AAEF33B071B29559E1E
+       0xF33B071B29559E1E
 
 RUN set -xe \
        && curl -SL "$JETTY_TGZ_URL" -o jetty.tar.gz \

[joakime]

Comment by draeath
Friday Mar 22, 2019 at 21:23 GMT

---

OK! So bumping the JRE to 8u212 upstream of your own 9-jre8 Dockerfile fixes this issue

Note that I did have to drop in an ALPN for this java version - but it seems to work OK. I can't find any details on changes for this java release to really dig any deeper to see if it's really compatible - my gut tells me it's just 202 with the new license.

[joakime]

Comment by draeath
Tuesday Mar 26, 2019 at 20:17 GMT

---

I suspect the security fixes backported by the debian security team break the mapping of 8u181 to alpn-boot 8.1.12.v20180117.

[stale]

This issue has been automatically marked as stale because it has been a full year without activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been closed due to it having no activity.