che-operator auto-grabbing of self-signed certificate does not work correctly

[sleshchenko]

Describe the bug

che-operator auto-grabbing of self-signed certificate does not work correctly.

Che version

• latest
• nightly
• other: please specify

Steps to reproduce

Generate CA and TLS certfificate:

CA_CN=minishift-signer
DOMAIN=*.192.168.99.100.nip.io

#Generate root key # add -des3 if you want to set pass phrase
openssl genrsa -out rootCA.key 4096

# Generate root cert
openssl req -x509 \
  -new -nodes \
  -key rootCA.key \
  -sha256 \
  -days 1024 \
  -out rootCA.crt \
  -subj /CN=${CA_CN} \
  -reqexts SAN \
  -extensions SAN \
  -config <(cat /etc/ssl/openssl.cnf \
      <(printf '[SAN]\nbasicConstraints=critical, CA:TRUE\nkeyUsage=keyCertSign, cRLSign, digitalSignature, keyEncipherment'))

#Create cert key for domain
openssl genrsa -out domain.key 2048

#create csr for domain
openssl req -new -sha256 \
    -key domain.key \
    -subj "/C=UA/ST=CK/O=RedHat/CN=${DOMAIN}" \
    -reqexts SAN \
    -config <(cat /etc/ssl/openssl.cnf \
        <(printf "\n[SAN]\nsubjectAltName=DNS:${DOMAIN}\nbasicConstraints=critical, CA:FALSE\nkeyUsage=keyCertSign, digitalSignature, keyEncipherment\nextendedKeyUsage=serverAuth")) \
    -out domain.csr

# create cert for domain
openssl x509 -req -extfile <(printf "subjectAltName=DNS:${DOMAIN}\nbasicConstraints=critical, CA:FALSE\nkeyUsage=keyCertSign, digitalSignature, keyEncipherment\nextendedKeyUsage=serverAuth") -days 365 -in domain.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out domain.crt

2. Deploy Che on minishift with self-signed certificate enabled:
   Configure Router with generated certificate:
   2.1

oc login -u system:admin --insecure-skip-tls-verify=true
oc project default
oc delete secret router-certs
cat domain.crt domain.key > minishift.crt
oc create secret tls router-certs --key=domain.key --cert=minishift.crt
oc rollout latest router

2.2 Deploy Che Server itself

chectl server:start --platform=minishift --installer=operator --multiuser --tls --self-signed-cert

Expected behavior

Che Server is run and it's possible to start workspace and use its functionality (like run task)

Actual

Che Server is run successfully but after starting of workspace it's not possible to run tasks, it's caused by incorrect self-signed certificate, so Theia is not able to requests WS Master.

Theia log

root ERROR Request currentWorkspace failed with error:
root INFO Error: Request 'currentWorkspace' failed
    at Proxy.<anonymous> (https://static.developers.redhat.com/che/theia_artifacts/theia.80ec0f4ae0530039ebdd.js:1:1742008)
    at e.$getCurrentWorkspace (https://static.developers.redhat.com/che/theia_artifacts/che.a7fdfd34ea10642c22a7.js:1:104113)
    at e.doInvokeHandler (https://static.developers.redhat.com/che/theia_artifacts/theia.80ec0f4ae0530039ebdd.js:1:1874314)
    at e.invokeHandler (https://static.developers.redhat.com/che/theia_artifacts/theia.80ec0f4ae0530039ebdd.js:1:1874042)
    at e.receiveRequest (https://static.developers.redhat.com/che/theia_artifacts/theia.80ec0f4ae0530039ebdd.js:1:1873174)
    at e.receiveOneMessage (https://static.developers.redhat.com/che/theia_artifacts/theia.80ec0f4ae0530039ebdd.js:1:1872647)
    at https://static.developers.redhat.com/che/theia_artifacts/theia.80ec0f4ae0530039ebdd.js:1:1871409
    at https://static.developers.redhat.com/che/theia_artifacts/theia.80ec0f4ae0530039ebdd.js:1:1874663
    at https://static.developers.redhat.com/che/theia_artifacts/theia.80ec0f4ae0530039ebdd.js:1:3715641
    at e.invoke (https://static.developers.redhat.com/che/theia_artifacts/theia.80ec0f4ae0530039ebdd.js:1:3715859)
root INFO { status: NaN,
  name: 'Error',
  message: 'unable to verify the first certificate',
  config:
   { adapter: [Function],
     transformRequest: { '0': [Function] },
     transformResponse: { '0': [Function] },
     timeout: 0,
     xsrfCookieName: 'XSRF-TOKEN',
     xsrfHeaderName: 'X-XSRF-TOKEN',
     maxContentLength: -1,
     validateStatus: [Function: validateStatus],
     headers:
      { Accept: 'application/json, text/plain, */*',
        Authorization:
         'Bearer eyJhbGciOiJSUzI1NiIsImtpbmQiOiJtYWNoaW5lX3Rva2VuIiwia2lkIjoid29ya3NwYWNlZjIyam43eGF2YTV0NHFidCJ9.eyJ3c2lkIjoid29ya3NwYWNlZjIyam43eGF2YTV0NHFidCIsInVpZCI6ImIwN2UzYTU4LWVkNTAtNGE2ZS1iZTE3LWZjZjQ5ZmY4YjI0MiIsImF1ZCI6IndvcmtzcGFjZWYyMmpuN3hhdmE1dDRxYnQiLCJuYmYiOi0xLCJ1bmFtZSI6ImFkbWluIiwiaXNzIjoid3NtYXN0ZXIiLCJleHAiOjE1OTY4MDgzMTksImlhdCI6MTU2NTI3MjMxOSwianRpIjoiM2M4OTRkMmItMmJjMi00ZmVlLThkNDYtMGUwM2FhYjFlYzYyIn0.FKA441LVoMgZeF3ww4-U9R83YKBeURuukZzd548xKdNDlG-M15a1dIi4A669oxsRW6qfv_GauzzfCzZgdlSZnfk0GyRTTLdsNFKRS5uS5poAV_r2lBgW-aSRbyjLq9xZxn32krhYebemFOBaRkHf1OsaaMDJp8xQBGap6bIkzbCrIYrLl66Cs_3RCd3xzv889vjexf3XFXKTF8JN7srM7cEpInaxJHop-ozAO7pRFAGsPTwE3QwyoLP0eb1D4B8AGoa3kTLQteOJuu3UUnO7etsdjpZnt8qtAyO6r6SiPNjoPK_RD29qREiv7-gNH0UQT6nYdh0pmmyrQJWfeL7tFw',
        'User-Agent': 'axios/0.18.0' },
     method: 'get',
     httpsAgent:
      Agent {
        _events: [Object],
        _eventsCount: 1,
        _maxListeners: undefined,
        defaultPort: 443,
        protocol: 'https:',
        options: [Object],
        requests: {},
        sockets: [Object],
        freeSockets: {},
        keepAliveMsecs: 1000,
        keepAlive: false,
        maxSockets: Infinity,
        maxFreeSockets: 256,
        maxCachedSessions: 100,
        _sessionCache: [Object] },
     baseURL: 'https://che-che.192.168.99.100.nip.io/api',
     url:
      'https://che-che.192.168.99.100.nip.io/api/workspace/workspacef22jn7xava5t4qbt',
     data: undefined },
  request:
   Writable {
     _writableState:
      WritableState {
        objectMode: false,
        highWaterMark: 16384,
        finalCalled: false,
        needDrain: false,
        ending: false,
        ended: false,
        finished: false,
        destroyed: false,
        decodeStrings: true,
        defaultEncoding: 'utf8',
        length: 0,
        writing: false,
        corked: 0,
        sync: true,
        bufferProcessing: false,
        onwrite: [Function: bound onwrite],
        writecb: null,
        writelen: 0,
        bufferedRequest: null,
        lastBufferedRequest: null,
        pendingcb: 0,
        prefinished: false,
        errorEmitted: false,
        emitClose: true,
        autoDestroy: false,
        bufferedRequestCount: 0,
        corkedRequestsFree: [Object] },
     writable: true,
     _events:
      [Object: null prototype] { response: [Function], error: [Function] },
     _eventsCount: 2,
     _maxListeners: undefined,
     _options:
      { maxRedirects: 21,
        maxBodyLength: 10485760,
        protocol: 'https:',
        path: '/api/workspace/workspacef22jn7xava5t4qbt',
        method: 'get',
        headers: [Object],
        agent: [Agent],
        auth: undefined,
        hostname: 'che-che.192.168.99.100.nip.io',
        port: null,
        nativeProtocols: [Object],
        pathname: '/api/workspace/workspacef22jn7xava5t4qbt' },
     _ended: true,
     _ending: true,
     _redirectCount: 0,
     _redirects: [],
     _requestBodyLength: 0,
     _requestBodyBuffers: [],
     _onNativeResponse: [Function],
     _currentRequest:
      ClientRequest {
        _events: [Object],
        _eventsCount: 6,
        _maxListeners: undefined,
        output: [],
        outputEncodings: [],
        outputCallbacks: [],
        outputSize: 0,
        writable: true,
        _last: true,
        chunkedEncoding: false,
        shouldKeepAlive: false,
        useChunkedEncodingByDefault: false,
        sendDate: false,
        _removedConnection: false,
        _removedContLen: false,
        _removedTE: false,
        _contentLength: 0,
        _hasBody: true,
        _trailer: '',
        finished: true,
        _headerSent: true,
        socket: [TLSSocket],
        connection: [TLSSocket],
        _header:
         'GET /api/workspace/workspacef22jn7xava5t4qbt HTTP/1.1\r\nAccept: application/json, text/plain, */*\r\nAuthorization: Bearer eyJhbGciOiJSUzI1NiIsImtpbmQiOiJtYWNoaW5lX3Rva2VuIiwia2lkIjoid29ya3NwYWNlZjIyam43eGF2YTV0NHFidCJ9.eyJ3c2lkIjoid29ya3NwYWNlZjIyam43eGF2YTV0NHFidCIsInVpZCI6ImIwN2UzYTU4LWVkNTAtNGE2ZS1iZTE3LWZjZjQ5ZmY4YjI0MiIsImF1ZCI6IndvcmtzcGFjZWYyMmpuN3hhdmE1dDRxYnQiLCJuYmYiOi0xLCJ1bmFtZSI6ImFkbWluIiwiaXNzIjoid3NtYXN0ZXIiLCJleHAiOjE1OTY4MDgzMTksImlhdCI6MTU2NTI3MjMxOSwianRpIjoiM2M4OTRkMmItMmJjMi00ZmVlLThkNDYtMGUwM2FhYjFlYzYyIn0.FKA441LVoMgZeF3ww4-U9R83YKBeURuukZzd548xKdNDlG-M15a1dIi4A669oxsRW6qfv_GauzzfCzZgdlSZnfk0GyRTTLdsNFKRS5uS5poAV_r2lBgW-aSRbyjLq9xZxn32krhYebemFOBaRkHf1OsaaMDJp8xQBGap6bIkzbCrIYrLl66Cs_3RCd3xzv889vjexf3XFXKTF8JN7srM7cEpInaxJHop-ozAO7pRFAGsPTwE3QwyoLP0eb1D4B8AGoa3kTLQteOJuu3UUnO7etsdjpZnt8qtAyO6r6SiPNjoPK_RD29qREiv7-gNH0UQT6nYdh0pmmyrQJWfeL7tFw\r\nUser-Agent: axios/0.18.0\r\nHost: che-che.192.168.99.100.nip.io\r\nConnection: close\r\n\r\n',
        _onPendingData: [Function: noopPendingOutput],
        agent: [Agent],
        socketPath: undefined,
        timeout: undefined,
        method: 'GET',
        path: '/api/workspace/workspacef22jn7xava5t4qbt',
        _ended: false,
        res: null,
        aborted: undefined,
        timeoutCb: null,
        upgradeOrConnect: false,
        parser: null,
        maxHeadersCount: null,
        _redirectable: [Circular],
        [Symbol(isCorked)]: false,
        [Symbol(outHeadersKey)]: [Object] },
     _currentUrl:
      'https://che-che.192.168.99.100.nip.io/api/workspace/workspacef22jn7xava5t4qbt' } }
root ERROR Request currentWorkspace failed with error:
root INFO Error: Request 'currentWorkspace' failed
    at Proxy.<anonymous> (https://static.developers.redhat.com/che/theia_artifacts/theia.80ec0f4ae0530039ebdd.js:1:1742008)
    at e.$getCurrentWorkspace (https://static.developers.redhat.com/che/theia_artifacts/che.a7fdfd34ea10642c22a7.js:1:104113)
    at e.doInvokeHandler (https://static.developers.redhat.com/che/theia_artifacts/theia.80ec0f4ae0530039ebdd.js:1:1874314)
    at e.invokeHandler (https://static.developers.redhat.com/che/theia_artifacts/theia.80ec0f4ae0530039ebdd.js:1:1874042)
    at e.receiveRequest (https://static.developers.redhat.com/che/theia_artifacts/theia.80ec0f4ae0530039ebdd.js:1:1873174)
    at e.receiveOneMessage (https://static.developers.redhat.com/che/theia_artifacts/theia.80ec0f4ae0530039ebdd.js:1:1872647)
    at https://static.developers.redhat.com/che/theia_artifacts/theia.80ec0f4ae0530039ebdd.js:1:1871409
    at https://static.developers.redhat.com/che/theia_artifacts/theia.80ec0f4ae0530039ebdd.js:1:1874663
    at https://static.developers.redhat.com/che/theia_artifacts/theia.80ec0f4ae0530039ebdd.js:1:3715641
    at e.invoke (https://static.developers.redhat.com/che/theia_artifacts/theia.80ec0f4ae0530039ebdd.js:1:3715859)
root ERROR [hosted-plugin: 48] Failed to get workspace configuration

Runtime

• kubernetes (include output of kubectl version)
• Openshift (include output of oc version)
• minikube (include output of minikube version and kubectl version)
• minishift (include output of minishift version and oc version)
• docker-desktop + K8S (include output of docker version and kubectl version)
• other: (please specify)

Screenshots

Installation method

• chectl
• che-operator
• minishift-addon
• I don't know

Environment

• my computer
  • Windows
  • Linux
  • macOS
• Cloud
  • Amazon
  • Azure
  • GCE
  • other (please specify)
• other: please specify

Additional context

Note that if you create self-signed-cert based on rootCA before deploying Che

oc create namespace che
cp rootCA.crt ca.crt
oc create secret self-signed-cert generic --from-file=ca.crt

the che-operator will use it instead of auto-grabbing and Che Server and workspaces will work correctly.

I've already investigated this issue a bit and discovered that it's like a best practice to generate CA certificate, propagate it to clients to configure their trust stores.
And generate another non-CA certificate based on CA for establishing https connection.
See https://wiki.mozilla.org/SecurityEngineering/x509Certs
https://gist.github.com/fntlnz/cf14feb5a46b2eda428e000157447309

Looks like Java and Go does not have any issues with using non-CA certificate as trusted,
but the current implementation of Theia - has.
Possible solutions:

1. Delete auto-grabbing self-signed cert by che-operator and always asks user to provide it as pre-created secret.
2. Try to fix this issue on Che Server + Theia side. Looks like a library which we use for Theia provides an ability to configure CA or just cert, but they must be provided as different fields.
   So, Che Server may inform Theia which cert type is used, or maybe Theia may decode it by itself.
   Note that I took a look on curl command and there is no ability to use non-CA cert for trusting. The only option that can be used cacert
   

[rhopp]

Seems pretty complicated issue and I don't think we can do it properly in 7.0 release...
And as a workaround exists (if I understood it correctly), I'm setting 7.1 as a milestone for this issue.

[sleshchenko]

@rhopp

And as a workaround exists (if I understood it correctly)

yes, a user has to pre-created secret with CA certificate before deploying Che. I do not think that it's documented somewhere but a workaround exists.

[bryantson]

@sleshchenko Hello, can you explain what is the workaround?

[sleshchenko]

@bryantson

a user has to pre-created secret with CA certificate before deploying Che.

So, user should execute

# this is from issue description
oc create namespace che
cp rootCA.crt ca.crt
oc create secret self-signed-cert generic --from-file=ca.crt

and only then deploy Che with operator, like

chectl server:start --self-signed-cert --tls --multiuser --platform=openshift --installer=operator

[azatsarynnyy]

removed area/che-theia as looks like it's not related

[tolusha]

@mmorhun
Could you have a look pls

[tolusha]

duplicate
#16280