Hybrid Ingress Self signed Cert Walkthrough
We are going to visit all stops on creating and configuring self-signed certificates for Istio Ingress configuration.
TODO: apigeeroute
As usual, we also will refresh and collect here useful troubleshooting commands.
In a generic case, there is a CA root key that is used generate a ca authority certificate. The ca key is used to sign an intermediate certificate request to produce an intermediate certificate and there is an intermediate key that was used to generate an intermediate certificate authority request, that is used to sign a leaf certificate request that was signed by a leaf key, that is used to sign a certificate request to generate a leaf certificate. The ca, intermediate, and leaf certificates comprise a trust chain that is used to validate the certificate. Self-signed certificates do not change the mechanics of PKI framework. They merely make CA root certificate configuration step explicit.
If the sentence above sounds like a mouthful and feels complex, that because, without some investment and practice, it is. Keep reading on PKI until you can easily explain it to a colleague during an office party.
In practice, in many cases it makes sense to cut some corners and generate a single self-signed certificate that would be used as both, ca root certificate, and the trust chain.
If not configured correctly, your certificate will be rejected. Stackoverflow is full of advice to use -k to 'solve' this problem. Don't! The ignorance is never a good solution, especially in matters of security.
The problem is that nowadays there are some security-related predicates and conditions that need to be satisfied before you can add --cafile option and drop -k option.
Originally, a simple openssl command was needed to create a certificate you were able to trust and use. As the time goes by, vulnerabilities, computational progress, and security best practices gradually introduce new requirements for a well-defined certificate. It happens in the form of checks that OSs, Browsers, and/or underlying libraries that PKI-related tools are using. If a check failed, the certificate is not accepted as valid.
Only some examples:
- SANs are mandatory;
- basicConstrains and keyUsage declarations as appropriate are required;
- Minimal strength ciphers and key lengths are cryptographic properties need to be considered;
- Digital signature algorithms selection (RSA vs ECDSA).
export PROJECT=apigee-sme-academy-emea-anthos
export REGION=europe-west1
export HYBRID_HOME=~/qwiklabs-hybrid-player-envs/$PROJECT
export ORG=$PROJECT
export ENV=default
export RUNTIME_HOST_ALIAS=api.exco.com
export RUNTIME_SSL_CERT=$HYBRID_HOME/exco-hybrid-crt.pem
export RUNTIME_SSL_KEY=$HYBRID_HOME/exco-hybrid-key.pem
gcloud config set project $PROJECT
curl --cacert $RUNTIME_SSL_CERT https://$RUNTIME_HOST_ALIAS/ping -v --resolve "$RUNTIME_HOST_ALIAS:443:$RUNTIME_IP" --http1.1
* Added api.exco.com:443:35.205.115.132 to DNS cache
* Hostname api.exco.com was found in DNS cache
* Trying 35.205.115.132...
* TCP_NODELAY set
* Connected to api.exco.com (35.205.115.132) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /home/yuriyl/qwiklabs-hybrid-player-envs/apigee-sme-academy-emea-anthos/exco-hybrid-crt.pem
CApath: /etc/ssl/certs
...
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
...
We now can save the certificate we want to look closer at using openssl s_client command.
openssl s_client -showcerts -connect $RUNTIME_IP:443 -servername $RUNTIME_HOST_ALIAS
CONNECTED(00000003)
depth=0 CN = api.exco.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = api.exco.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:/CN=api.exco.com
i:/CN=api.exco.com
-----BEGIN CERTIFICATE-----
MIIC7jCCAdagAwIBAgIJAOrQpN22klnHMA0GCSqGSIb3DQEBCwUAMBcxFTATBgNV
BAMMDGFwaS5leGNvLmNvbTAeFw0yMDA1MjcxMjExNDdaFw0yMDA2MjYxMjExNDda
MBcxFTATBgNVBAMMDGFwaS5leGNvLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEP
ADCCAQoCggEBALMUfQw4LfOPNX0sNEgVOUUOOSp87ZMOx/O4CuwCFlLMYJz65YP/
FiIzIdQYKjZz5sBkDZln+3hsGevQYs86AA+HyPwTrJh77wo96MTnGMWZEtSRgkzw
Y8NkLKGARTX42pccAh0fSjTRYD67XCbW8krCS3jzEqWjKZCgmRSUTjq5N4VPoun6
sa0cHVukDkkphl/uLm2TIh+jJWij21v4GrKu6nGfnGffoxYQsvs36fUw2j3e3oXv
9hG+WEa7sQ0f0GhCsFx9qvDEU2IJPZV6peTME3/Q3hQ24UJoH2TvP2Kq9zELVRXL
JXmxfK4qkwqt6qpDNa9A89rux2h/iY6MfvMCAwEAAaM9MDswFwYDVR0RBBAwDoIM
YXBpLmV4Y28uY29tMAsGA1UdDwQEAwIHgDATBgNVHSUEDDAKBggrBgEFBQcDATAN
BgkqhkiG9w0BAQsFAAOCAQEAKnsikR1lxemiodzaxHM8Hpb+QptEM68fZCPJzM3W
8WDyL+gWEyRF6hnYSs/2d7MV1NODpxlZf/JcZTDHlj7hhnrpccyiTaowkxoykHOM
00sNlV4xUIb48JzDqokMulxnTlSfR0mNNRZiLZhnkrVxGFIjsh+q+KXrlXe/1laV
VwyMJb492kVx8Wk0kmIbS4nujIoui9jmMnUp3+r72ee+bT+2H+799hF1AWFttDzC
2+y6RF7DukYz/c13wAfDh2iH4BvB9JLi/ZR47sVi2SqUca6M/n/BpJO4DwKQrO9U
4VoYs4OR1fb4MQgXIPbZd+hsNuyzbLzRSciNnby0Uwz+qg==
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=api.exco.com
issuer=/CN=api.exco.com
---
No client certificate CA names sent
Peer signing digest: SHA256
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1244 bytes and written 394 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID:
Session-ID-ctx:
Resumption PSK:
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1591188011
Timeout : 7200 (sec)
Verify return code: 21 (unable to verify the first certificate)
Extended master secret: no
Max Early Data: 0
---
We can spot same verification error, but a bit better worded:
Verification error: unable to verify the first certificate
?. Copy and save the certificate in the bad-cert.pem file.
?. Look at the certificate contents
openssl x509 -in bad-cert.pem -text -noout
62:cf:3a:00:0f:87:c8:fc:13:ac:98:7b:ef:0a:3d:
<snip>
7e:f3
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Alternative Name:
DNS:api.exco.com
X509v3 Key Usage:
Digital Signature
X509v3 Extended Key Usage:
TLS Web Server Authentication
Signature Algorithm: sha256WithRSAEncryption
2a:7b:22:91:1d:65:c5:e9:a2:a1:dc:da:c4:73:3c:1e:96:fe:
<snip>
53:0c:fe:aa
We can see that openssl ssl library cannot identify this certificate as CA authority one. There is no CA:TRUE basic constraint. There is no keyCertSign keyUsage attribute.
We still can keep simple use cases simple. The one-liner of today looks like follows:
openssl req -x509 -out $RUNTIME_SSL_CERT -keyout $RUNTIME_SSL_KEY -newkey rsa:2048 -nodes -sha256 -subj '/CN=api.exco.com' -extensions EXT -config <( printf "[dn]\nCN=api.exco.com\n[req]\ndistinguished_name=dn\n[EXT]\nbasicConstraints=critical,CA:TRUE,pathlen:1\nsubjectAltName=DNS:api.exco.com\nkeyUsage=digitalSignature,keyCertSign\nextendedKeyUsage=serverAuth")
The key features to observe are:
- both, key and certificate are generated
- critical, CA:TRUE basic constraint;
- keyUsage: digitalSignature, keyCertSign;
- SAN name is used;
- extendedKeyUsage: serverAuth.
- RSA 2048 key length
- hash sha256
Please, take your time to locate all components of the certificate in the openssl x509 command output.
openssl x509 -in $RUNTIME_SSL_CERT -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
ab:3e:0c:1e:65:6f:30:58
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = api.exco.com
Validity
Not Before: Jun 3 18:12:18 2020 GMT
Not After : Jul 3 18:12:18 2020 GMT
Subject: CN = api.exco.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:e5:51:16:66:5f:45:22:34:3f:32:17:76:f4:4d:
<snip>
de:a3
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints: critical
CA:TRUE, pathlen:1
X509v3 Subject Alternative Name:
DNS:api.exco.com
X509v3 Key Usage:
Digital Signature, Certificate Sign
X509v3 Extended Key Usage:
TLS Web Server Authentication
Signature Algorithm: sha256WithRSAEncryption
ad:2d:0c:55:d1:63:2f:8a:f5:c3:2b:80:1c:51:d0:1c:0b:27:
<snip>
cb:e0:46:9d
org: $ORG
virtualhosts:
- name: default
hostAliases:
- "$RUNTIME_HOST_ALIAS"
sslCertPath: $RUNTIME_SSL_CERT
sslKeyPath: $RUNTIME_SSL_KEY
routingRules:
- paths:
- /
env: $ENV
The Gateway manifest is generated during apigeectl apply command execution by processing templates/virtualhosts.yaml template for kind: ApigeeRoute CRD.
?. Get list of gateways
kubectl get gw -n apigee
NAME AGE
apigee-mart-apigee-sme-academy-emea-anthos 7d
apigee-sme-academy-emea-anthos-default 7d
Get details of api endpoint gateway
kubectl get gw apigee-sme-academy-emea-anthos-default -n apigee -o yaml
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
creationTimestamp: "2020-05-27T12:14:45Z"
generation: 1
name: apigee-sme-academy-emea-anthos-default
namespace: apigee
ownerReferences:
- apiVersion: apigee.cloud.google.com/v1alpha1
blockOwnerDeletion: true
kind: ApigeeRoute
name: apigee-sme-academy-emea-anthos-default
uid: a7113466-a013-11ea-a670-42010a84005e
resourceVersion: "13019"
selfLink: /apis/networking.istio.io/v1alpha3/namespaces/apigee/gateways/apigee-sme-academy-emea-anthos-default
uid: a72be4f9-a013-11ea-a670-42010a84005e
spec:
selector:
app: istio-ingressgateway
servers:
- hosts:
- api.exco.com
port:
name: apigee-https-443
number: 443
protocol: HTTPS
tls:
credentialName: apigee-sme-academy-emea-anthos-default-certs
mode: SIMPLE
- hosts:
- api.exco.com
port:
name: apigee-http-80
number: 80
protocol: HTTP
tls:
httpsRedirect: true
kubectl get secret $ORG-$ENV-certs -n istio-system -o yaml
apiVersion: v1
data:
cert: LS0tLS<snip>0tLS0tCg==
key: LS0tLS1C<snip>LS0tLS0K
kind: Secret
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"cert":"LS0tLS1<snip>0tLS0K\n"},"kind":"Secret","metadata":{"annotations":{},"name":"apigee-sme-academy-emea-anthos-default-certs","namespace":"istio-system"},"type":"Opaque"}
creationTimestamp: "2020-05-27T12:14:45Z"
name: apigee-sme-academy-emea-anthos-default-certs
namespace: istio-system
resourceVersion: "13013"
selfLink: /api/v1/namespaces/istio-system/secrets/apigee-sme-academy-emea-anthos-default-certs
uid: a70635cf-a013-11ea-a670-42010a84005e
type: Opaque
apigectl generates secrete using following template from templates/virtualhosts.yaml:
- for 1.3.x and later:
{{- if $.Org }}
{{- range $index, $vh := $.Virtualhosts }}
{{- $certName := (printf "%s-%s" $.Org $vh.Name) }}
{{- if and (not $vh.SSLSecret) (and $vh.SSLCertPath $vh.SSLKeyPath) }}
---
## tls certs secret for {{ $.Org }}-{{ $vh.Name }}
apiVersion: v1
kind: Secret
metadata:
name: "{{ $certName }}"
namespace: {{ $.IstioNamespace }}
type: Opaque
data:
cert: |
{{ loadFileForSecret $vh.SSLCertPath }}
key: |
{{ loadFileForSecret $vh.SSLKeyPath }}
{{- if $vh.CaCertPath }}
cacert: |
{{ loadFileForSecret $vh.CaCertPath }}
{{- end }}
- for 1.2.x and earlier:
## tls certs secret for {{ $.org }}-{{ $vh.Name }}
apiVersion: v1
kind: Secret
metadata:
name: {{ $.org }}-{{ $vh.Name }}-certs
namespace: {{ $.istioNamespace }}
type: Opaque
data:
cert: |
{{ loadFileForSecret $vh.SSLCertPath }}
key: |
{{ loadFileForSecret $vh.SSLKeyPath }}
---
{{- end }}
NOTE: We are using SDS/credentialName to provide zero downtime for certificate rotation.
kubectl -n istio-system delete secret $ORG-$ENV_GROUP
kubectl -n istio-system create secret tls $ORG-$ENV_GROUP --key $RUNTIME_SSL_KEY --cert $RUNTIME_SSL_CERT
[1.2.x and earlier]
kubectl -n istio-system delete secret $ORG-$ENV-certs
kubectl -n istio-system create secret tls $ORG-$ENV-certs --key $RUNTIME_SSL_KEY --cert $RUNTIME_SSL_CERT
?. Notice some unique sequence in a .pem file and verify the correct certificate is provided by the server
openssl s_client -showcerts -connect $RUNTIME_IP:443 -servername $RUNTIME_HOST_ALIAS
curl --cacert $RUNTIME_SSL_CERT https://$RUNTIME_HOST_ALIAS/ping -v --resolve "$RUNTIME_HOST_ALIAS:443:$RUNTIME_IP" --http1.1
* Added api.exco.com:443:35.205.115.132 to DNS cache
* Hostname api.exco.com was found in DNS cache
* Trying 35.205.115.132...
* TCP_NODELAY set
* Connected to api.exco.com (35.205.115.132) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /home/yuriyl/qwiklabs-hybrid-player-envs/apigee-sme-academy-emea-anthos/exco-hybrid-crt.pem
CApath: /etc/ssl/certs
<snip>
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: CN=api.exco.com
* start date: Jun 3 18:12:18 2020 GMT
* expire date: Jul 3 18:12:18 2020 GMT
* subjectAltName: host "api.exco.com" matched cert's "api.exco.com"
* issuer: CN=api.exco.com
* SSL certificate verify ok.
> GET /ping HTTP/1.1
> Host: api.exco.com
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 200 OK
< host: api.exco.com
<snip>
* Curl_http_done: called premature == 0
* Connection #0 to host api.exco.com left intact
curl --version | grep -E 'OpenSSL|LibreSSL'
Use --dry-run=client -o yaml to capture a manifest output. Use | kubectl apply -f - to pipe it to apply.
https://istio.io/docs/tasks/traffic-management/ingress/secure-ingress/
AHR-*-CTL - TADA: Apigee Hybrid Container Traffic Analysis with tcpdump for Target Requests
- TADAA: Cloud Code IDE Java Callout Debugging
- Hybrid Ingress Walkthrough 1.5
- Hybrid CRD Objects Diagram 1.3.2
- List of Components for Air-gapped Deployments
- GKE/EKS Multi-cloud Small Topology 1.4 [single-project] | AHR Profile Quick Start
- Single-Zone Cluster, Small footprint 1.1
- Multi-Zone Cluster, Large footprint 1.1
- Multi-Region Cluster, Large footprint 1.1
-
Private Cluster
-
Performance Testing: distGatling
- Delete Hybrid Installation
- Hybrid Ingress Walkthrough 1.3