Can't get Kong 0.14.0 to serve my certificates on HTTPS #3721

itaimalek · 2018-08-22T12:20:35Z

Summary

After configuring Certs with SNIs Kong route is ignoring the configuration and serves localhost cert (default)

Steps To Reproduce

Configure .pem and .key strings to Certificate resource in Kong with SNI
Configure a Service and Route objects with Host in Route as in above SNI with https enabled for the Route
Try to access via https

Expected result

Kong will serve the certificate that it's SNI fits the Host of the Route.

Additional Details & Logs

Kong version - 0.14.0
setup is on kubernetes with postgres DB.
HTTP access is good- I get response as expected.

my Service object:

{
            "host": "some.service.domain",
            "connect_timeout": 60000,
            "id": <service-id>,
            "protocol": "http",
            "name": <service-name>,
            "read_timeout": 60000,
            "port": 80,
            "path": "/",
            "updated_at": 1534887459,
            "retries": 5,
            "write_timeout": 60000
        }

my Route object:

{
    "created_at": 1534938341,
    "strip_path": false,
    "hosts": [
        "host.some.domain"
    ],
    "preserve_host": false,
    "regex_priority": 0,
    "updated_at": 1534938341,
    "paths": [
        "/some-path"
    ],
    "service": {
        "id": <service-id>
    },
    "methods": [
        "GET",
        "POST"
    ],
    "protocols": [
        "http",
        "https"
    ],
    "id": <route ID>
}

my SNI object:

{
            "certificate": {
                "id": <cert object ID>
            },
            "created_at": 1534852534,
            "name": "host.some.domain",
            "id": <SNI ID>
        }

The text was updated successfully, but these errors were encountered:

ionosphere80 · 2018-08-22T17:17:51Z

How did you add the certificate(s) to Kong?

itaimalek · 2018-08-22T17:22:20Z

Hi, Thanks for replying, I added certificates just like any other object, via kong admin api.

ionosphere80 · 2018-08-22T17:26:34Z

Right... but did you use a JSON file with the application/json content type?

itaimalek · 2018-08-22T17:45:07Z

yes, I did a POST request with body like:

{ "cert": "<.pem file content>", "key": "<key file content>", "snis": [ "<host.some.domain>" ] }
also, the SNI was created from this request.
I didn't use a json file, the JSON object was sent via the request itself.
Also when doing GET to kong-admin-api/certificates, I can see my certificate there.

itaimalek · 2018-08-22T17:57:23Z

adding my kong configurations:

{
    "plugins": {
        "enabled_in_cluster": [],
        "available_on_server": {
            "response-transformer": true,
            "oauth2": true,
            "acl": true,
            "correlation-id": true,
            "pre-function": true,
            "jwt": true,
            "cors": true,
            "ip-restriction": true,
            "basic-auth": true,
            "key-auth": true,
            "rate-limiting": true,
            "request-transformer": true,
            "http-log": true,
            "file-log": true,
            "hmac-auth": true,
            "ldap-auth": true,
            "datadog": true,
            "tcp-log": true,
            "zipkin": true,
            "post-function": true,
            "request-size-limiting": true,
            "bot-detection": true,
            "syslog": true,
            "loggly": true,
            "azure-functions": true,
            "udp-log": true,
            "response-ratelimiting": true,
            "aws-lambda": true,
            "statsd": true,
            "prometheus": true,
            "request-termination": true
        }
    },
    "tagline": "Welcome to kong",
    "configuration": {
        "plugins": [
            "bundled"
        ],
        "admin_ssl_enabled": true,
        "lua_ssl_verify_depth": 1,
        "trusted_ips": {},
        "prefix": "/usr/local/kong",
        "loaded_plugins": {
            "response-transformer": true,
            "request-termination": true,
            "prometheus": true,
            "ip-restriction": true,
            "pre-function": true,
            "jwt": true,
            "cors": true,
            "statsd": true,
            "basic-auth": true,
            "key-auth": true,
            "ldap-auth": true,
            "aws-lambda": true,
            "http-log": true,
            "response-ratelimiting": true,
            "hmac-auth": true,
            "request-size-limiting": true,
            "datadog": true,
            "tcp-log": true,
            "zipkin": true,
            "post-function": true,
            "bot-detection": true,
            "acl": true,
            "loggly": true,
            "syslog": true,
            "azure-functions": true,
            "udp-log": true,
            "file-log": true,
            "request-transformer": true,
            "correlation-id": true,
            "rate-limiting": true,
            "oauth2": true
        },
        "cassandra_username": "kong",
        "admin_ssl_cert_csr_default": "/usr/local/kong/ssl/admin-kong-default.csr",
        "ssl_cert_key": "/usr/local/kong/ssl/kong-default.key",
        "admin_ssl_cert_key": "/usr/local/kong/ssl/admin-kong-default.key",
        "dns_resolver": {},
        "pg_user": "kong",
        "mem_cache_size": "128m",
        "cassandra_data_centers": [
            "dc1:2",
            "dc2:3"
        ],
        "nginx_admin_directives": {},
        "custom_plugins": {},
        "pg_host": "postgres",
        "nginx_acc_logs": "/usr/local/kong/logs/access.log",
        "proxy_listen": [
            "0.0.0.0:8000",
            "0.0.0.0:8443 ssl"
        ],
        "client_ssl_cert_default": "/usr/local/kong/ssl/kong-default.crt",
        "ssl_cert_key_default": "/usr/local/kong/ssl/kong-default.key",
        "dns_no_sync": false,
        "db_update_propagation": 0,
        "nginx_err_logs": "/usr/local/kong/logs/error.log",
        "cassandra_port": 9042,
        "dns_order": [
            "LAST",
            "SRV",
            "A",
            "CNAME"
        ],
        "dns_error_ttl": 1,
        "headers": [
            "server_tokens",
            "latency_tokens"
        ],
        "dns_stale_ttl": 4,
        "nginx_optimizations": true,
        "database": "postgres",
        "pg_database": "kong",
        "nginx_worker_processes": "auto",
        "lua_package_cpath": "",
        "admin_acc_logs": "/usr/local/kong/logs/admin_access.log",
        "lua_package_path": "./?.lua;./?/init.lua;",
        "nginx_pid": "/usr/local/kong/pids/nginx.pid",
        "upstream_keepalive": 60,
        "cassandra_contact_points": [
            "127.0.0.1"
        ],
        "admin_access_log": "/dev/stdout",
        "client_ssl_cert_csr_default": "/usr/local/kong/ssl/kong-default.csr",
        "proxy_listeners": [
            {
                "ssl": false,
                "ip": "0.0.0.0",
                "proxy_protocol": false,
                "port": 8000,
                "http2": false,
                "listener": "0.0.0.0:8000"
            },
            {
                "ssl": true,
                "ip": "0.0.0.0",
                "proxy_protocol": false,
                "port": 8443,
                "http2": false,
                "listener": "0.0.0.0:8443 ssl"
            }
        ],
        "proxy_ssl_enabled": true,
        "pg_password": "******",
        "cassandra_ssl": false,
        "enabled_headers": {
            "latency_tokens": true,
            "X-Kong-Proxy-Latency": true,
            "Via": true,
            "server_tokens": true,
            "Server": true,
            "X-Kong-Upstream-Latency": true,
            "X-Kong-Upstream-Status": false
        },
        "ssl_cert_csr_default": "/usr/local/kong/ssl/kong-default.csr",
        "client_ssl": false,
        "db_resurrect_ttl": 30,
        "error_default_type": "text/plain",
        "cassandra_consistency": "ONE",
        "client_max_body_size": "0",
        "admin_error_log": "/dev/stderr",
        "pg_ssl_verify": false,
        "dns_not_found_ttl": 30,
        "pg_ssl": false,
        "db_update_frequency": 5,
        "ssl_ciphers": "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256",
        "cassandra_repl_strategy": "SimpleStrategy",
        "cassandra_repl_factor": 1,
        "log_level": "notice",
        "admin_ssl_cert": "/usr/local/kong/ssl/admin-kong-default.crt",
        "real_ip_header": "X-Real-IP",
        "kong_env": "/usr/local/kong/.kong_env",
        "cassandra_schema_consensus_timeout": 10000,
        "dns_hostsfile": "/etc/hosts",
        "admin_listeners": [
            {
                "ssl": false,
                "ip": "0.0.0.0",
                "proxy_protocol": false,
                "port": 8001,
                "http2": false,
                "listener": "0.0.0.0:8001"
            },
            {
                "ssl": true,
                "ip": "0.0.0.0",
                "proxy_protocol": false,
                "port": 8444,
                "http2": false,
                "listener": "0.0.0.0:8444 ssl"
            }
        ],
        "cassandra_timeout": 5000,
        "ssl_cert": "/usr/local/kong/ssl/kong-default.crt",
        "proxy_access_log": "/dev/stdout",
        "admin_ssl_cert_key_default": "/usr/local/kong/ssl/admin-kong-default.key",
        "cassandra_ssl_verify": false,
        "ssl_cipher_suite": "modern",
        "cassandra_lb_policy": "RoundRobin",
        "real_ip_recursive": "off",
        "proxy_error_log": "/dev/stderr",
        "client_ssl_cert_key_default": "/usr/local/kong/ssl/kong-default.key",
        "nginx_daemon": "off",
        "anonymous_reports": true,
        "db_cache_ttl": 0,
        "nginx_proxy_directives": {},
        "pg_port": 5432,
        "nginx_kong_conf": "/usr/local/kong/nginx-kong.conf",
        "client_body_buffer_size": "8k",
        "lua_socket_pool_size": 30,
        "admin_ssl_cert_default": "/usr/local/kong/ssl/admin-kong-default.crt",
        "nginx_http_directives": [
            {
                "value": "prometheus_metrics 5m",
                "name": "lua_shared_dict"
            }
        ],
        "cassandra_keyspace": "kong",
        "ssl_cert_default": "/usr/local/kong/ssl/kong-default.crt",
        "nginx_conf": "/usr/local/kong/nginx.conf",
        "admin_listen": [
            "0.0.0.0:8001",
            "0.0.0.0:8444 ssl"
        ]
    },
    "version": "0.14.0",
    "node_id": "4d52dcd3-420c-4482-840f-110d9d3eb274",
    "lua_version": "LuaJIT 2.1.0-beta3",
    "prng_seeds": {
        "pid: 57": 921281731435,
        "pid: 58": 236194211245,
        "pid: 60": 188183197199,
        "pid: 59": 159121015489
    },
    "timers": {
        "pending": 5,
        "running": 0
    },
    "hostname": "kong-rc-6c4dbfbd9c-l8n4n"
}

ionosphere80 · 2018-08-22T18:05:40Z

Did your certificate contain \n in place of CRs?

itaimalek · 2018-08-22T18:21:50Z

Did your certificate contain \n in place of CRs?
.
nope -verified that.

you can see Kong is serving a default certificate:

ionosphere80 · 2018-08-22T18:24:50Z

If using JSON, the certificate and key objects should use \n in place of CRs. The Kong API has a tendency to accept certificates/keys with an incorrect format, but won't enable them.

itaimalek · 2018-08-22T18:36:00Z

NICE !
That did it !

Thanks allot !

ionosphere80 · 2018-08-22T18:37:37Z

No problem. Kong should do a better job of validating body content.

benjaminprojas · 2018-09-19T14:06:00Z

I am having the same symptoms as the original issue here. My certificates are being loaded properly (I think), and yet kong isn't picking them up. When it returns the created or updated response, I see the line breaks are \n like @ionosphere80 mentioned.

I used to add these certificates in 0.13.x like this:

curl -i -X POST http://localhost:8001/certificates/ \
  -F 'cert=@/link/to/cert.pem' \
  -F 'key=@/link/to/key.pem' \
  -F 'snis=domain.com'

But if I do that now, it returns a "missing key" error. If I use JSON instead, it seems to load just fine, but it doesn't use it properly on the endpoint. Anything I am missing here?

benjaminprojas · 2018-09-19T14:09:49Z

I forgot to mention the way I am doing it now:

curl -i -X POST http://localhost:8001/certificates/ \
  -H 'Content-Type: application/json' \
  -d "{\"cert\": \"$(cat /link/to/cert.pem")\", \"key\": \"$(cat /link/to/key.pem")\", \"snis\": [ \"domain.com\" ] }"

benjaminprojas · 2018-09-19T16:10:15Z

Never mind, turns out I was generating the certs improperly. Thanks!

DevOpsRoot · 2018-10-08T08:11:45Z

Thanks @benjaminprojas, you save my day 💯

bungle · 2018-10-17T09:16:18Z

I feel like this is resolved now. Btw. the multipart/form-data support was added back with this (well 80% solution, but should work especially for this case):
#3776

xiaoshumiao6 · 2019-04-10T11:24:16Z

that is a so big trap

cmutrufflepig · 2021-03-18T06:52:45Z

I forgot to mention the way I am doing it now:

curl -i -X POST http://localhost:8001/certificates/ \
  -H 'Content-Type: application/json' \
  -d "{\"cert\": \"$(cat /link/to/cert.pem")\", \"key\": \"$(cat /link/to/key.pem")\", \"snis\": [ \"domain.com\" ] }"

Thank you! This is the only way that worked after an hour of trying.
One small thing though, the double quotes immediately after /link/to/xxx.pem are redundant. Saves other people a few minute.

itaimalek changed the title ~~Can't get Kong 0.14.0 to serve my certs~~ Can't get Kong 0.14.0 to serve my certificates on HTTPS Aug 22, 2018

bungle closed this as completed Oct 17, 2018

thonatos mentioned this issue Nov 20, 2018

Certification SSL not running when saved in software PGBI/kong-dashboard#189

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't get Kong 0.14.0 to serve my certificates on HTTPS #3721

Can't get Kong 0.14.0 to serve my certificates on HTTPS #3721

itaimalek commented Aug 22, 2018 •

edited

ionosphere80 commented Aug 22, 2018

itaimalek commented Aug 22, 2018 via email •

edited

ionosphere80 commented Aug 22, 2018

itaimalek commented Aug 22, 2018 •

edited

itaimalek commented Aug 22, 2018

ionosphere80 commented Aug 22, 2018

itaimalek commented Aug 22, 2018 •

edited

ionosphere80 commented Aug 22, 2018

itaimalek commented Aug 22, 2018

ionosphere80 commented Aug 22, 2018

benjaminprojas commented Sep 19, 2018

benjaminprojas commented Sep 19, 2018

benjaminprojas commented Sep 19, 2018

DevOpsRoot commented Oct 8, 2018

bungle commented Oct 17, 2018

xiaoshumiao6 commented Apr 10, 2019

cmutrufflepig commented Mar 18, 2021

Can't get Kong 0.14.0 to serve my certificates on HTTPS #3721

Can't get Kong 0.14.0 to serve my certificates on HTTPS #3721

Comments

itaimalek commented Aug 22, 2018 • edited

Summary

Steps To Reproduce

Expected result

Additional Details & Logs

ionosphere80 commented Aug 22, 2018

itaimalek commented Aug 22, 2018 via email • edited

ionosphere80 commented Aug 22, 2018

itaimalek commented Aug 22, 2018 • edited

itaimalek commented Aug 22, 2018

ionosphere80 commented Aug 22, 2018

itaimalek commented Aug 22, 2018 • edited

ionosphere80 commented Aug 22, 2018

itaimalek commented Aug 22, 2018

ionosphere80 commented Aug 22, 2018

benjaminprojas commented Sep 19, 2018

benjaminprojas commented Sep 19, 2018

benjaminprojas commented Sep 19, 2018

DevOpsRoot commented Oct 8, 2018

bungle commented Oct 17, 2018

xiaoshumiao6 commented Apr 10, 2019

cmutrufflepig commented Mar 18, 2021

itaimalek commented Aug 22, 2018 •

edited

itaimalek commented Aug 22, 2018 via email •

edited

itaimalek commented Aug 22, 2018 •

edited

itaimalek commented Aug 22, 2018 •

edited