Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[gate] SSL w/ apiPort and ssl: enabled causes tomcat to start 2 https ports #2533

Open
bobbytables opened this issue Mar 11, 2018 · 17 comments

Comments

Projects
None yet
7 participants
@bobbytables
Copy link

commented Mar 11, 2018

An issue is not a place to ask questions. Please use Slack or Stack Overflow.

Before you open an issue, please check if a similar issue already exists or has been closed before.

Make sure you have read through the Spinnaker FAQ and Halyard FAQ to provide as much information as possible.

A descriptive title.

Starting gate with a gate-local.yml file with an apiPort and ssl: enabled setting causes gate to start with 2 https ports (instead of one for apiPort and the other on http)

Environment

Kubernetes on AWS

Feature Area

Gate SSL

Description

Based on the docs from Armory.io and Spinnaker's Authentication section, I expect my default.apiPort setting to start an SSL port, and leave 8084 as HTTP. (We terminate SSL at an AWS ELB, forwarding to 8084).

Steps to Reproduce

Given this gate-local.yml file:

default:
  apiPort: 8085

x509:
  enabled: true
server:
  port: '8084' # we don't actually use this, we use the legacy port to get a non-ssl port. This is just to enabled ssl on 8084 since spinnaker docs seem to lie about this
  address: '0.0.0.0'
  tomcat:
    protocolHeader: X-Forwarded-Proto
    remoteIpHeader: X-Forwarded-For
    internalProxies: .*
  ssl:
    enabled: true
    keyStore: /opt/spinnaker/config/keystore.jks
    keyStorePassword: "{reacted}"
    keyAlias: server
    trustStore: /opt/spinnaker/config/keystore.jks
    trustStorePassword: "{redacted}"
    clientAuth: want

It starts an SSL listener for both 8084 and 8085, as printed by Tomcat logs:

Tomcat started on port(s): 8084 (https) 8085 (https)

My workaround is to use the undocumented field legacyServerPort field and setting that to 8084:

default:
  apiPort: 8085
  legacyServerPort: 8084

x509:
  enabled: true
server:
  port: '9001' # we don't actually use this at this point but we get port collisions if its 8084

This solves the problem:

Tomcat started on port(s): 9001 (https) 8084 (http) 8085 (https)

This page is what I followed for the expected behavior: https://www.spinnaker.io/setup/security/authentication/x509/

You can move the client certificate-enabled port by setting default.apiPort value to something other > than 8084. This enables an additional port configuration that is hardcoded to need a valid X.509 > certificate before allowing the request to proceed.

The 3rd party, Armory.io, also expects this in a guide: https://docs.armory.io/install-guide/auth/#x509

@spinnakerbot

This comment has been minimized.

Copy link

commented Aug 15, 2018

This issue hasn't been updated in 155 days, so we are tagging it as 'stale'. If you want to remove this label, comment:

@spinnakerbot remove-label stale

@spinnakerbot spinnakerbot added the stale label Aug 15, 2018

@wheleph

This comment has been minimized.

Copy link

commented Sep 13, 2018

@spinnakerbot remove-label stale

@spinnakerbot spinnakerbot removed the stale label Sep 13, 2018

@wheleph

This comment has been minimized.

Copy link

commented Sep 13, 2018

@bobbytables I tried to apply your workaround and now Tomcat does start properly:

Tomcat started on port(s): 9001 (https) 8084 (http) 8085 (https)

However the pod never becomes healthy because the health check still uses https:

  Warning  Unhealthy              54s (x20 over 4m)  kubelet, gke-bolcom-sbx-89s-platform-89b85952-f1m9  Readiness probe failed: Get https://10.19.8.47:8084/health: http: server gave HTTP response to HTTPS client

I'm not able to override the probe because healthEndpoint mentioned in Custom Configuration only configures the last part if it:

healthEndpoint: 'abcde'
  Warning  Unhealthy              8s (x7 over 1m)  kubelet, gke-bolcom-sbx-89s-platform-89b85952-f1m9  Readiness probe failed: Get https://10.19.8.41:8084/abcde: dial tcp 10.19.8.41:8084: getsockopt: connection refused

Were you able to solve this problem?

@wheleph

This comment has been minimized.

Copy link

commented Sep 28, 2018

I solved the issue that I mentioned aboveIf by specifying

port: 9001

in gate.yml (as opposed to gate-local.yml).

In this case the healthcheck correctly probes https://:9001/health

@spinnakerbot

This comment has been minimized.

Copy link

commented Nov 12, 2018

This issue hasn't been updated in 45 days, so we are tagging it as 'stale'. If you want to remove this label, comment:

@spinnakerbot remove-label stale

@spinnakerbot spinnakerbot added the stale label Nov 12, 2018

@wheleph

This comment has been minimized.

Copy link

commented Nov 12, 2018

@spinnakerbot remove-label stale

@spinnakerbot spinnakerbot removed the stale label Nov 12, 2018

@spinnakerbot

This comment has been minimized.

Copy link

commented Dec 27, 2018

This issue hasn't been updated in 45 days, so we are tagging it as 'stale'. If you want to remove this label, comment:

@spinnakerbot remove-label stale

@spinnakerbot spinnakerbot added the stale label Dec 27, 2018

@jeckhart

This comment has been minimized.

Copy link

commented Jan 8, 2019

@spinnakerbot remove-label stale

@spinnakerbot spinnakerbot removed the stale label Jan 8, 2019

@spinnakerbot

This comment has been minimized.

Copy link

commented Feb 22, 2019

This issue hasn't been updated in 45 days, so we are tagging it as 'stale'. If you want to remove this label, comment:

@spinnakerbot remove-label stale

@spinnakerbot spinnakerbot added the stale label Feb 22, 2019

@jeckhart

This comment has been minimized.

Copy link

commented Feb 22, 2019

@spinnakerbot remove-label stale

@spinnakerbot spinnakerbot removed the stale label Feb 22, 2019

@roger-king

This comment has been minimized.

Copy link

commented Mar 21, 2019

Any update on this issue? I am experiencing this issue when enabling x509 for apiSecurity on gate.

@rodriguezsergio

This comment has been minimized.

Copy link

commented Mar 21, 2019

Note: Replace default with whatever the name of your current profile is. In most cases it is default, but just be aware to not blindly copy-pasta this.

For anyone else getting tripped up on this, what you want is

~/.hal/default/profiles/gate-local.yml

server:
  port: 9002

default:
  apiPort: 8085
  legacyServerPort: 8084

~/.hal/default/service-settings/gate.yml

scheme: http

Then Tomcat will start with
Tomcat started on port(s): 9002 (https) 8084 (http) 8085 (https)

Everything else should be configured with the appropriate hal config command. Filenames in gate-local.yml or similar will NOT be translated when you run hal deploy apply and you will get FileNotFoundException in your logs.

When you inform halyard of files via hal config it will drop those files into /home/spinnaker/.hal/default/staging/dependences (or whatever profile name you have in place of default) and then update the file paths in your main config YAML file to account for this dependency directory. *-local.yml files are literally just dropped into /opt/spinnaker/config as-is with no updates.

e.g.

# don't do this
# ~/.hal/default/profiles/gate-local.yml
server:
  ssl:
    enabled: true
    keyStore: /home/spinnaker/.hal/custom/path/keystore.jks 
    # ^^^^ this DOES NOT change between your local halyard install 
    # and your gate container after a 'hal deploy apply'

Hope that helps.

@roger-king

This comment has been minimized.

Copy link

commented Mar 22, 2019

Note: Replace default with whatever the name of your current profile is. In most cases it is default, but just be aware to not blindly copy-pasta this.

For anyone else getting tripped up on this, what you want is

~/.hal/default/profiles/gate-local.yml

server:
  port: 9002

default:
  apiPort: 8085
  legacyServerPort: 8084

~/.hal/default/service-settings/gate.yml

scheme: http

Then Tomcat will start with
Tomcat started on port(s): 9002 (https) 8084 (http) 8085 (https)

Everything else should be configured with the appropriate hal config command. Filenames in gate-local.yml or similar will NOT be translated when you run hal deploy apply and you will get FileNotFoundException in your logs.

When you inform halyard of files via hal config it will drop those files into /home/spinnaker/.hal/default/staging/dependences (or whatever profile name you have in place of default) and then update the file paths in your main config YAML file to account for this dependency directory. *-local.yml files are literally just dropped into /opt/spinnaker/config as-is with no updates.

e.g.

# don't do this
# ~/.hal/default/profiles/gate-local.yml
server:
  ssl:
    enabled: true
    keyStore: /home/spinnaker/.hal/custom/path/keystore.jks 
    # ^^^^ this DOES NOT change between your local halyard install 
    # and your gate container after a 'hal deploy apply'

Hope that helps.

I tried what you stated above and gate is still failing with port in use on 9002. It the same error just different port.

gate-local.yml

x509:
  enabled: true

server:
  port: 9002

default:
  apiPort: 8085
  legacyServerPort: 8084

service-settings/gate.yml
scheme: http

@rodriguezsergio

This comment has been minimized.

Copy link

commented Mar 22, 2019

You will need to be more specific as to what the error is. Is SSL enabled with the appropriate keystore and keystore password?

@roger-king

This comment has been minimized.

Copy link

commented Mar 22, 2019

You will need to be more specific as to what the error is. Is SSL enabled with the appropriate keystore and keystore password?

***************************
APPLICATION FAILED TO START
***************************

Description:

The Tomcat connector configured to listen on port 9002 failed to start. The port may already be in use or the connector may be misconfigured.

Action:

Verify the connector's configuration, identify and stop any process that's listening on port 9002, or configure this application to listen on another port.

yes, SSL enabled with correct keystore/keystore password

@rodriguezsergio

This comment has been minimized.

Copy link

commented Mar 26, 2019

Yeah, I'm not sure why it's not working for you there. Anything below or after those lines that indicate the root cause?

@spinnakerbot

This comment has been minimized.

Copy link

commented May 10, 2019

This issue hasn't been updated in 45 days, so we are tagging it as 'stale'. If you want to remove this label, comment:

@spinnakerbot remove-label stale

@spinnakerbot spinnakerbot added the stale label May 10, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.