documentation: auto encrypt on an existing Consul datacenter not possible?

[42wim]

I've upgraded to v1.5.2 and wanted to migrate to auto encrypt. According to the documentation on https://learn.hashicorp.com/consul/security-networking/certificates this should be possible

If you are configuring RPC encryption on an existing Consul datacenter, verify_incoming and verify_outgoing must be set to false to allow existing communication to continue uninterrupted.

But, with auto encrypt it's not possible to set verify_incoming to false.

Consul bails with ==> if auto_encrypt.allow_tls is turned on, TLS must be configured in order to work properly.

consul/agent/config/builder.go

Lines 1113 to 1117 in 97ecc05

	if rt.AutoEncryptAllowTLS {
	if !rt.VerifyIncoming {
	return fmt.Errorf("if auto_encrypt.allow_tls is turned on, TLS must be configured in order to work properly.")
	}
	}

[rollerd]

This also seems to contradict the documentation in the Learn Consul guide:

There is a combination of options however that allows us to keep using verify_incoming for RPC, but not for HTTPS:
{ "verify_incoming": false, "verify_incoming_rpc": true }

[isaac-mj]

I'm having the same issues here. Is this being looked after?

[hanshasselberg]

Thanks for reporting @42wim, @rollerd and @isaac-mj. The documentation is not correct and will be fixed.

Setting up auto_encrypt in an existing datacenter should be possible. You need to enable it on every server first and then you can let clients use it. Servers needs to have TLS enabled, which is the code sample you provided.

Let me know if it works for you or if you need more guidance.

[TheManyula]

I'm having a similar issue. I'm currently setting up a new single-node cluster (for now) on 1.5.3 and I'm following the guide to securing RPC communication with TLS encryption. I'm trying to configure the UI for HTTPS which requires setting verify_incoming to false and verify_incoming_rpc to true. However, as soon as I do that and restart the server, I'm also getting if auto_encrypt.allow_tls is turned on, TLS must be configured in order to work properly.

[42wim]

@i0rek for an existing it's not possible to use auto_encrypt with verify_incoming disabled, as such you can't update your servers/clients without outage.

Also on a new setup you can't use the UI anymore like @thevanitas reports

[hanshasselberg]

I created an issue for this #6338. I think you should be able to serve the UI from a client in the meanwhile. Sorry for the troubles.

[42wim]

@i0rek i've upgraded to 1.6.1, it's still not possible to enable auto encrypt on an existing datacenter though.

I'm trying to configure first of the 3 servers.

I can now set verify_incoming to true and verify_incoming_rpc to false or verify_incoming_rpc to true and verify_incoming to false but the result is the same.

Oct 09 16:11:29 consul-3 consul[25692]: 2019/10/09 16:11:29 [WARN] consul.rpc: Non-TLS connection attempted with VerifyIncoming set from=[clientip]:44313
Oct 09 16:12:38 consul-3 consul[25819]: 2019/10/09 16:12:38 [ERR] consul.rpc: failed to read byte: tls: failed to verify client's certificate: x509: certificate has expired or is not yet valid from=[serverip]:34485

Full config

{
  "verify_incoming": true,
  "verify_incoming_rpc": false,
  "verify_outgoing": false,
  "verify_server_hostname": false,
  "auto_encrypt": {
    "allow_tls": true
  },
  "ca_file": "/usr/share/consul/consul-agent-ca.pem",
  "cert_file": "/usr/share/consul/server-consul.pem",
  "key_file": "/usr/share/consul/server-consul-key.pem"
}

[hanshasselberg]

@TheManyula your issues should be fixed, that true?
@42wim thanks for your update. Could you describe the steps you are going through in detail? I am wondering if you are starting with a non-TLS cluster and want to upgrade to auto_encrypt?

Thanks!

[42wim]

@i0rek Yes I'm trying to upgrade a non-TLS cluster to auto_encrypt and following the procedure on https://learn.hashicorp.com/consul/security-networking/certificates

I'm doing basically step 1 and then changing the verify_incoming and verify_outgoing to false like in the documentation

If you are configuring TLS encryption on an existing Consul deployment, verify_incoming and verify_outgoing must be set to false to allow existing communication to continue uninterrupted. This will require a rolling restart on all the servers. Once the clients are configured, you can then enable verify_incoming and verify_outgoing on the servers.

[hanshasselberg]

@42wim I think you need to

1. Setup servers with TLS and auto_encrypt with the caveat of disabling verify_incoming and verify_outgoing like you said.
2. Enable auto_encrypt on the clients
3. Enable verify_incoming and verify_outgoing on the servers

The logs you posted indicate that a client tries to communicate via TLS without having a certificate.

[42wim]

@i0rek well the main issue is that it's not possible to disable verify_incoming and verify_outgoing AND enable auto_encrypt on the servers. Consul doesn't want to start up.

Nov 18 15:27:22 icts-q-consul-3 consul[41542]: ==> if auto_encrypt.allow_tls is turned on, either verify_incoming or verify_incoming_rpc must be enabled

[hanshasselberg]

@42wim thanks for pointing that out! I am relaxing that requirement in #6811 to enable migrations.

And I would like to ask everyone to explain your use case of verify_incoming on consul clients in that PR.

[42wim]

@i0rek I'm running a release/1.6.x build with your PR in it, and now it works perfectly (as per documentation)
Thanks again!

[hanshasselberg]

Thank you for confirming @42wim!

[ghost]

Hey there,

This issue has been automatically locked because it is closed and there hasn't been any activity for at least 30 days.

If you are still experiencing problems, or still have questions, feel free to open a new one 👍.