Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nomad Connect doesn't manage TLS Consul endpoints #6594

Open
vvanholl opened this issue Oct 30, 2019 · 2 comments
Open

Nomad Connect doesn't manage TLS Consul endpoints #6594

vvanholl opened this issue Oct 30, 2019 · 2 comments
Milestone

Comments

@vvanholl
Copy link

@vvanholl vvanholl commented Oct 30, 2019

Hi,

Some context :

I am using Nomad 0.10.0 and Consul 1.6.1. Both Nomad and Consul are working with TLS and ACLs enabled.

I try to make my Nomad jobs running with Connect but in the logs I always have these error messages:

2019-10-30T20:34:42.894Z [ERROR] client.alloc_runner.task_runner.task_hook.envoy_bootstrap: error creating bootstrap configuration for Connect proxy sidecar: alloc_id=4660d74d-c834-9219-e8ee-c0fbd6911732 task=connect-proxy-test error="exit status 1" stderr="==> Failed looking up sidecar proxy info for _nomad-task-4660d74d-c834-9219-e8ee-c0fbd6911732-group-test_group-test-1313: Unexpected response code: 400 (Client sent an HTTP request to an HTTPS server.
Then trying to understand more, I noticed Nomad runs this process without success
consul connect envoy -grpc-addr unix://alloc/tmp/consul_grpc.sock -http-addr endpoint.local.compuscene.net:8500 -bootstrap -sidecar-for _nomad-task-4660d74d-c834-9219-e8ee-c0fbd6911732-group-test_group-test-131

This doen't work too with exactly the same error message.

But if I put https:// before endpoint.local.compuscene.net:8500 this command works nice.

It seems Nomad doesn't take care about it's configuration, and in particular the ssl=true option :
"consul": { "address": "endpoint.local.compuscene.net:8500", "auto_advertise": true, "checks_use_advertise": true, "ssl": true, "token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" },
Moreover when I dig in the Nomad code, I see no reference to the Consul ssl option when creating Connect classes. Only the address is used.

I don't know if this is clear. If you have any question don't hesitate to ask me more if needed.

Vincent

@rkettelerij

This comment has been minimized.

Copy link
Contributor

@rkettelerij rkettelerij commented Oct 31, 2019

What I coincidence. I was about to create a ticket for this since I'm also running into the same issue. Like @vvanholl says: Nomad currently assumes the local Consul agent is available over plain HTTP. Our configuration has TLS enabled on the Consul clients and Consul servers and we don't expose a plain HTTP endpoint on the Consul agent.

The problem is Nomad start the Consul Envoy proxy without any HTTP flags: https://github.com/hashicorp/nomad/blob/master/client/allocrunner/taskrunner/envoybootstrap_hook.go#L89

Therefore the Consul proxy fails to connect to the local Consul agent: https://github.com/hashicorp/consul/blob/cc9a6f79934a6da58b7aec63c057681d82aded5a/command/connect/proxy/proxy.go#L221

What Nomad should do is grab the Consul client configuration (the consul stanza in the Nomad config) and pass this (the TLS settings) along when starting the Consul proxy binary. The latter already accepts these settings.

@tgross

This comment has been minimized.

Copy link
Member

@tgross tgross commented Oct 31, 2019

Thanks for reporting this @vvanholl and @rkettelerij !

As of right now Consul ACL support is one of the known limitations of our implementation but is in the works. For TLS, I do see that we have an open issue for testing that properly (#6502) but this looks like a bug in how we look up the Consul address.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.