Consul Ingress-Gateway example does not work when ACL's are enabled

[spuder]

I have an environment using nomad 0.11.1 and consul 1.8.2. Consul connect works well and services are able to communicate with each other. TLS and ACL's are enabled.

consul --version
Consul v1.8.2
Revision ba7d9435e
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

envoy --version
envoy  version: 923c4111bb48405ac96ef050c4f59ebbad3d7761/1.14.4/clean-getenvoy-2aa564b-envoy/RELEASE/BoringSSL

The tutorial for ingress-gateways does not work. The services are unable to communicate. I've ensured that intentions are configured correctly, and that the services are listening on the correct ports.

Setup counting service

consul services register counting.hcl

counting.hcl

service {
 name = "counting"
 id = "counting-1"
 port = 9003

 connect {
   sidecar_service {}
 }

 check {
   id       = "counting-check"
   http     = "http://localhost:9003/health"
   method   = "GET"
   interval = "10s"
   timeout  = "1s"
 }
}

consul connect envoy -sidecar-for counting-1 -admin-bind localhost:19001 > dashboard-proxy.log

wget https://github.com/hashicorp/demo-consul-101/releases/download/0.0.3/counting-service_linux_amd64.zip
unzip counting-service_linux_amd64.zip
PORT=9003 ./counting-service

Setup Ingress gateway

consul config write ingress-gateway.hcl  

ingress-gateway.hcl

Kind = "ingress-gateway"
Name = "ingress-service"

Listeners = [
 {
   Port = 8080
   Protocol = "tcp"
   Services = [
     {
       Name = "counting"
     }
   ]
 }
]

consul connect envoy -gateway=ingress -register -service ingress-service -admin-bind localhost:19002 -address '{{ GetInterfaceIP "eth0" }}:8888'

Setup dashboard

wget https://github.com/hashicorp/demo-consul-101/releases/download/0.0.3/dashboard-service_linux_amd64.zip
unzip dashboard-service_linux_amd64.zip
# Note using 127.0.0.1 because running on the same host
PORT=9002 COUNTING_SERVICE_URL="http://127.0.0.1:8080" ./dashboard-service_linux_amd64

Setup Intentions

consul intention create -allow ingress-service counting

Troubleshooting

To troubleshoot this I've ensured

• Using bootstrap tokens for all service registrations
• Look at consul logs
• Look at envoy logs
• Able to curl services curl localhost:9002, curl localhost:8080

I've repeated these steps 5 separate times. Twice using the exact examples in the learning document, and 3 times using the count-dash example shown in the nomad documentation which I have working with consul connect. I've also tried various other nomad jobs in my infrastructure with the same results

• Other ideas

I've also tried changing the service type from tcp to http with no luck.

Either I'm really bad at following directions, or something is missing from the tutorial.