Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

connect service upstream with prepared query not registering #224

Closed
cchatfield opened this issue Mar 5, 2020 · 1 comment · Fixed by #233
Closed

connect service upstream with prepared query not registering #224

cchatfield opened this issue Mar 5, 2020 · 1 comment · Fixed by #233
Assignees
@adilyse
Labels
area/connect Related to Connect service mesh, e.g. injection type/bug Something isn't working

Comments

@cchatfield
Copy link

Envoy: envoyproxy/envoy:v1.10.0
Consul: consul:1.6.2
Consul k8s: "hashicorp/consul-k8s:0.9.5"

I am using the following annotation to define upstreams:

"consul.hashicorp.com/connect-service-upstreams": "prepared_query:active_vault:8200, vault:8201, prepared_query:6687bd19-5654-76be-d764-5bcc67fe8fb5:8202"

The prepared query is registered with consul and returns this when executed:

{
    "Service": "vault",
    "Nodes": [
        {
            "Node": {
                "ID": "c6e224c5-669e-762a-b778-e764f34510d3",
                "Node": "ip-10-163-43-96.us-west-2.compute.internal",
                "Address": "10.163.43.22",
                "Datacenter": "dc1",
                "TaggedAddresses": {
                    "lan": "10.163.43.22",
                    "wan": "10.163.43.22"
                },
                "Meta": {
                    "consul-network-segment": "",
                    "pod-name": "hashicorp-consul-jwst9"
                },
                "CreateIndex": 3585749,
                "ModifyIndex": 3585759
            },
            "Service": {
                "ID": "vault:10.163.43.101:8200",
                "Service": "vault",
                "Tags": [
                    "active"
                ],
                "Address": "10.163.43.101",
                "Meta": null,
                "Port": 8200,
                "Weights": {
                    "Passing": 1,
                    "Warning": 1
                },
                "EnableTagOverride": false,
                "Proxy": {
                    "MeshGateway": {},
                    "Expose": {}
                },
                "Connect": {},
                "CreateIndex": 3731021,
                "ModifyIndex": 3731451
            },
            "Checks": [
                {
                    "Node": "ip-10-163-43-96.us-west-2.compute.internal",
                    "CheckID": "serfHealth",
                    "Name": "Serf Health Status",
                    "Status": "passing",
                    "Notes": "",
                    "Output": "Agent alive and reachable",
                    "ServiceID": "",
                    "ServiceName": "",
                    "ServiceTags": null,
                    "Type": "",
                    "Definition": {},
                    "CreateIndex": 3585749,
                    "ModifyIndex": 3585749
                },
                {
                    "Node": "ip-10-163-43-96.us-west-2.compute.internal",
                    "CheckID": "vault:10.163.43.101:8200:vault-sealed-check",
                    "Name": "Vault Sealed Status",
                    "Status": "passing",
                    "Notes": "Vault service is healthy when Vault is in an unsealed status and can become an active Vault server",
                    "Output": "Vault Unsealed",
                    "ServiceID": "vault:10.163.43.101:8200",
                    "ServiceName": "vault",
                    "ServiceTags": [
                        "active"
                    ],
                    "Type": "ttl",
                    "Definition": {},
                    "CreateIndex": 3731022,
                    "ModifyIndex": 3731452
                }
            ]
        }
    ],
    "DNS": {
        "TTL": ""
    },
    "Datacenter": "dc1",
    "Failovers": 0,
    "Index": 0,
    "LastContact": 0,
    "KnownLeader": true,
    "ConsistencyLevel": "leader"
}

The "vault" upstream is available on 8201, but neither prepared query is registered.

Envoy Sidecar:

[2020-03-05 01:03:43.697][000001][info][main] [source/server/server.cc:206] initializing epoch 0 (hot restart version=10.200.16384.323.options=capacity=16384, num_slots=8209 hash=228984379728933363 size=5931112)
[2020-03-05 01:03:43.697][000001][info][main] [source/server/server.cc:208] statically linked extensions:
[2020-03-05 01:03:43.697][000001][info][main] [source/server/server.cc:210]   access_loggers: envoy.file_access_log,envoy.http_grpc_access_log
[2020-03-05 01:03:43.698][000001][info][main] [source/server/server.cc:213]   filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.rbac,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash
[2020-03-05 01:03:43.698][000001][info][main] [source/server/server.cc:216]   filters.listener: envoy.listener.original_dst,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
[2020-03-05 01:03:43.698][000001][info][main] [source/server/server.cc:219]   filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.dubbo_proxy,envoy.filters.network.rbac,envoy.filters.network.sni_cluster,envoy.filters.network.thrift_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy
[2020-03-05 01:03:43.698][000001][info][main] [source/server/server.cc:221]   stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd
[2020-03-05 01:03:43.698][000001][info][main] [source/server/server.cc:223]   tracers: envoy.dynamic.ot,envoy.lightstep,envoy.tracers.datadog,envoy.zipkin
[2020-03-05 01:03:43.698][000001][info][main] [source/server/server.cc:226]   transport_sockets.downstream: envoy.transport_sockets.alts,envoy.transport_sockets.capture,raw_buffer,tls
[2020-03-05 01:03:43.698][000001][info][main] [source/server/server.cc:229]   transport_sockets.upstream: envoy.transport_sockets.alts,envoy.transport_sockets.capture,raw_buffer,tls
[2020-03-05 01:03:43.703][000001][info][main] [source/server/server.cc:271] admin address: 127.0.0.1:19000
[2020-03-05 01:03:43.704][000001][info][config] [source/server/configuration_impl.cc:50] loading 0 static secret(s)
[2020-03-05 01:03:43.704][000001][info][config] [source/server/configuration_impl.cc:56] loading 2 cluster(s)
[2020-03-05 01:03:43.711][000001][info][config] [source/server/configuration_impl.cc:67] loading 0 listener(s)
[2020-03-05 01:03:43.711][000001][info][config] [source/server/configuration_impl.cc:92] loading tracing configuration
[2020-03-05 01:03:43.711][000001][info][config] [source/server/configuration_impl.cc:101]   loading tracing driver: envoy.zipkin
[2020-03-05 01:03:43.711][000001][info][config] [source/server/configuration_impl.cc:112] loading stats sink configuration
[2020-03-05 01:03:43.711][000001][info][main] [source/server/server.cc:463] starting main dispatch loop
[2020-03-05 01:03:43.712][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:132] cm init: initializing cds
[2020-03-05 01:03:43.716][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:495] add/update cluster local_app during init
[2020-03-05 01:03:43.721][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:495] add/update cluster vault.default.dc1.internal.d16dc2ba-6d47-f242-bc30-77d364f5596e.consul during init
[2020-03-05 01:03:43.721][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:112] cm init: initializing secondary clusters
[2020-03-05 01:03:43.721][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:136] cm init: all clusters initialized
[2020-03-05 01:03:43.721][000001][info][main] [source/server/server.cc:435] all clusters initialized. initializing init manager
[2020-03-05 01:03:43.724][000001][info][upstream] [source/server/lds_api.cc:80] lds: add/update listener 'public_listener:10.163.42.200:20000'
[2020-03-05 01:03:43.725][000001][info][upstream] [source/server/lds_api.cc:80] lds: add/update listener 'vault:127.0.0.1:8201'
[2020-03-05 01:03:43.725][000001][info][config] [source/server/listener_manager_impl.cc:961] all dependencies initialized. starting workers
[2020-03-05 01:18:43.723][000001][info][main] [source/server/drain_manager_impl.cc:63] shutting down parent after drain
@adilyse adilyse added area/connect Related to Connect service mesh, e.g. injection type/bug Something isn't working labels Mar 12, 2020
@adilyse adilyse self-assigned this Mar 12, 2020
adilyse added a commit that referenced this issue Mar 12, 2020
@adilyse
Trim spaces before checking if an upstream is a prepared query
57add2c 
Fixes #224.

If a user puts spaces between their upstreams in the annotation,
the first section of a prepared query later in the list shows up
as ` prepared_query`. We were doing a direct comparison of this
value with `prepared_query`, which is incorrect. This change trims
the spaces on the first part of an upstream before checking its type.

Adds several different tests to verify the behavior with multiple
upstreams defined.
@adilyse adilyse mentioned this issue Mar 12, 2020
Merged
@adilyse
Copy link
Contributor

adilyse commented Mar 12, 2020

Hi @cchatfield,

Thanks for filing this! I've looked into it, and there's definitely a bug here. We're not properly stripping spaces off of the prepared_query tag while processing, so your second prepared query is not getting recognized and properly parsed.

For your case, this manifests as only two upstreams being registered:

  • The first prepared query
    • [2020-03-05 01:03:43.721][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:495] add/update cluster vault.default.dc1.internal.d16dc2ba-6d47-f242-bc30-77d364f5596e.consul during init
  • The vault service definition
    • [2020-03-05 01:03:43.725][000001][info][upstream] [source/server/lds_api.cc:80] lds: add/update listener 'vault:127.0.0.1:8201'

I've put together a fix in #233 , but you could fix it for now by taking out the spaces between the upstreams in your annotation:

"consul.hashicorp.com/connect-service-upstreams": "prepared_query:active_vault:8200,vault:8201,prepared_query:6687bd19-5654-76be-d764-5bcc67fe8fb5:8202"

adilyse added a commit that referenced this issue Apr 21, 2020
@adilyse
Trim spaces before checking if an upstream is a prepared query
11a84df 
Fixes #224.

If a user puts spaces between their upstreams in the annotation,
the first section of a prepared query later in the list shows up
as ` prepared_query`. We were doing a direct comparison of this
value with `prepared_query`, which is incorrect. This change trims
the spaces on the first part of an upstream before checking its type.

Adds several different tests to verify the behavior with multiple
upstreams defined.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@adilyse adilyse

Labels
area/connect Related to Connect service mesh, e.g. injection type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Trim spaces before checking if an upstream is a prepared query hashicorp/consul-k8s
2 participants
@cchatfield @adilyse