X-Consul-Index behaves strangely on non-existing services with an existing service prefix

[pierresouchay]

consul version for both Client and Server

Client: 0.9.3
Server: 0.9.3

Description of the Issue (and unexpected/desired result)

We have several large clusters (~5k hosts per clusters) connected thru WAN and have some scalability issues. During one of my investigations, I noticed a strange bug:

1. When requesting a non-existing service with a X-Consul-Index, consul blocks as long as the timeout occurs or the service is being declared => OK
2. When requesting an existing service with a X-Consul-Index fetched on this service, consul blocks as the timeout occurs or the service gets updated => OK
3. But, when requesting a non-existing service with a X-Consul-Index, if another service with a shorter name exists, Consul does not block and all updates of X-Consul-Index are getting catched.

Reproduction steps

Create this script called consul_check_services_changes.sh

#!/bin/bash
if test ${1:-none} = "none"
then
  echo "USAGE: $0 Consul_URL"
  echo "       Example: localhost:8500/v1/health/service/MY_SUPER_SERVICE"
  exit 1
fi
url_to_check=$1

headers=$(mktemp)
content=$(mktemp)
index=0
while true;
do
  url="${url_to_check}?wait=5m&index=${index}&pretty=true&stale"
  curl -fs --dump-header "$headers" -o "${content}.new" "${url}" || { "echo Failed to query ${url}"; exit 1; }
  if test $index -ne 0
  then
    diff -u "$content" "$content.new" && echo " diff: No Differences found in service"
  fi
  index=$(grep "X-Consul-Index" "$headers" | sed 's/[^0-9]*\([0-9][0-9]*\)[^0-9]*/\1/g')
  mv "$content.new" "$content"
  printf "X-Consul-Index: $index at $(date) \b"
done

Call it on a non-existing service:

./consul_check_services_changes.sh consul-agent:8500/v1/catalog/service/non-existing-service
X-Consul-Index: 37968409 at Sat 25 Nov 2017 01:27:09 CET

Blocks -> OK

Call it on a real service:

consul_check_services_changes.sh consul-agent:8500/v1/catalog/service/consul
X-Consul-Index: 37968524 at Sat 25 Nov 2017 01:27:31 CET

Blocks as long as the service is not updated -> OK

Call it on a service that does not exists but has an existing service as prefix

consul_check_services_changes.sh consul-agent:8500/v1/catalog/service/consul2
X-Consul-Index: 37968837 at Sat 25 Nov 2017 01:28:06 CET diff: No Differences found in service
X-Consul-Index: 37968864 at Sat 25 Nov 2017 01:28:07 CET diff: No Differences found in service
X-Consul-Index: 37968865 at Sat 25 Nov 2017 01:28:07 CET diff: No Differences found in service
X-Consul-Index: 37968868 at Sat 25 Nov 2017 01:28:07 CET diff: No Differences found in service
X-Consul-Index: 37968869 at Sat 25 Nov 2017 01:28:07 CET diff: No Differences found in service
X-Consul-Index: 37968877 at Sat 25 Nov 2017 01:28:08 CET diff: No Differences found in service
X-Consul-Index: 37968878 at Sat 25 Nov 2017 01:28:08 CET diff: No Differences found in service
X-Consul-Index: 37968879 at Sat 25 Nov 2017 01:28:08 CET diff: No Differences found in service
X-Consul-Index: 37968880 at Sat 25 Nov 2017 01:28:09 CET diff: No Differences found in service
X-Consul-Index: 37968887 at Sat 25 Nov 2017 01:28:09 CET diff: No Differences found in service
X-Consul-Index: 37968888 at Sat 25 Nov 2017 01:28:09 CET diff: No Differences found in service
X-Consul-Index: 37968896 at Sat 25 Nov 2017 01:28:12 CET diff: No Differences found in service
X-Consul-Index: 37968898 at Sat 25 Nov 2017 01:28:12 CET diff: No Differences found in service
X-Consul-Index: 37968914 at Sat 25 Nov 2017 01:28:12 CET diff: No Differences found in service
X-Consul-Index: 37968915 at Sat 25 Nov 2017 01:28:12 CET diff: No Differences found in service
X-Consul-Index: 37968919 at Sat 25 Nov 2017 01:28:13 CET diff: No Differences found in service
X-Consul-Index: 37968920 at Sat 25 Nov 2017 01:28:13 CET diff: No Differences found in service
X-Consul-Index: 37968921 at Sat 25 Nov 2017 01:28:14 CET diff: No Differences found in service
[...]

Basically, we are notified on each change of X-Consul-Index for such services. Thus I suspect a significant but in the implementation. (It should behave the same way as either and existing service or a non-existing service right ?

[slackpad]

Hi @pierresouchay thanks for opening an issue - I was wondering when someone would notice this :-)

This is actually working as intended, though it is a little weird. Internally, Consul stores all of its data like the catalog using https://github.com/hashicorp/go-memdb which uses radix trees for prefix-based indexing. There's a watch mechanism whereby queries return a "watch set" with a set of channels to be notified if something changes (here's an example). When you query for a non-existent entry, you end up with a high-level node in the radix tree to watch on, up to possibly the root node if there's not much data in the index. Because blocking queries can always wake up without returning new data we decided not to optimize this further and add more complexity to the query code, though we did think about it. The results for a non-existent query are cheap to serve, and once the index gets populated with some real data you usually end up watching a more specific prefix node.

Your case is kind of pathological because you have some other service that's a prefix of the one you are watching, so indeed you'd get some churn in there. If you are really seeing a few QPS in practice it's probably not a huge deal, though it might be worth addressing if it's causing huge amounts of bogus requests.

[weiwei04]

Hi @slackpad, unlike @pierresouchay, we rely on this intended behavior.

We have about 600+ different services and the number is still growing, the services from same department have same prefix in service names, so our LoadBalancer use prefix to block query those services instead of 1 block query per services. Can we rely on this intended behavior on future consul release?

For example:

services:
com.a.a0, com.a.a1, com.a.a2
com.b.b0, com.b.b1, com.b.b2

we block query com.a to get a0, a1, a2 service update, block query com.b to get b0, b1, b2 service update.

[slackpad]

Hi @weiwei04 that's an interesting use case. Because of the internal implementation this should work because everything is managed with a prefix tree, and we don't have plans to change that, but it's not an explicit part of the interface contract. It might be better as an enhancement request to make a more explicit new API to watch for changes for service by prefix.

[pierresouchay]

Fixed in Prometheus and Consul