Problems when re-writing service config-entry

[truehhart]

When filing a bug, please include the following headings if possible. Any example text in this template can be deleted.

Overview of the Issue

Consul can not re-define a config entry for a service once it's been written.

Reproduction Steps

I was using nomad to spin up the service, so:

1. Run a job which includes 1 task and includes a connect {} stanza with a consul sidecar.
2. Wait for the service to appear, write a config entry as follows:

Kind = "service-defaults"
Name = "redis"
Protocol = "http"

3. Realize that it's actually tcp and you should double check things before running your template jobs
4. Re-write the config entry

Kind = "service-defaults"
Name = "redis"
Protocol = "tcp"

5. >

$ consul config write /etc/consul.d/files/redis.hcl
Error writing config entry service-defaults/redis: Unexpected response code: 500 (service "redis" has protocol "tcp", which does not match defined listener protocol "http")

7. Try to delete the config entry via

$consul config delete -kind service-defaults -name redis
Error deleting config entry service-defaults/redis: Unexpected response code: 500 (service "redis" has protocol "tcp", which does not match defined listener protocol "http")

Consul info for both Client and Server

Client info

agent:
	check_monitors = 0
	check_ttls = 1
	checks = 16
	services = 12
build:
	prerelease =
	revision = 10bb6cb3
	version = 1.9.4
consul:
	acl = disabled
	known_servers = 3
	server = false
runtime:
	arch = amd64
	cpu_count = 2
	goroutines = 163
	max_procs = 2
	os = linux
	version = go1.15.8
serf_lan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 15
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 10943
	members = 15
	query_queue = 0
	query_time = 1

Server info

agent:
	check_monitors = 0
	check_ttls = 0
	checks = 3
	services = 4
build:
	prerelease =
	revision = 10bb6cb3
	version = 1.9.4
consul:
	acl = disabled
	bootstrap = false
	known_datacenters = 1
	leader = true
	leader_addr = 10.0.1.7:8300
	server = true
raft:
	applied_index = 1398004
	commit_index = 1398004
	fsm_pending = 0
	last_contact = 0
	last_log_index = 1398004
	last_log_term = 15
	last_snapshot_index = 1392997
	last_snapshot_term = 15
	latest_configuration = [{Suffrage:Voter ID:c63cfb18-2fc6-fccc-ee9e-9d41c8a23d08 Address:10.0.0.236:8300} {Suffrage:Voter ID:87bce418-8ac6-402f-935b-58ac84b6422c Address:10.0.1.7:8300} {Suffrage:Voter ID:d46f54e9-f2bc-6915-6e13-bc5c9edfa0fe Address:10.0.2.52:8300}]
	latest_configuration_index = 0
	num_peers = 2
	protocol_version = 3
	protocol_version_max = 3
	protocol_version_min = 0
	snapshot_version_max = 1
	snapshot_version_min = 0
	state = Leader
	term = 15
runtime:
	arch = amd64
	cpu_count = 2
	goroutines = 373
	max_procs = 2
	os = linux
	version = go1.15.8
serf_lan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 15
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 10943
	members = 15
	query_queue = 0
	query_time = 1
serf_wan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 1
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 31
	members = 3
	query_queue = 0
	query_time = 1

Operating system and Environment details

$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Log Fragments

May 27 14:57:22 ip-10-0-1-7 consul: 2021-05-27T14:57:22.938Z [ERROR] agent.http: Request error: method=DELETE url=/v1/config/service-defaults/redis from=127.0.0.1:45328 error="service "redis" has protocol "tcp", which does not match defined listener protocol "http""

[rboyer]

That specific error is emitted if there is still an ingress gateway defined that references your service and there's a protocol mismatch between the two config entries: https://github.com/hashicorp/consul/blob/v1.9.4/agent/consul/state/config_entry.go#L681

You likely need to remove references to the redis service from that ingress gateway config entry first.

[truehhart]

That actually worked like a charm!

Thank you so much @rboyer for your assistance on this, I wish I was able to google this stuff on my own.

Have a great day!