Service health check not working when deployed on Nomad-cluster

[oschistad]

Context: Using this module to deploy a nomad job against a Nomad-cluster of 3 server + 3 clients. Cluster is ACL-enabled and runs in an environment where Consul ACLs have also been enabled.

Consul Connect is enabled for the MInIO service.

Consul is reporting the Service as unhealth, and logs the following errors:

minio-live

ServiceName
    minio

CheckID
    _nomad-check-152ee3424af83fbcbb628873facbad73408e823d

Output

    Get http://<HOSTIP>:27237/minio/health/live: dial tcp <HOSTIP>:27237: connect: connection refused

[oschistad]

I suspect this issue is caused by connect fully isolating the container, including from the process performing the service checks.

If so then this thread may contain a solution:
https://discuss.hashicorp.com/t/consul-connect-with-health-checks/7591/2

[Neha-Sinha2305]

Also this module has starting behaving this way because of the following fix that was done to vagrant-hashistack box :Skatteetaten/vagrant-hashistack#344
Proxying fails when consul_default_policy=deny, works fine when consul_default_policy=allow.

[fredrikhgrelland]

@Neha-Sinha2305 So this means we need to fix this asap ;)

[Neha-Sinha2305]

@oschistad : Is it possible for you to set the consul_default_policy=allow in your cluster and redeploy the module and check that the service health check works fine?

[zhenik]

@oschistad Could you also provide consul and nomad versions?

[Neha-Sinha2305]

Context: Using this module to deploy a nomad job against a Nomad-cluster of 3 server + 3 clients. Cluster is ACL-enabled and runs in an environment where Consul ACLs have also been enabled.

Consul Connect is enabled for the MInIO service.

Consul is reporting the Service as unhealth, and logs the following errors:

minio-live

ServiceName
    minio

CheckID
    _nomad-check-152ee3424af83fbcbb628873facbad73408e823d

Output

    Get http://<HOSTIP>:27237/minio/health/live: dial tcp <HOSTIP>:27237: connect: connection refused

@oschistad: I looked into this issue and the minio module in the current state is capable of running without any errors in environments that have the following configurations:

Consul ACL	Consul default policy	Nomad ACL
enabled	deny	enabled
enabled	allow	enabled
disabled	deny	enabled
enabled	deny	disabled
enabled	allow	disabled
disabled	deny	disabled

Also, the checks that you mentioned are already in place in the minio nomad job and the terraform apply runs without any error so that means the healthchecks pass when run with the above configurations.
https://github.com/fredrikhgrelland/terraform-nomad-minio/blob/4c6087cbc3133a3b0f50622dcf61433e882bb62a/conf/nomad/minio.hcl#L15-L34

So, probably the issue lies in some other place and would need more details to debug it.

[fredrikhgrelland]

This is under investigation: Skatteetaten/vagrant-hashistack#368

[zhenik]

Is this still relevant @oschistad @fredrikhgrelland @pdmthorsrud @zhenik @dangernil @claesgill

[oschistad]

@zhenik Unknown - the issue referenced above relates to a flaw in the automated testing but not I believe an actual fix for the issue.