Unable to connect to nats servers: nats: Authorization Violation

[Mrizwanshaik]

Issue

• nats VM is not running after update. Review logs for failed jobs: Metrics-discovery-registrar

Affected Versions

metrics-discovery-release: 3.0.7
routing-release:0.228.0

Context

• Three nats-tls servers
• On the same VM nats-tls job and metrics-discovery-registrar job are running
• We removed nats/user and password from the cf-deployment using ops file as it is made optional in routing-release:0.228.0
• During update cf-deployment:
  • The first nats-tls VM got error during update as metric-discovery-registrar failed due "nats: Authorization Violation"
  • The first nats-tls VM could not also connect to not-yet-updated nats as it does not provide password for authentication anymore
  • diego-api and uaa were also failing due to "nats: Authorization Violation" between route_registrar and not-yet-updated nats

Steps to Reproduce

• Bump routing-release to 0.228.0 and nats-release to 41.
• Remove nats/password and nats/user from cf-deployment using ops file.
• Set the mTLS certificates in nats/internal/tls/* for passwordless authentication
• Run deploy-cf

Expected result

• Deployment should run through and succeed.

Current result

• Failing deployment that tries to access username/password fields that are not available anymore

Possible Fix

• We have provided a PR in Nats-release that adds a configuration flag to disables password based authentication but allows providing the fields in the configuration: Introduce nats.auth_required for nats-tls nats-release#43. This works around the need of an existing deployment to have username/password, while a new deployment can move to mTLS based authentication.

Additional Context

[domdom82]

This is an interesting edge-case that is amplified by the fact that BOSH DNS seems to remove IPs of VMs that are not yet fully running:

While testing this, I logged into nats/f597cde7-5fec-43e5-acc2-88450fb7dc3d (10.0.65.3) which was the canary VM.

nats/f597cde7-5fec-43e5-acc2-88450fb7dc3d:/var/vcap/jobs/nats-tls/config# dig +short nats.service.cf.internal
10.0.65.5
10.0.65.4
nats/f597cde7-5fec-43e5-acc2-88450fb7dc3d:/var/vcap/jobs/nats-tls/config# exit
nats/f597cde7-5fec-43e5-acc2-88450fb7dc3d:~$ logout
Connection to 10.0.65.3 closed.

On that VM, the nats-tls job was already updated and running without password authentication. However, the
metrics-discovery-registrar uses the hostname NATS_HOSTS: nats://nats.service.cf.internal:4224. This name only resolves to the two other NATS VMs (10.0.65.5, 10.0.65.4) that have not yet been updated, therefore still require a password to connect.

So as @Mrizwanshaik and @b1tamara suggest, introducing a flag could help, because the flag is not part of the BOSH link that other deployments use to connect to NATS.

You could then update in two steps:

1. Set auth_required to false and deploy CF completely, this will only remove the password from NATS but not the clients.
2. Remove the password from cf-deployment.yml and deploy CF again so it also disappears from the BOSH link that is used by clients.

[domdom82]

Fixed in the latest NATS release v43 which is part of https://github.com/cloudfoundry/cf-deployment/releases/tag/v18.0.0