Fix issue with changing the agent token causing failure to renew the auto-encrypt certificate

[mkeeler]

The fallback method would still work but it would get into a state where it would let the certificate expire for 10s before getting a new one. And the new one used the less secure RPC endpoint.

This is also a pretty large refactoring of the auto encrypt code. I was going to write some tests around the certificate monitoring but it was going to be impossible to get a TestAgent configured in such a way that I could write a test that ran in less than an hour or two to exercise the functionality.

Moving the certificate monitoring into its own package will allow for dependency injection and in particular mocking the cache types to control how it hands back certificates and how long those certificates should live. This will allow for exercising the main loop more than would be possible with it coupled so tightly with the Agent.

[crhino]

Saw some interesting behavior while testing this out locally.

The scenario is as follows:

1. Update agent token to non-existent ACL token: curl -sv -H"X-Consul-Token: roottoken" localhost:30100/v1/agent/token/agent -XPUT -d '{"token": "adfasdf"}'
2. Create new token with correct privileges (I just created another token with global-management permissions to test)
3. Update agent token again to valid token: curl -sv -H"X-Consul-Token: roottoken" localhost:30100/v1/agent/token/agent -XPUT -d '{"token": "f725a93c-3b16-5ee3-ef20-7f31c2725d39"}'

After updating to a new, valid token, I still see log lines on the client stating:

    2020-07-20T17:09:12.962Z [ERROR] agent.client: RPC failed to server: method=ConnectCA.Sign server=172.24.0.5:8300 error="rpc error making call: ACL not found"

After putting in a debug log line into the ConnectCA.Sign endpoint, I can that somehow the client is still using the adfasdf token to attempt to retrieve a leaf cert. This is after we have already successfully retrieved a leaf certificate with the valid token:

    2020-07-20T17:04:29.687Z [DEBUG] agent.server: ConnectCA.Sign: what token am I using?: token=roottoken service_id=<nil> agent_id="&{bd5896ce-0f58-b0b3-128d-23e379782001.consul chris1 consul-dc1-client0}"
    2020-07-20T17:08:26.770Z [DEBUG] agent.server: ConnectCA.Sign: what token am I using?: token=adfasdf service_id=<nil> agent_id="&{bd5896ce-0f58-b0b3-128d-23e379782001.consul chris1 consul-dc1-client0}"
...
    2020-07-20T17:08:39.336Z [DEBUG] agent.server: ConnectCA.Sign: what token am I using?: token=f725a93c-3b16-5ee3-ef20-7f31c2725d39 service_id=<nil> agent_id="&{bd5896ce-0f58-b0b3-128d-23e379782001.consul chris1 consul-dc1-client0}"
    2020-07-20T17:08:39.385Z [DEBUG] agent.server: ConnectCA.Sign: what token am I using?: token=adfasdf service_id=<nil> agent_id="&{bd5896ce-0f58-b0b3-128d-23e379782001.consul chris1 consul-dc1-client0}"

It feels like there is still a goroutine running for the leaf certificate cache request? Looking through the code I was not able to figure why this was happening though, it seems like we are correctly cancelling contexts during the token updates.

[crhino]

For posterity, we figured out that the issue above is related to the cache package keeping around a background refresh of the certificate, which is currently expected behavior. So, not necessary to address in this PR.

[crhino]

LGTM! Works as expected and nicer to reason about.

[hanshasselberg]

LGTM, left a small comment about a comment.

[mkeeler]

It turned out we could just disable the background refresh for leaf certs as that functionality was unused for that type and it was causing issues.

[hashicorp-ci]

🍒❌ Cherry pick of commit 3c09482 onto release/1.8.x failed! Build Log

[mkeeler]

Backport PR is here: #8352