Validate tokens and show results on OneAgent status

[lrgar]

Description

Since having mis-configured tokens with OneAgent objects is a relatively common scenario. The idea with this PR is for the Operator to validate different common situations and provide a place where users can look for issues, if any.

The OneAgent CRD now has two conditions, PaaSToken and APIToken, to indicate the state of the API and PaaS tokens, respectively, and can be looked at with kubectl get oneagent -o yaml or kubectl describe oneagent.

The Operator will now set different reasons for each condition depending of the validation result:

TokenReady

Set when all checks have passed.

- type: PaaSToken
  status: "True"
  reason: TokenReady
  message: Ready

TokenSecretNotFound

When the secret doesn't exist. By the default it comes from the OneAgent object name, and can be customized by the CR field tokens.

- type: PaaSToken
  status: "False"
  reason: TokenSecretNotFound
  message: Secret 'dynatrace:custom-oneagent-token' not found

TokenMissing

Set when the token field is missing on the secret. Two fields must be on the secret, apiToken, and paasToken.

- type: APIToken
  status: "False"
  reason: TokenMissing
  message: Token apiToken on secret dynatrace:custom-oneagent-token missing

TokenUnauthorized

When the token can't be found in the Dynatrace environment. Is it correct?

- type: APIToken
  status: "False"
  reason: TokenUnauthorized
  message: Token on secret dynatrace:custom-oneagent-token unauthorized

TokenScopeMissing

Set when the token doesn't have the required permission on Dynatrace environment, e.g., using API token on the PaaS token field, etc. Currently, API and PaaS tokens need DataExport, and InstallerDownload, respectively.

- type: APIToken
  status: "False"
  reason: TokenScopeMissing
  message: Token on secret dynatrace:custom-oneagent-token missing scope DataExport

TokenError

Set when the probe request failed by an unrecognized error. Is it the API URL correct?

- type: PaaSToken
  status: "False"
  reason: TokenError
  message: 'error when querying token on secret dynatrace:custom-oneagent-token:
    error making post request to dynatrace api: Post https://wrong.domain.com/api/v1/tokens/lookup:
    dial tcp: lookup wrong.domain.com on 1.2.3.4:53: no
    such host'

Implementation details

• To verify whether a token is valid, we make a request against the Dynatrace API for that token, which returns the assigned permissions (scopes).
• To try to avoid reconciliation loops trying to make DDoS over the API, we wait 5 minutes before a new request.
  • To keep track of the timestamps, two fields have been added to the CRD Status schema, LastAPITokenProbeTimestamp, and LastPaaSTokenProbeTimestamp.
• The Operator's workflow remains as before, so, if for example, the API token is invalid, that won't stop the Operator from creating the DaemonSet (version update checks will still fail of course.)
• Moved setStatusCondition() and findCondition() from the reconciler to be OneAgent's methods.
• Regarding the Dynatrace Client, I've made ServerError to be an error type. If specific behavior is needed with these kind of errors, the Go 1.13's errors package can be used.

[lrgar]

I'm aware of a few issues with my changes here,

• If only a token is missing from the secret, then the condition for the other is not updated, so it could say valid even if we actually don't know.
• Extra new lines on tokens will appear currently as TokenError, perhaps we want to handle this better since it's a problem seen from time to time.
• Finally, while the messages indicate what the problem is, it doesn't say what the customer needs to do to fix it. I've added the scenarios on the description of this PR but perhaps we want to add them to the documentation somewhere.

That being said, I'd suggest leaving these changes to a follow-up PR, since this one is growing quite a bit.