[Fleet] Use Elasticsearch API Keys instead of custom JWT tokens

[nchaulet]

Work in progress

Context

To authenticate the agent fleet uses tokens:

• An enrolment token to enrol a new agent
• An access token to authenticate the checkin with Fleet AND to output to ES

We have one enrollment token per policy, and when a agent enroll we create a new access token.

Change

During the first implementation of Fleet we use custom JWT token, we want to use ES API Key instead, (standard, more secure, …)

Also now an enrollment token could be assigned to a policy or not.

Vocabulary change

Enrollment token -> Enrollment Api Key
Access token -> Access Api Key

API change

Enrollment and checkin

For the agent instead of using a kbn-* headers use the standard Authorization: ApiKey {API_KEY} with the new enrollment token for enroll and access token for checkin

Enrollment api keys

• GET /api/fleet/enrollment_api_keys list enrollment api keys
• POST /api/fleet/enrollment_api_keys create an enrollment api key with an optionnal policy id

Kibana Internal change

For enrolment token:

• Generate an API Key (using system user)
• Do not put any permission on the API Key
• Store the encrypted API key in the Fleet tokens saved object collection
• We want to also use the policy update hood to add / delete AP keys for enrolling into a policy when a policy is added/removed

For access token:

• Generate an API key (using system user)
• The API key should have the permissions to append to logs-* metrics-* (we could do better in the future)
• Store API Key hash on the saved object agent
• Checkin will need to make an auth call to ES using this token, if it passes then query for the agent that has the hash of the supplied token.
• When delete agent delete the API Key

Open questions, need to solve later

• How we create API keys? (we are going to temporary use the elastic user as kibana user in the fleet CI branch until we figure this.)
  • A new "ingest" system user?
  • give more permission to the kibana user?

[mattapperson]

"When regenerate delete the old API key" -- I think this concept of having a single enrollment token is going away. That we will need to display a list of enrollment tokens somewhere. @clintongormley Perhaps you can clarify?

[ph]

Store API Key hash on the agent

Do we really need this? I don't see why we would need to keep the hash on our side?

I presume that you will want to query based on the hashed api key? It means that at some point you did the hash with the Kibana secrets. So If I do a call on your api with the token you can hash it again with the same secret.

[ph]

Just want to clarify, that API KEY can be used for both ES and Kibana authentification, correct?

[mattapperson]

Store API Key hash on the agent

Do we really need this? I don't see why we would need to keep the hash on our side?

I presume that you will want to query based on the hashed api key? It means that at some point you did the hash with the Kibana secrets. So If I do a call on your api with the token you can hash it again with the same secret.

Agreed, I don't think storing the hash is needed.

[mattapperson]

Just want to clarify, that API KEY can be used for both ES and Kibana authentification, correct?

Correct

[nchaulet]

Store API Key hash on the agent

We need a way to link an agent (saved object in kibana) and an ApiKey

Just want to clarify, that API KEY can be used for both ES and Kibana authentification, correct?

Yes

[mattapperson]

We need a way to link an agent (saved object in kibana) and an ApiKey

Right but fleet can do this by grabbing and re-hashing the API Key that is in the headers from the Agents request

[nchaulet]

So to resume just some misunderstanding here, we are going to save the apiKey hash in the saved object agent not the fleet agent

[mattapperson]

Perhaps an additional open question... This token is supposed to be used for the output to ES... in the future, we plan to support outputting to another cluster... should we worry now about how that would work with an API Key only existing on a single cluster?
The old API Key strategy tied this key to the output not to fleet so we could have N tokens as needed based on outputs

[ph]

Good question @mattapperson for multiple clusters, I don't think we need a resolution for now but.
Previously we were assuming that the key was part of the received configuration (in the output section) so that would have worked for N tokens.

Now, are we assuming that everything we received an elasticsearch output configuration we use the saved token api key?

[ph]

So to resume just some misunderstanding here, we are going to save the apiKey hash in the saved object agent, not the fleet agent

Yes.

Steps:

-I send you the plain text version of the key.

• You hash it with your secrets
• You use the hashed key for the query.

The secret is the weakest link here.

[mattapperson]

Now, are we assuming that everything we received an elasticsearch output configuration we use the saved token api key?

Yes

[clintongormley]

We have one enrollment token per policy, and when a agent enroll we create a new access token.

No. Enrolment tokens API keys are not per policy, but per space. An Agent enrolls into a Kibana Space. Once enrolled, any policy in that space can be applied to any agent in that space.

More than one enrolment API key can exist in the same space. The first one should be autogenerated. Subsequent enrollment API keys can be generated by the user via the UI.

For enrolment token:

• Generate an API Key (using system user) when a user call the enrollment token endpoint

Generate the first enrolment API key automatically so it is immediately available.

• Do not put any permission on the API Key

It should grant permission to the Agent to enroll in the current space.

• Store the encrypted API key in the Fleet tokens saved object collection

I don't know about this

• When regenerate delete the old API key

Multiple enrolment keys can co-exist - the user would actively choose to delete a key.

For access token:

• When delete agent delete the API Key and delete/regenerate the enrollment key used to enroll the agent

The enrolment key should be reusable across many agents until such time as it is deleted or expired.

"When regenerate delete the old API key" -- I think this concept of having a single enrollment token is going away. That we will need to display a list of enrollment tokens somewhere.

I agree that we should allow for multiple enrolment keys, but not one per agent. Start with one default autogenerated key and add/remove as needed for a particular deployment. Any agent with an enrolment key can enroll, which gives them an API token associated with the Agent's UUID. An enrolled agent never needs to use the enrollment key again.

[nchaulet]

@clintongormley

No. Enrolment tokens API keys are not per policy, but per space. An Agent enrolls into a Kibana Space. Once enrolled, any policy in that space can be applied to any agent in that space.

Can you clarify this? when we enroll an agent he is not assigned to a policy, and the user need to do an extra step to assign it? or you can have enrollment token with a policy (optional)?

[mattapperson]

@nchaulet I think, unless I am wrong, we want to make enrolling to a particular policy optional. Because in the future we want to be able to have optional auto-assignment rules in fleet vs designating one at enroll.

[clintongormley]

@nchaulet An agent can be assigned to 0-1 policies, which means that we can change which policy is assigned to an agent, or we can unassign an agent (which remains enrolled, and still keeps checking with kibana in case it is assigned a new policy).

Now when an agent is enrolled, we could do one of the following:

• leave the agent in an unassigned state
• specify the policy id on install
• default to the "Default" policy

I think I'm leaning towards the 3rd option which can include things which don't need configuration, eg system monitoring

[nchaulet]

I agree with most of that,

Have unassigned agents => you can assign the policy later from Kibana, (or automatic rules in a future versions).

I think we should not allow to specify the policy id on install seems to be sensitive: from one enrolment token you can get all policies
Instead we can assign a default policy to an enrollment api key.

[mattapperson]

@nchaulet @clintongormley FWIW one of our top asks for Beats CM was to enroll directly to tags. We don't have tags now but we do have policies.
So my thoughts are:

• Enrolling to a "default" policy does not fix this for users. Just prevents from being enrolled into nothing. A good step, but does not solve the assignment issue.
• Allowing an agent to enroll into any policy (within a space) is flexible, but perhaps too flexible... this does seem like a major (potential) security risk.
• Enrollment keys being assigned to only one policy is more secure, but adds complexity, and won't work well for auto discovery in a container environment.

[clintongormley]

OK, so turn this around. Instead of the agent specifying which policy it should belong to, and instead of an enrollment key belonging to a policy, have an enrollment key specify the policy to assign.

Perhaps that was the model you originally had but I understood it as "policy-has-enrollment-key" instead of "enrollment-key-specifies-policy"

[mattapperson]

@clintongormley yes this is how we had it before. The UX was token on policy because it felt easier to us. but the implementation is and was "enrollment-key-specifies-policy"