Allow multiple ES outputs as long as they are the same ES

[AndersonQ]

What is the problem this PR solves?

It fixes #1672. Currently fleet-server does not handle correctly the multiple ES outputs the fleet-ui might create.

How does this PR solve the problem?

It creates a non-breaking change on the agent documents to store ES API key information per output.

How to test this PR locally

• add a second ES output:
  • go to Fleet > Settings
  • outputs > add output
  • just copy the data from the default output and git it another name. Let's call it non-default
• create/edit a policy to use non-default output for integrations. Let the monitoring using the default:
  • navigate to Fleet > Agent policies > your_policy
  • go to the settings "tab"
  • select non-default as the output for Output for integrations. (You'll need a paid license or use Elastic Cloud)
• enroll a Elastic Agent on this policy with some integration

At first both outputs should have their on API keys. Use elastic-agent inspect to check that.

Now modify the policy in a way that one of the outputs has its permissions changed, but not the other. I usually add and remove the Endpoint Security. Then both outputs should have the same API key, again, use the inspect command to check.

Here you can see how I reproduced the issue and my investigation

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's checklist

• Modify the agent model to support multiple ES outputs: add fields, don't change the existing ones, keep backward compatible
• ensure new agent model is correctly saved and loaded (Marshal/UnMarshaled)
• add migration process to internal/pkg/dl/migration.go
  • add doc explaining the migration
• when preparing ES output:
  • set the default API key to empty -> force all API keys to be regenerated
  • set old fields to empty string/nil
  • use new fields to store output API key data
• update painless script to use new agent model
• ensure the model.Agent aren't used (except for tests checking they aren't populated anymore)
  • update handleAck to use the new model
  • update monitor to use the new model
• on upgrade bump the coordinator index
• ensure we invalidate the api keys from DefaultAPIKeyHistory (the migration should take care of it)
• add output name to api key metadata
• use only new agent model
  • adjust internal/pkg/coordinator/monitor_integration_test.go to use new Agent model
  • adjust internal/pkg/coordinator/monitor.go to use new Agent model
  • ensure there is no reference to the deprecated agent fields. As the migration takes care of migrating every agent document, then it should be safe to remove the deprecated fields from the agent Schema
• try/nice to have: on agent checkin, after fleet-server upgrade to 8.4, try to search api key by metadata.agentId and invalidate the "old" ones. So far I don't think it's possible.
  • I think the most viable here would be to invalidate all the API keys during the migration

Things to remember to test

• ensure all API keys (access + outputs) are invalidated on Unenroll
• ensure that given an agent which the default_api_key_history field ins't empty, these keys get retired on ack and unenroll
• ensure the existing agents documents get migrated (a.k.a the migration happens)
• ensure the migrations is idempotent. Start the new fleet-server a few times and ensure it does not try to update the already updated agent documents. it can also be checked by looking at the logs. At the en of each migration a log like the following is produced.
  log message: migration MIGRATION_NAME done
  full log:

{"_index":".ds-logs-elastic_agent.fleet_server-default-2022.08.11-000001","_id":"sft_kYIB00fSKmPF2k05","_version":1,"_score":0,"_source":{"fleet.migration.retries.search":0,"agent":{"name":"fleet-server-dev","id":"308052ab-bca4-4eaa-8787-904feea56fa0","type":"filebeat","ephemeral_id":"b21f3b26-c6a3-4da6-9edf-56fc01813ebf","version":"8.5.0"},"log":{"file":{"path":"/opt/Elastic/Agent/data/elastic-agent-6d830e/logs/default/fleet-server-20220812-15.ndjson"},"offset":6005},"fleet.migration.version_conflicts":0,"elastic_agent":{"id":"308052ab-bca4-4eaa-8787-904feea56fa0","version":"8.5.0","snapshot":true},"fleet.migration.es.timed_out":false,"fleet.migration.total":1,"ecs":{"version":"8.0.0"},"fleet.migration.total.duration":27.307903,"fleet.migration.deleted":0,"fleet.migration.updated":1,"host":{"hostname":"fleet-server-dev","os":{"kernel":"5.13.0-52-generic","codename":"impish","name":"Ubuntu","type":"linux","family":"debian","version":"21.10 (Impish Indri)","platform":"ubuntu"},"containerized":false,"ip":["10.0.2.15","fe80::6c:bfff:fe6f:c40","192.168.56.43","fe80::a00:27ff:fed8:3139"],"name":"fleet-server-dev","id":"c287e219ca4f4643aa206fc914242e26","mac":["02:6c:bf:6f:0c:40","08:00:27:d8:31:39"],"architecture":"x86_64"},"log.level":"info","event":{"agent_id_status":"verified","ingested":"2022-08-12T10:00:51Z","dataset":"elastic_agent.fleet_server"},"fleet.migration.es.took":26,"fleet.migration.retries.bulk":0,"fleet.migration.name":"AgentOutputs","service.name":"fleet-server","fleet.migration.noops":0,"message":"migration AgentOutputs done","input":{"type":"filestream"},"@timestamp":"2022-08-12T10:00:43.961Z","data_stream":{"namespace":"default","type":"logs","dataset":"elastic_agent.fleet_server"},"fleet.migration.batches":1},"fields":{"fleet.migration.retries.search":[0],"elastic_agent.version":["8.5.0"],"host.os.name.text":["Ubuntu"],"host.hostname":["fleet-server-dev"],"host.mac":["02:6c:bf:6f:0c:40","08:00:27:d8:31:39"],"fleet.migration.total":[1],"host.os.version":["21.10 (Impish Indri)"],"fleet.migration.total.duration":[27.307903],"host.os.name":["Ubuntu"],"log.level":["info"],"agent.name":["fleet-server-dev"],"host.name":["fleet-server-dev"],"event.agent_id_status":["verified"],"fleet.migration.retries.bulk":[0],"fleet.migration.name":["AgentOutputs"],"host.os.type":["linux"],"fleet.migration.noops":[0],"input.type":["filestream"],"log.offset":[6005],"data_stream.type":["logs"],"host.architecture":["x86_64"],"agent.id":["308052ab-bca4-4eaa-8787-904feea56fa0"],"ecs.version":["8.0.0"],"host.containerized":[false],"agent.version":["8.5.0"],"host.os.family":["debian"],"fleet.migration.version_conflicts":[0],"fleet.migration.es.timed_out":[false],"host.ip":["10.0.2.15","fe80::6c:bfff:fe6f:c40","192.168.56.43","fe80::a00:27ff:fed8:3139"],"agent.type":["filebeat"],"host.os.kernel":["5.13.0-52-generic"],"fleet.migration.deleted":[0],"fleet.migration.updated":[1],"elastic_agent.snapshot":[true],"host.id":["c287e219ca4f4643aa206fc914242e26"],"fleet.migration.es.took":[26],"service.name":["fleet-server"],"elastic_agent.id":["308052ab-bca4-4eaa-8787-904feea56fa0"],"data_stream.namespace":["default"],"host.os.codename":["impish"],"message":["migration AgentOutputs done"],"event.ingested":["2022-08-12T10:00:51.000Z"],"@timestamp":["2022-08-12T10:00:43.961Z"],"host.os.platform":["ubuntu"],"data_stream.dataset":["elastic_agent.fleet_server"],"log.file.path":["/opt/Elastic/Agent/data/elastic-agent-6d830e/logs/default/fleet-server-20220812-15.ndjson"],"fleet.migration.batches":[1],"agent.ephemeral_id":["b21f3b26-c6a3-4da6-9edf-56fc01813ebf"],"event.dataset":["elastic_agent.fleet_server"]}}

Related issues

• Fixes Fleet-server does not correctly update API keys on policy output permissions change #1672

Acceptance criteria

Considering #1779 is merged, which changes the API key life cycle

for all scenarios

• On start up, 8.5 migrations runs successfully.
  • there should be no error log related to migrations, they'd contain either of the following.
    If any of them appear, please provide us the full log, with all its fields.
    • failed to run migrations for BC3+
    • failed to run subsystems for BC1 and BC2
  • a info log like migration [MIGRATION_NAME] done shows the migration is done. Where [MIGRATION_NAME] is one of AgentMetadata, AgentOutputs or PolicyCoordinatorIdx.
    • a error log like failed applying [MIGRATION_NAME] migration might appear after the migration done log.
      • please report the full log, with all its fields if that happens
• Agents enroll successfully, and their integrations can send data to ES. That means the API keys have tne necessary permissions
• Start with a basic/default policy, add then remove Endpoint Security and check:
  • the integrations can always send data to ES (before Endpoint Security, with Endpoint Security, after removing Endpoint Security)
  • the API key for each output remain the same
  • fleet-server does not invalidate them.
    • to check if fleet invalidate them, look for the log Invalidate old API keys. There will a fleet.policy.apiKeyIDsToRetire field with the list of API Keys invalidated

New cluster, no document on .fleet-agents index

• nothing special to look for.

Upgrade scenario

• .fleet-agents index with active agents
• the following fields are populated on the agent documents:
  • default_api_key
  • default_api_key_id
  • policy_output_permissions_hash
• upgrade fleet-server to 8.5
• confirm the agents are health and their integrations are sending data
• check that:
  • a new outputs field was created
  • the following fields are empty, null or not present
    • default_api_key
    • default_api_key_id
    • default_api_key_history
    • policy_output_permissions_hash
• restart fleet-server and check the AC for all scenarios again.

[aleksmaus]

small typo in the comments, otherwise LGTM

[aleksmaus]

typos: "mre noise tan" -> "more noise than"

[michalpristas]

looks ok, let's make sure qa will test this once again with upgrade flow.
i'm not looking forward to conflicts in my PR :D

[AndersonQ]

@amolnater-qasource it's finally merged and ready for your tests. Please let me know if the necessary tests scenarios described in the PR are clear of if you need more details. Thank you!

[amolnater-qasource]

Hi @AndersonQ
Thank you for the update.
We will revalidate this issue as soon as #1836 is fixed on latest 8.5 SNAPSHOT.

Thanks

[amolnater-qasource]

Hi @AndersonQ

We have revalidated this PR on latest 8.5 BC1 Kibana cloud environment and had below observations:

Details are shared below:

Steps:

1. Created same Elasticsearch output with name: non-default.
2. Created policy with Integrations output as non-default.
3. Installed an agent with this policy and run elastic-agent inspect command.
4. Added Endpoint Security and again run elastic-agent inspect command.
5. Removed Endpoint Security and again run elastic-agent inspect command.

Observations:

Scenario 01: On installing agent on fresh 8.5 BC1.

• On initial installation we observed different api_key for both outputs.
• On adding Endpoint security and removing Endpoint security we observed no change in api_key.

Screenshots:

Scenario 02: On uninstalling and reinstalling agent.

• On initial installation we observed different api_key for both outputs.
• On adding Endpoint security and removing Endpoint security we observed no change in api_key.

Screenshots:

Build details:
BUILD: 56595
COMMIT: 0d8de4df69f8084a94cdd9638d7de510813cb5ce

Could you please confirm if these are the required fixes?
Please let us know if we are missing anything here.

Thanks

[AndersonQ]

The PR to update API the API keys permission is also on the 8.5, that means, it changes how it needs to be validated. Besides there was a problem with 8.5 BC1. That said, I'll need to update what needs to be checked on the tests and due to the recent changes I made, it's better to validate again. But I'll update the ACs and then ping you folks.

[amolnater-qasource]

Hi @AndersonQ

Thank you for the update.
Please let us know once it is available for revalidation.

[AndersonQ]

@amolnater-qasource, here they are. I updated the PR description with them as well

Acceptance criteria

for all scenarios

• On start up, 8.5 migrations runs successfully.
  • there should be no error log related to migrations, they'd contain either of the following.
    If any of them appear, please provide us the full log, with all its fields.
    • failed to run migrations for BC3+
    • failed to run subsystems for BC1 and BC2
  • a info log like migration [MIGRATION_NAME] done shows the migration is done. Where [MIGRATION_NAME] is one of AgentMetadata, AgentOutputs or PolicyCoordinatorIdx.
    • a error log like failed applying [MIGRATION_NAME] migration might appear after the migration done log.
      • please report the full log, with all its fields if that happens
• Agents enroll successfully, and their integrations can send data to ES. That means the API keys have tne necessary permissions
• Start with a basic/default policy, add then remove Endpoint Security and check:
  • the integrations can always send data to ES (before Endpoint Security, with Endpoint Security, after removing Endpoint Security)
  • the API key for each output remain the same
  • fleet-server does not invalidate them.
    • to check if fleet invalidate them, look for the log Invalidate old API keys. There will a fleet.policy.apiKeyIDsToRetire field with the list of API Keys invalidated

New cluster, no document on .fleet-agents index

• nothing special to look for.

Upgrade scenario

• .fleet-agents index with active agents
• the following fields are populated on the agent documents:
  • default_api_key
  • default_api_key_id
  • policy_output_permissions_hash
• upgrade fleet-server to 8.5
• confirm the agents are health and their integrations are sending data
• check that:
  • a new outputs field was created
  • the following fields are empty, null or not present
    • default_api_key
    • default_api_key_id
    • default_api_key_history
    • policy_output_permissions_hash
• restart fleet-server and check the AC for all scenarios again.

[amolnater-qasource]

Hi @AndersonQ

Thank you for sharing the detailed testing guidelines.
We will revalidate this once 8.5 BC3 build will be available.

Thanks!

[amolnater-qasource]

Hi @AndersonQ
We have revalidated this issue on latest 8.5 BC6 Kibana cloud-production environment and had below observations:

Non-Upgrade scenario:

• Two different api_keys for different outputs
• On any operation adding/removing endpoint defend, we didn't observe any change in api_keys

Screenshot:

Upgrade scenario:

• We checked data for fields available under .feet-agents:
  default_api_key
default_api_key_id
default_api_key_history
policy_output_permissions_hash

• The same data is available on 8.5.0 BC6 post upgrade under .feet-agents:
  default_api_key
default_api_key_id
default_api_key_history
policy_output_permissions_hash

Screenshots:
8.4.3:

8.5.0 BC6:

Post Kibana and Agent Upgrade:

Could you please confirm if this is the expected fix.
Please let us know if we are missing any scenario to be tested.

Thanks