[Fleet] Modify Agent policy and agent upgrades to handle custom source_uri

[criamico]

Summary

Part of #133828

• Allow to set a specific download_source object per agent policy. download_source.host is then set as source_uri in the agent policy
• Modify the agent policy compilation logic to include agent.download.source_uri. If download_ source_id is set on the policy, the agent policy will have a specific source_uri, otherwise the default will be used
• Revert changes introduced in [Fleet] Disable custom source_uri option on upgrade APIs by default #123464; source_uri can now be passed in the Agent upgrade logic (both single agent and bulk upgrade).
  • If passed in the route, it will be written to the actions (this logic is already in main but it's prevented)
  • If not passed, the source_uri in the agent policy will be used

Tests

Verify that any agent policy has the default value for source_uri

• Navigate to agent policies, select any agent policy and click on "view policy"
  • Under agent.download.source_uri you should see the default source_uri (if you didn't set one, it should be https://artifacts.elastic.co)

Create an agent policy with a custom source_uri

• Create a new download_source object (something different than the default artifacts.elastic.co) and get the id:

  POST kbn:/api/fleet/agent_download_sources
  {
    "name": "custom source_uri for policy",
    "host": "http://custom-registry-per-policy",
    "is_default": "false"
  }
  

  {
    "item": {
      "id": "b5112840-fdf8-11ec-93d1-01bd67999927",
      "name": "custom source_uri for policy",
      "host": "http://custom-registry-per-policy",
      "is_default": false
    }
  }
  

• Create also a new agent policy and get the id (in my case was 1dcaf600-fdf8-11ec-93d1-01bd67999927)

• Set download_source_id to the id of the previously created download_source object:

  PUT kbn:/api/fleet/agent_policies/1dcaf600-fdf8-11ec-93d1-01bd67999927
  { 
    "name": "Agent policy with custom registry",
    "namespace": "default",
    "description": "",
    "monitoring_enabled": [
      "logs",
      "metrics"
    ],
    "download_source_id": "b5112840-fdf8-11ec-93d1-01bd67999927"
  }
  

• click on view policy on agent policies page and check that the agent.download.source_uri has http://custom-registry-per-policy

Specify source_uri when upgrading an agent

• The POST upgrade endpoint it should accept source_uri parameter:

• POST kbn:/api/fleet/agents/e73868ac-94cf-4e9d-8630-906a8a53caf3/upgrade
  {
    "source_uri": "https://my-registry-test",
    "version": "8.4.0",
    "force": true
  }
  

Specify source_uri when bulk upgrading agents:

• POST kbn:/api/fleet/agents/bulk_upgrade
  {
    "agents": [
      "e73868ac-94cf-4e9d-8630-906a8a53caf3", 
      "9bcd329e-4890-4353-b8dc-882000553bab"
      ],
    "source_uri": "https://my-registry-test",
    "version": "8.3.0"
  }
  

• For now there's no way to test this locally, but I added some integration tests to cover this part.

[criamico]

@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[criamico]

@elasticmachine merge upstream

[criamico]

@elasticmachine merge upstream

[nchaulet]

I do not think this one is used anywhere

[nchaulet]

Looks good to me 🚀

[kpollich]

Thanks for all your work here!

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: b61f935
• Storybooks Preview
• Webpack Bundle Analyzer

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
fleet	1389	1390	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
fleet	111.6KB	111.6KB	+17.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id	before	after	diff
ingest-agent-policies	17	18	+1

Unknown metric groups

API count

id	before	after	diff
fleet	1521	1522	+1

History

• 💚 Build #56954 succeeded a3822b4
• 💚 Build #56550 succeeded 740ea9e
• 💚 Build #56233 succeeded 8ce5c58
• 💔 Build #56223 failed 4e4d5ce
• 💚 Build #56001 succeeded a713220
• 💔 Build #55943 failed 73bd79f

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @criamico