[Fleet] Setup fail with: Default download source host is not setup

[nchaulet]

Description

While migrating from Kibana 8.3.3 to 8.4.0 user get that error during Fleet setup

In 8.4.0 we introduced download source to configure where agent download artifacts during upgrade #135629

When we generate the full policy to populate the .fleet-policies index we retrieve that download source, looks like we generate that policy (maybe by updating outputs) before generating the default download source.

How to reproduce

From a fresh environement go to Fleet and wait for the setup to be done than

in your dev tools

DELETE .kibana/_doc/ingest-download-sources:fleet-default-download-source

and add this to your kibana config file

xpack.fleet.agents.elasticsearch.hosts: ['http://elasticsearch:9200']
xpack.fleet.outputs:
  - name: test123
    id: test123
    is_default: true
    hosts: ['test.fr']
    type: 'logstash'

Than you should see during setup
[2022-08-25T15:49:24.727-04:00][WARN ][plugins.fleet] Error: Default download source host is not setup

Workaround

You can manually create that download source by calling Fleet API

curl 'https://<KIBANA_URL>:5601/api/fleet/agent_download_sources' \
  -X 'POST' \
  -H 'Content-Type: application/json' \
  -H 'kbn-xsrf: xx' \
  -u "<USERNAME>:<PASSWORD>" \
  --data-raw '{"id": "fleet-default-download-source", "name":"Elastic Artifacts","host":"https://artifacts.elastic.co/downloads/","is_default":true}' \
  --compressed

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[L00Cyph3r]

I've had the same with upgrading from 8.3.2 to 8.4.1.
I did find out however that we didn't have any transform-nodes in our cluster at the time of the upgrade, and I have a feeling it hasn't run the migration. Is there a way to forcefully run the migration?
I have tried removing kibana 8.4.1 and installing 8.3.2, after which I did an upgrade to 8.4.1 to try to trigger the migration, but to no avail.

And in the worst case scenario, should I just delete the ".kibana" index and start from scratch with kibana, or is there some other advice?

[lucabelluccini]

I did find out however that we didn't have any transform-nodes in our cluster at the time of the upgrade, and I have a feeling it hasn't run the migration.
Transform role should not have any relationship with this.

I would suggest to avoid deleting the .kibana or any other "system" index as you will delete a lot of entities (such as Dashboards etc...).
Did you try to run the following request from Kibana dev tools?

POST kbn:/api/fleet/agent_download_sources
{"id": "fleet-default-download-source", "name":"Elastic Artifacts","host":"https://artifacts.elastic.co/downloads/","is_default":true}

If you want to get some help, I would suggest opening a thread in discuss.elastic.co or if you have a subscription or you're an Elastic Cloud user, open a ticket to Support.

[L00Cyph3r]

Eventually fixed it with your workaround. Normal auth didn't work for me, so had to create API key and such. Thanks for the workaround at least, and let's hope this gets fixed.

[ppikna97]

Thank you, your workaround was really helpful.

[juliaElastic]

@nchaulet I can't reproduce this anymore on main, do you know if there were any fixes?

[nchaulet]

@nchaulet I can't reproduce this anymore on main, do you know if there were any fixes?

@juliaElastic It may have been fixed when we added the new Fleet server host, I reorganized a little the setup

[juliaElastic]

Closing as can't reproduce, we can reopen if we encounter it again.