[Fleet] Maintain versions list for Agent upgrade modal

[criamico]

Follow up to #130259

The versions dropdown in the Agent upgrade modal is currently a static list. The main requirements are

• Filters out any prerelease versions, like 8.0.0-alpha1
• Filters out any version > current Kibana version
• Doesn't have any version < 7.17.0 since only 7.17.0+ is supported against Elastic 8.x. - Note that the UI already filters out any version < minimum installed elastic agent.

The web team provided a public endpoint that lists all the available versions: https://www.elastic.co/api/product_versions

We have the following possible options:

1. Creating a fleet owned endpoint that provides the list of available versions GET /api/fleet/agents/available_upgrade_versions This API should fetch data from the product_versions endpoint and fallback to an hardcorded list to still provide options to air-gapped environments or when the data are not available.
2. Writing a test that fetches from the product_versions endpoint and fails when a new version was release. This means maintaining the list manually, however the static list needs to be there anyway
3. Adding a github actions that automatically opens a PR when there are new versions. This would remove the need to fix a failing test and open a PR manually

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[criamico]

@kpollich I'm working on this issue and I basically have two options:

1. Writing a test that compares the hardcoded versions with the response from https://www.elastic.co/api/product_versions. This test should fail when a new minor/major version is released, remainding us to update the hardcoded list, however while writing it the endpoint became unavailable and the test might become flaky.
2. Adding some code that fetches the above url, filters those versions and displays them in the UI. This way the hardcoded versions would only be used when that endpoint is not available. This would be a better way to do it, but we still need to maintain the hardcoded list for air-gapped environments and all those cases where the endpoint is not available.

What do you think? Either way I think that we need some mechanism to keep the version list updated.

[kpollich]

It's pretty unfortunate that that API is somehow down while we need to figure this out 😅

Option number one doesn't sound ideal to me, because as soon as the product versions API goes down like it is now every Kibana PR will fail CI when our test runs. Depending on a network location like this during CI is generally going to introduce flakiness.

Option number two is a bit better because we can say "we'll check the product versions API if possible, otherwise use a hardcoded list of versions," but then updating the hardcoded list becomes something that's maybe easier to forget about.

[kpollich]

Looks like the API is maybe back now, but it's extremely slow.

8 seconds for a response here means we probably can't rely on this API to block the rendering of the version dropdown.

[nchaulet]

Did we consider an option 3: a script that populate the list during Kibana build (similar to bundled packages)? we do not introduce flakyness in the UI if this endpoint is down or private network .... and we have an up to date version list

[kpollich]

Build task to write the product versions to a gitignored file would probably work well here, I like that idea @nchaulet

[criamico]

Yes exactly, I think that we initially ruled out the idea of using this endpoint for production because of its slowness.

@nchaulet actually I wasn't aware of this option :) could you send an example of it? I already have the code that filters out the fetched list, I just need to move it to the correct place.

[nchaulet]

@criamico the kyle PR for bundled package for inspiration I think we could have something similar #122297

[criamico]

Thanks @nchaulet. I have a couple of questions about it:

• Are those scripts executed at every kibana restart? I'm trying to see when they are executed but cannot find them in the logs.
• Do you suggest to create an internal API to provide the content of the file? I don't know if reading directly the file from the UI is possible or even advised.

[nchaulet]

Are those scripts executed at every kibana restart? I'm trying to see when they are executed but cannot find them in the logs.

No they are executed when building Kibana, or manually for local dev

Do you suggest to create an internal API to provide the content of the file? I don't know if reading directly the file from the UI is possible or even advised.

Yes I think we probably need a new API to provide versions