[Fleet] Check for version conflict errors during package installation

[skh]

At the start of a package installation, EPM checks if a saved object already exists for a package, and updates it immediately to indicate that an installation is in process. If two processes are trying to install or update a package, this may cause a race condition on the saved object, which results in a 409 error and a failure of the API call.

UPDATE: When the two installations are started within about 500ms from each other, the version conflict errors do not happen at the beginning of _installPackage(), but somewhere in the install*() functions that are called by _installPackage() to install the actual assets. This makes this a bit trickier to solve.

This task is to handle this situation more gracefully.

A possible approach could be to check for this specific error condition with SavedObjectsClient.errors.isConflictError() in _install_package() and react accordingly:

• read the saved object again
• check again for install_status and install_started_at
• return early or continue installation accordingly

The description of #84190 and the script in https://gist.github.com/skh/cc695952031c9e349874b898c7066e42 may be good starting points for this work.

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[elasticmachine]

Pinging @elastic/ingest-management (Feature:EPM)

[ph]

@jfsiii and @neptunian would like your input for this issue? What is the current behavior?

[skh]

@ph The current behavior is described in the issue description.

[skh]

I've looked into this a bit deeper, and it seems the version conflicts happen a bit later in the installation process. When starting the installations 500ms apart, I see them during installKibanaAssets() but I guess that is a bit random and depends on the system. I've updated the initial description.

[neptunian]

Maybe we should ask the Kibana team if there is any experience in handling this scenario. Perhaps we can use a library like https://www.npmjs.com/package/async-lock to avoid more than one install at a time for a package key.