[Fleet] Don't attempt package installation while another installation might still be running

[skh]

Summary

Implements #75810

This changes _install_package() to return early when it detects that the package might be in the process of being installed by another process.

It does so by checking the install_status and install_started_at fields on the epm-packages saved object. If the previous installation was started longer than MAX_TIME_COMPLETE_INSTALL (currently, 60 seconds) ago, the installation will be attempted again.

MAX_TIME_COMPLETE_INSTALL is also used by ensurePackagesCompletedInstall() to decide whether to reattempt package installations. This should not break, or be broken by, the change made in this PR.

How to test this

I've created https://gist.github.com/skh/cc695952031c9e349874b898c7066e42 to test this locally. On my system, with a local ES instance

• when the two installation attempts are about 2-5 seconds apart, the behavior is as expected: the second installation attempt returns early and lists the assets that have been installed by the first attempt up until then. That means the second attempt lists less assets in its response than the first.
• when the two installation attempts are more than 10 seconds apart, the first one completes before the second is started, so two complete installations are performed. Both attempts list the same number of assets in their responses.
• when the two installation attempts are really close together, we still run into a race condition with the saved object itself. I can trigger this by sending the two install attempts within about 500 milliseconds of each other:

Response for second install attempt: {"statusCode":404,"error":"Not Found","message":"Saved object [epm-packages/apache] not found"}
Response for first install attempt: {"statusCode":409,"error":"Conflict","message":"[epm-packages:apache]: version conflict, required seqNo [1879], primary term [1]. current document has seqNo [1880] and primary term [1]: version_conflict_engine_exception"}
Starting package delete
Response for delete attempt: {"statusCode":400,"error":"Bad Request","message":"apache is not installed"}

Fixing this is beyond the scope of this PR.

UPDATE: after discussing with the Kibana Core team, the 409 error above happens when the saved object version has changed between a read and a subsequent update operation. We can use SavedObjectsClient.errors.isConflictError() to catch this specific case, and repeat the check for installed_status and install_started_at.

Follow-up issues: #84651 and #84656

[elasticmachine]

Pinging @elastic/ingest-management (Feature:Fleet)

[skh]

@neptunian and @jonathan-buttner I would be very interested to hear if you think this change is worth it at all.

[jonathan-buttner]

Hmm, @skh do you think returning a partial list of the assets is helpful to the requester? I wonder if it'd be better to return some status code that indicates that an installation request is already running? Or that the service is busy and the requester should try again later. That way it could try, then wait for a bit and try again and receive the complete list of assets.

[skh]

do you think returning a partial list of the assets is helpful to the requester?

Good point. I was thinking of how the UI currently uses the return value, and by returning just a list, albeit a shorter or empty one, the UI doesn't need additional changes for what should be a rare edge case.

[jonathan-buttner]

do you think returning a partial list of the assets is helpful to the requester?

Good point. I was thinking of how the UI currently uses the return value, and by returning just a list, albeit a shorter or empty one, the UI doesn't need additional changes for what should be a rare edge case.

Gotcha, the Security Solution app calls the bulk upgrade endpoint which uses this code path I believe. We don't check the return value so we're fine there. I still think it might be worth refactoring this in the future because other apps might call the apis frequently which could cause the issue to happen more often.

[skh]

@jonathan-buttner Thanks! I've opened #84656 and #84651 as follow-up issues.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: e0aad6c

Metrics [docs]

✅ unchanged

History

• 💚 Build #90638 succeeded e0aad6c
• 💚 Build #89902 succeeded 3ebb4bb

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream