[Fleet] catching only mapper errors

[juliaElastic]

Summary

Closes #162772

Changed the logic to only rollover on mapper errors, will check in ES what are the possible errors.
I think it would be useful to completely fail on unexpected errors instead of hiding bugs (like the simulate template error).

Added 2 flags in package install API to be able to ignore unexpected mapping errors and skip rollover, this can be useful if we want to bypass throwing an error / doing rollover in a support case.

To verify, run the package install API with the flags to skip rollover:

// latest prerelease version in latest main
POST kbn:/api/fleet/epm/packages/apm/8.11.0-preview-1695202343?skipDataStreamRollover=true&ignoreMappingUpdateErrors=true
{
  "force": true
}

// take latest from registry, this installs 8.4.2
POST kbn:/api/fleet/epm/packages/apm?skipDataStreamRollover=true&ignoreMappingUpdateErrors=true
{
  "force": true
}

See in the logs that rollover is skipped:

[2023-09-26T11:29:09.802+02:00][INFO ][plugins.fleet] Mappings update for metrics-apm.internal-default failed due to ResponseError: illegal_argument_exception
        Root causes:
                illegal_argument_exception: Mapper for [clr.gc.gen0size] conflicts with existing mapper:
        Cannot update parameter [index] from [false] to [true]
[2023-09-26T11:29:09.802+02:00][INFO ][plugins.fleet] Skipping rollover for metrics-apm.internal-default as "skipDataStreamRollover" is enabled

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[apmmachine]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• /oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[juliaElastic]

My only concern here is that this might surface existing errors that were ignored before, but I think it's better to see the errors early.

[nchaulet]

@juliaElastic I think it's a great improvement but I am wondering if we should push that changes with a new flag to bypass mappings update/rollover and log the errors, so we have an escape hatch for future SDH if things go wrong.

[nchaulet]

I am wondering if it make more sense to have these flags on the install package endpoint, so we can apply this on case per case basis, if an error happen or a package upgrade is stuck, wdyt?

[juliaElastic]

Yeah that could be easier to apply than a kibana config, the only caveat is that we can't stop Fleet setup to re-run the upgrade if a preconfigured package has an issue.
Though if the API call succeeds, the setup should not try to reinstall the package, so it might be enough to have the flags on the API.

[juliaElastic]

the API flags are ready

[nchaulet]

LGTM 🚀 This will be a great tool for future SDHs

[kibana-ci]

💔 Build Failed

• Buildkite Build
• Commit: cba97a5
• Interpreting CI Failures

Failed CI Steps

• FTR Configs #32
• Check Types Commit Diff

Test Failures

• [job] [logs] FTR Configs #32 / serverless observability UI Observability Log Explorer Header menu "before all" hook for "should inject the app header menu on the top navbar"

Metrics [docs]

✅ unchanged

History

• 💔 Build #162313 failed 6e072a6
• 💔 Build #162292 failed 584c7d9
• 💔 Build #162277 failed 9cd6816
• 💔 Build #162209 failed c40e845

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @juliaElastic

[kpollich]

Looks great thank you!

[juliaElastic]

Tests seem to fail due to this issue: #167255

[watson]

I've verified that the 26 TypeScript errors coming from x-pack/test/tsconfig.json in the "Check Types Commit Diff" check are not related to this PR and so we can safely merge it as long as the other CI checks are green