[Fleet] Add support for bundling Stack-version aligned packages with Kibana

[joshdover]

In #111858 we will be adding support for automatically upgrading a few select packages when Kibana upgrades (or downgrades) to ensure that Stack features that depend on package ingest assets being installed (specifically, index templates and ingest pipelines) will work correctly when a user upgrades Kibana. This is necessary for Kibana features to work correctly, for example new features in the APM UI that depend on new fields that were added in the same Stack version.

To improve the reliability of Stack upgrades, we eventually want to make some changes to this upgrade-on-boot process:

• Block Kibana's startup sequence on package upgrades
  • This aids operators and orchestration systems by allowing them to wait to upgrade Fleet Server until Kibana has completely started up and is serving traffic.
  • Bundling packages would remove a point of failure (accessing EPR) and slightly improve the performance of this process (see [Fleet] Improve performance of Fleet setup API / package installation #110500)
• Support air-gapped & firewalled clusters out-of-the-box
  • Kibana nodes that are operating without access to EPR will not be able to fetch packages that need to be upgraded on Kibana upgrades.
  • The workaround for this is to deploy a private EPR instance and configure Kibana to read it, but this manual step could be removed for Stack version-aligned packages if we could bundle these packages with Kibana's distributable.
  • This will also allow us to report Fleet's status more accurately instead of "skipping" errors w/ package installation as reporting Fleet as Available.

Challenges

• Bundled packages will need to leverage Fleet's "upload package" feature, which is half-baked at best.
  • There are many code paths that do not take into account packages that were uploaded instead of fetched from EPR.
  • We need to refactor the EPM code to add abstractions around packages and where they originated from (upload, disk, EPR, etc.) and ensure all installation and upgrade logic takes this into account.
  • We likely also need the ability to still search for newer versions of the package from EPR in the case a package bug was fixed since the Kibana release.
• Release process
  • We should add tooling for importing packages from the package-storage repository, ideally opening automatic PRs when new package versions become available for a given Kibana release.
  • This likely does not have to ship in the initial version and could happen as a process improvement follow-up task.

Related

• Allow Kibana to setup the stack #89827
• [Fleet] Initiate Fleet setup on boot #111858
• [Fleet] Use epm-packages saved object as main source of truth for installed packages #81995
• [Fleet] Refactor package installation / upgrade / downgrade and error recovery / rollback #82007
• [Fleet] Allow to install a package without the registry. #70582

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[joshdover]

cc @axw @ruflin

[axw]

We should add tooling for importing packages from the package-storage repository, ideally opening automatic PRs when new package versions become available for a given Kibana release.

Is package-storage still relevant for these packages once they're bundled in Kibana? Can we skip package-storage and go straight to Kibana? This is what I had in mind for APM. A few options come to mind for how to do this:

1. develop the package directly in Kibana (probably not ideal for us)
2. develop in apm-server like we do today, and sync to the Kibana repo regularly (e.g. nightly)
3. develop in apm-server like we do today, and have release-manager copy from apm-server into the artifacts it builds

We likely also need the ability to still search for newer versions of the package from EPR in the case a package bug was fixed since the Kibana release.

For APM, if there's an issue with the package that needs fixing it should go into the next stack patch or minor release. I don't know about other stack-aligned packages (@ruflin?) but I would have thought the same would apply there.

[joshdover]

Is package-storage still relevant for these packages once they're bundled in Kibana? Can we skip package-storage and go straight to Kibana?

If we don't need the ability to upgrade these packages out-of-band from stack releases (to fix a bug for instance), then I agree we can skip package-storage completely and just sync with the Kibana repository.

2. develop in apm-server like we do today, and sync to the Kibana repo regularly (e.g. nightly)

This is the option I would prefer so that we can test this in development easily without requiring a nightly build.

For APM, if there's an issue with the package that needs fixing it should go into the next stack patch or minor release.

As I mentioned above, if we don't need this ability, I totally agree. We've been shipping the APM product without the ability to update ingest pipelines and templates out-of-band from Stack releases for some time now, so I think we'd be safe to hold off on supporting both.

[axw]

2. develop in apm-server like we do today, and sync to the Kibana repo regularly (e.g. nightly)

This is the option I would prefer so that we can test this in development easily without requiring a nightly build.

SGTM. We would also want to be able to override what's in the Kibana image when running tests in apm-server. e.g. by bind-mounting a package into the Kibana Docker container.

[joshdover]

SGTM. We would also want to be able to override what's in the Kibana image when running tests in apm-server. e.g. by bind-mounting a package into the Kibana Docker container.

Makes sense, we'll make sure we document where these files are kept on disk for this use case.

[ruflin]

I expect the packages we ship with Kibana to also be available in the registry. This has the side effect that assuming someone is on 7.20.0 and the package 7.20.1 comes up, it could be installed. But the package itself should always only be compatible within the same minor. I would expect this could also become helpful during testing to test upgrades.

For the testing the other part is to be able to upload a package to Kibana. It is a feature that partially exist today but will likely be improved in the context of getting this feature done. Having this would eliminate the problem of bind-mount packages into Kibana potentially.

[kpollich]

High-level implementation plan

Checking in here with a high-level plan for the Fleet team's implementation.

Our immediately obtainable goals for this are:

• Check stack-aligned packages into the Kibana Git repo
• Update Fleet setup to install bundled packages from disk
• Ensure other Fleet code paths account for bundled packages when possible

Thinking about these goals from a minimum viable implementation, we're going to pursue the following:

1. Create a fleet/server/bundled_packages directory and manually check in our initial set of stack-aligned packages
2. Make code changes in Fleet's setup process to include these bundled packages in a package installation step that does not require a connection to EPR

For the initial implementation, any updates to the bundled packages will be manual through GitHub PR's. As an immediate follow-up we'd like to implement some automation that allows us to do the following on a nightly basis:

1. Use a fleet/bundle_packages.yml config file that lists bundled packages to fetch package data from EPR or directly from the elastic/package-storage repo
2. Replace corresponding package files in the fleet/server/bundled_packages directory
3. If a diff is reported, file an auto-merge PR to update the bundled package files

Current state of the “Install Package by Upload” feature

As Ruflin mentioned above:

For the testing the other part is to be able to upload a package to Kibana. It is a feature that partially exist today but will likely be improved in the context of getting this feature done. Having this would eliminate the problem of bind-mount packages into Kibana potentially.

Fleet currently includes an API endpoint that allows for the installation of packages via upload. This endpoint simply takes the entire request body, converts it to a Buffer, then installs the package using that Buffer as its source. This API endpoint is currently undocumented. So, this feature at it exists today doesn't really include any logic - it's just a passthrough to Fleet's installation logic that expects the entire package source (e.g. https://epr.elastic.co/package/log/0.4.6/) to be provided as the request body.

Relevant code:

• Endpoint definition
• Request body schema
• API Handler
• Install package service method

---

I'm continuing to work on auditing the current landscape of the upload feature in the Fleet code. Mainly, I want to track down areas and potential breakages that don't account for packages installed via upload instead of the registry. We have some branching logic, etc, that might introduce concerns that should be documented.

[simitt]

Use a fleet/bundle_packages.yml config file that lists bundled packages to fetch package data from EPR or directly from the elastic/package-storage repo
Replace corresponding package files in the fleet/server/bundled_packages directory

Can this be implemented to pull the latest versions from staging? When Kibana bundles stack version aligned packages, these packages won't be pushed to production at the time when bundling, but they should already be pushed to staging, most certainly with pre-release versions (for APM we use x.y.z-dev1). Despite using SemVer and pre-release versions, the current Fleet, packages and EPR handling does not support pushing prerelease versions to production, as this would be shown as the latest available package to users in production.

[kpollich]

Can this be implemented to pull the latest versions from staging?

Yes this shouldn't be a problem. We can make it clear in the manual process that bundled packages should come from staging, and then when we automate the process enforce the same rule. Thanks for the clarity here!

[simitt]

@kpollich defining the flow in more detail, we suggest to even fetch from snapshot instead of staging, which gives some advantages. The package registry allows to republish a package with the same version for thesnapshot branch (but not for staging or production). This allows us to push packages with the actual release version instead of a pre-release version, simplifying the process.
Furthermore, the snapshot registry also includes packages from production and staging and therefore gives the most freedom in fetching the latest available package for a specific version.

[mtojek]

@kpollich @joshdover

Please be aware that the Ecosystem team is going to get rid of the Package Storage repository (deprecate) and replace it with GCP buckets. The entire solution will be fully automated and won't require any manual interaction. We have also decided to get rid of stages and keep all packages in the flat space. Stages will be replaced with semver rules. Long story short, a package can be released or "prereleased", Kibana decides whether include prerelease packages in their listing.

cc @jsoriano @jlind23

[jlind23]

ping @ruflin

[joshdover]

One thing I want to be clear on here is that this effort does not add support for installing bundled packages from the UI or API. This will only support installing packages via setting xpack.fleet.packages in kibana.yml and upgrading any existing bundled packages during Kibana upgrades.

This means that new clusters set up in air-gapped environments would either need to use the appropriate xpack.fleet.packages settings to install these packages, or should setup an on-prem EPR instance. On-prem EPR is the recommended approach as it allows access to a wide array of Elastic Agent integrations.

If installing bundled packages via the UI or API is desired, we will need to consider that separate and discuss the product and technical implications.

[mtojek]

If installing bundled packages via the UI or API is desired, we will need to consider that separate and discuss the product and technical implications.

Yes, it was and still is desired, as it will simplify the development workflow for elastic-package. We can take it to a separate discussion :) cc @jsoriano

[simitt]

This means that new clusters set up in air-gapped environments would either need to use the appropriate xpack.fleet.packages settings to install these packages, or should setup an on-prem EPR instance.

For ECE the xpack.fleet.packages setting is used for the setup. That does mean this should be working right?

[joshdover]

For ECE the xpack.fleet.packages setting is used for the setup. That does mean this should be working right?

Yes, the ECE scenario should work. We should include this in our test plan for 8.1 testing. @dikshachauhan-qasource do you all have experience testing against ECE?

[kpollich]

Hello all, I'm adding some notes on our follow-up implementation to this process in 8.2 below.

---

In 8.1, we're shipping #122297 - which contains a "first-step" implementation with baseline support for bundled packages as a concept. We're checking in zip archives of a hard-coded set of packages to the Kibana git repo and handling upgrades manually via pull requests. This is not an ideal process for a few reasons:

1. Additional manual steps are required to "promote" bundled packages and ensure they're bundled before a release
2. Large .zip files add bloat to Kibana's already too-large git repo
3. Lack of flexibility in the set of packages we consider bundled

So, moving forward in 8.2, the Fleet team will be expanding upon our initial implementation to solve these pain points. We're proposing an implementation that enhances Kibana's bootstrap process with a step to resolve bundled packages.

We'll add a step to Kibana's bootstrap process that does the following:

1. Read a configuration file that determines the list of bundled packages to fetch
2. Fetch the latest version of each of those packages from the EPR “archive” endpoint
3. Replace the contents of a Git-ignored directory with the new set of archive files

---

An open question I still have is how we should determine the EPR endpoint to fetch from. Fleet does have some existing logic we use to determine which registry endpoint we fetch from elsewhere that might be applicable here as well:

kibana/x-pack/plugins/fleet/server/services/epm/registry/registry_url.ts

Lines 22 to 31 in e45c352

	const getDefaultRegistryUrl = (): string => {
	const branch = appContextService.getKibanaBranch();
	if (branch === 'main') {
	return SNAPSHOT_REGISTRY_URL_CDN;
	} else if (appContextService.getKibanaVersion().includes('-SNAPSHOT')) {
	return STAGING_REGISTRY_URL_CDN;
	} else {
	return PRODUCTION_REGISTRY_URL_CDN;
	}
	};

@simitt mentioned previously that we should always fetch from snapshot, but I don't think that's necessarily true if we move this process to an automated build step. When we build on release branches, we probably want the bootstrap process to only bundle packages that are published to the production EPR.

Happy to hear any other thoughts folks might have on the matter as we look forward to a more fleshed-out implementation here.

[mtojek]

In the nearest future, we won't have different endpoints and will store packages in the same flat space (epr.elastic.co).

When we build on release branches, we probably want the bootstrap process to only bundle packages that are published to the production EPR.

Considering current deployment rules, it sounds reasonable. Revisions in production should be mature enough. If we decide to go with snapshot instead, keep in mind that packages should be published to that stage first. Yesterday (or the day before) we analyzed the issue caused by the apm package published to the staging branch, but not exposed/synced the snapshot one.

[joshdover]

An open question I still have is how we should determine the EPR endpoint to fetch from. Fleet does have some existing logic we use to determine which registry endpoint we fetch from elsewhere that might be applicable here as well:

We could just avoid this decision for now by requiring the full URL in the manifest of bundled packages. This would allow package owners to point to any location for the time being until we move to a single registry + semver.

[dikshachauhan-qasource]

Hi @joshdover

We have not done any specific testing against ECE, however, If you can specify details around it, then we can test that out.

Further, we are working upon creating proposed test cases around this ticket and have created at 8.1 Feature Proposed Testcases

Could you please take some time to look and confirm, if coverage done there is enough or we are missing anything.

Thanks
QAS

[ghost]

Hi @joshdover

We have created the future test cases for this ticket and uploaded the same on testrail. Please find below the test cases link:

• C165599
• C165598
• C165600

Further, we will create more test cases, if required.

Thanks!
QAS

[joshdover]

We have not done any specific testing against ECE, however, If you can specify details around it, then we can test that out.

We've delayed the changes to support air-gapped ECE to 8.2, so that can wait.

[kpollich]

I'm closing this parent issue. Further enhancements will be handled in their own issues and backlink here when appropriate. Thanks, all.

[ghost]

Hi Team
We have executed 03 testcases for this feature and marked it as Blocked under our Fleet Test plan at link:

• Stack Bundling

Note: This feature is not fully merged now. We will be retesting it on the release build once the feature will get merge.

Thanks!