[Fleet] Update Package Policy UI to Support Upgrading Package Policies

[kpollich]

Summary

This PR encapsulates the "Upgrade package policy UI" milestone laid out in #106048. It adds support for upgrading package policies to the latest installed version of a given integration.

See Figma Designs

Screen Recording

• Adjust Fleet > Agent policies > Agent policy details integrations table to match
  design
  • Remove description column, move description to a help icon with tooltip (only if the package policy has a description)
  • Add warning icon with tooltip if the policy package version =/= the currently installed package version
  • Add Upgrade policy action menu item if the policy package version =/= the currently installed package version
• Have the same changes in Integrations > Integration details > Policies table
  • Add warning icon with tooltip if the policy package version =/= the currently installed package version
  • Add Upgrade policy action menu item if the policy package version =/= the currently installed package version
• When user clicks Upgrade policy, take them to the policy editor
  • Perform a dry run against the upgrade package policies API
  • If the dry run comes back successful, add callout matching this design
    • Page title change for upgrade flow
    • Page description change for upgrade flow
    • Success callout
  • If the dry run comes back with errors, add callout matching this design
    • Page title & description change (same as above)
    • Warning callout
    • The policy editor should be populated with default values from the new package version (same as if the user were to add this integration for the first time)
    • Clicking previous configuration should open a flyout with the JSON value of the existing package policy to enable users to copy over previous values (diff field from API)
    • Investigate if time allows: how much of the previous configuration can we copy over "smartly"?
• When user clicks Save integration, save the adjusted policy with the new package version on the package metadata
  • Any upgraded package policies should also trigger an update on their parent agent policies
  • Redirect user back to wherever they came from (Integrations policies table, or Fleet agent policies integrations table)

Testing Instructions

1. Run the https://github.com/elastic/package-registry project locally, and ensure your local Kibana instance is pointing at your local package registry via xpack.fleet.registryUrl: http://localhost:8080 in your kibana.dev.yml. Also ensure you have https://github.com/elastic/integrations cloned locally and https://github.com/elastic/elastic-package installed.
2. Install the apache integration to your default agent policy
3. Manually bump the version (and optionally add some new stream/input variables) to the apache package in its manifest.yml
4. Build the apache package via elastic-package build
5. Copy the built contents of your new apache package version to your package registry's build directory, e.g. cp -R ../integrations/build/integrations/apache/<new_version> ./build/package-storage/packages/apache
6. Restart your local package registry
7. Install the latest version of apache via the Integrations -> Manage -> Apache -> Settings screen
8. Navigate to either the Integrations -> Manage -> Apache -> Policies or Fleet -> Agent Policies -> Default Policy -> Integrations screens
9. Click the "Upgrade" button next to the apache package policy, or select Upgrade Policy from the actions menu in the table
10. Make any changes to custom variables you may have added on the "Upgrade Integration" screen, then save to confirm
11. Note the version of the package and any variable changes have been persisted to your package policy

[kpollich]

@jen-huang @nchaulet - I was working on adding the "previous configuration" flyout to this UI when I came across a question.

When we have an input variable removed between versions (e.g. I removed "KP Test 4" in this latest version of the Apache package), how should we handle allowing the user to un-set that value? In this PR, I've made a change so that input variables are populated from the next version of the package before upgrading, so "KP Test 4" doesn't appear in the UI since it's not present in the latest version of Apache.

See screen recording for example:

Maybe I'm misunderstanding, but do we need to display the union of both sets of variables between the previous and next version of the package? We have some general thinking around field level conflicts in the designs here, for context:

Let me know if you have thoughts here.

[nchaulet]

When we have an input variable removed between versions (e.g. I removed "KP Test 4" in this latest version of the Apache package), how should we handle allowing the user to un-set that value? In this PR, I've made a change so that input variables are populated from the next version of the package before upgrading, so "KP Test 4" doesn't appear in the UI since it's not present in the latest version of Apache.

@kpollich 👍 on using the next package version as the source of truth for the policy builder variables. My guess here is if a package variable is deleted we should automatically remove the var for the user in the upgrade API I do not think there is a need to keep unused variables.

[nchaulet]

I am wondering here if it will make more sense for the user to show the compiled package policy as it's sent to the agent? we never show the package policy and I think we should probably hide that complexity to the user.

[kpollich]

Yeah definitely. Is there a utility for compiling it into that format?

[nchaulet]

maybe that function? https://github.com/elastic/kibana/blob/master/x-pack/plugins/fleet/common/services/package_policies_to_agent_inputs.ts#L13

[kpollich]

When we have an input variable removed between versions (e.g. I removed "KP Test 4" in this latest version of the Apache package), how should we handle allowing the user to un-set that value? In this PR, I've made a change so that input variables are populated from the next version of the package before upgrading, so "KP Test 4" doesn't appear in the UI since it's not present in the latest version of Apache.

@kpollich 👍 on using the next package version as the source of truth for the policy builder variables. My guess here is if a package variable is deleted we should automatically remove the var for the user in the upgrade API I do not think there is a need to keep unused variables.

This is something that I think requires further discussion. The API that was set up in #103017 will return an error if a variable is removed between versions. Right now, this is our only notion of "conflict detection", as far as I can tell. I believe this error surfaces when we attempt to compile the package inputs for the next version here:

kibana/x-pack/plugins/fleet/server/services/package_policy.ts

Lines 658 to 697 in 58054c3

	async function _compilePackagePolicyInput(
	registryPkgInfo: RegistryPackage,
	pkgInfo: PackageInfo,
	vars: PackagePolicy['vars'],
	input: PackagePolicyInput
	) {
	const packagePolicyTemplate = input.policy_template
	? pkgInfo.policy_templates?.find(
	(policyTemplate) => policyTemplate.name === input.policy_template
	)
	: pkgInfo.policy_templates?.[0];
	if (!input.enabled || !packagePolicyTemplate || !packagePolicyTemplate.inputs?.length) {
	return undefined;
	}
	const packageInputs = packagePolicyTemplate.inputs;
	const packageInput = packageInputs.find((pkgInput) => pkgInput.type === input.type);
	if (!packageInput) {
	throw new Error(`Input template not found, unable to find input type ${input.type}`);
	}
	if (!packageInput.template_path) {
	return undefined;
	}
	const [pkgInputTemplate] = await getAssetsData(registryPkgInfo, (path: string) =>
	path.endsWith(`/agent/input/${packageInput.template_path!}`)
	);
	if (!pkgInputTemplate || !pkgInputTemplate.buffer) {
	throw new Error(`Unable to load input template at /agent/input/${packageInput.template_path!}`);
	}
	return compileTemplate(
	// Populate template variables from package- and input-level vars
	Object.assign({}, vars, input.vars),
	pkgInputTemplate.buffer.toString()
	);
	}

[kpollich]

@elasticmachine merge upstream

[nchaulet]

This is something that I think requires further discussion. The API that was set up in #103017 will return an error if a variable is removed between versions.

In my opinion here if a variable is removed, and no new variable are required we should probably not throw an error what do you think? As a user I want to have the smoothest experience possible and if I can avoid to manage conflict it's probably better no? If we go in this direction this should probably be documented for packages maintainer

[kpollich]

This is something that I think requires further discussion. The API that was set up in #103017 will return an error if a variable is removed between versions.

In my opinion here if a variable is removed, and no new variable are required we should probably not throw an error what do you think? As a user I want to have the smoothest experience possible and if I can avoid to manage conflict it's probably better no? If we go in this direction this should probably be documented for packages maintainer

I spoke with @jen-huang about this a bit, and I think our approach will be to surface this a non-blocking error. e.g. just an "info" level callout that says something like.

The following fields have been deprecated, and will be removed from your package policy:

- field_one: "Value"
- field_two: "Value"

cc @dborodyansky - thoughts on this case? Non-conflicting notices before a user upgrades.

[kpollich]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 7aeefa4
• Storybooks Preview
• Documentation Changes

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
fleet	506	509	+3

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
fleet	716.8KB	731.4KB	+14.6KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
fleet	446.6KB	454.0KB	+7.4KB

Unknown metric groups

API count

id	before	after	diff
fleet	1153	1156	+3

API count missing comments

id	before	after	diff
fleet	1052	1055	+3

History

• 💔 Build #144218 failed 5632d80
• 💔 Build #144212 failed b5e4a6c
• 💚 Build #144131 succeeded 89e2940
• 💔 Build #144062 failed 91c4408
• 💔 Build #144039 failed d303cec

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @kpollich

[nchaulet]

Looks good to me, did some tests locally and everything worked 🚀

[kibanamachine]

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.