[Fleet] Support Input Packages

[hop-dev]

Summary

Part of #137750.

Add support for the installation of input only packages. Current caveats to be addressed in later PRs before release goes out:

• Upgrades from non-input to input package not supported
• Not compatible with package verification yet

What is an input only package (high level)?

Input only packages have type : input as part of the manifest.
Input only packages do not define the stream as part of the package definition only the input. This means that we have to build the data stream as part of package policy creation.

Test instructions

• Download the test data zip:
  input-packages.zip

• Start a registry:

docker run -p 8080:8080 -v <path to unzipped dir>/input-packages/packages:/packages/test-packages -v <path to unzipped dir>/input-packages/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main

• Add this to your kibana config:

xpack.fleet.registryUrl: http://localhost:8080

• the custom_logs and sql_input input integrations are now available for install (custom logs is a good test package as it behaves exactly the same as the logs integration) for example custom_logs integration can be found at <your kibana URL>/app/integrations/detail/custom_logs-1.0.0/overview:

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Unit or functional tests were updated or added to match the most common scenarios

[hop-dev]

@elasticmachine merge upstream

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: d7c9cf2

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
fleet	719	720	+1

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
fleet	880	870	-10

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
fleet	869.6KB	869.7KB	+137.0B

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
fleet	15	17	+2

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
fleet	105.9KB	107.0KB	+1.1KB

Unknown metric groups

API count

id	before	after	diff
fleet	979	969	-10

History

• 💚 Build #71612 succeeded a0aac92
• 💚 Build #71347 succeeded c21993d
• 💚 Build #71260 succeeded 1aea9c2
• 💔 Build #71226 failed 12ffa6c
• 💔 Build #71218 failed e78a688
• 💔 Build #71209 failed a52ea39

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[cauemarcondes]

LGTM

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.4	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 140035

Questions ?

Please refer to the Backport tool documentation

[ghost]

Hi @hop-dev @jen-huang,

We have re-validated this ticket on the latest 8.6.0 BC1 Kibana Self-managed environment and found the below observations.

Build details:

Version: 8.6.0 BC1
Build: 58392
Commit: 50a7feb0a5eb068d3acccc49c83b9ccb6db6734f

Below are the observations:

• Custom Logs Integrations is shown on searching the custom logs on Integrations page.

Screenshot:

Hence, marking this ticket as QA: Validated.

Please let us know if we are missing anything.

Thanks!

[hop-dev]

Hi Prachi, there is a further test step to be performed to confirm that data is generated.
Do you have any test scenarios that use the custom logs integration to generate log data?

Something like:

• Install custom logs integration
• On the agent box, create /tmp/test.log with some file content
• Add the custom logs integration to an agent policy with the agent assigned
  • set log file to /tmp/test.log
  • set tags to tag1
  • set dataset to "mydataset"
• GET /logs-mydataset-default/_search should return data (after a minute or so)

[ghost]

Hi @hop-dev,

We were setting up the environment to check the custom logs on Kibana on Ubuntu machine but Kibana is not getting launched after running the registry command.

docker run -p 8080:8080 -v <path to unzipped dir>/input-packages/packages:/packages/test-packages -v <path to unzipped dir>/input-packages/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main

Please find the below steps performed on setting up the environment:

1. Installed Docker on Ubuntu 20.04 machine.

2. Setup Self-Managed environment on 8.5.2 build on Ubuntu 20.04 machine.
3. Installed Fleet Server and Windows agent on 8.5.2 Kibana.
4. Start a registry using the command:

docker run -p 8080:8080 -v $(pwd)/input-packages/packages:/packages/test-packages -v $(pwd)/input-packages/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main

5. Stopped the Kibana and added xpack.fleet.registryUrl: http://localhost:8080/ in Kibana.yml

6. Restarted the Kibana.

7. Below error is shown on Kibana restart and Kibana is not launched.

Could you please confirm is there any other way to setup the environment other than above mentioned one.

Please let us know if we are missing anything.

Queries

Could you please confirm the following:

• What is the use case of this feature.
• Could this feature be done on Cloud environment as well.
• Do we need to add this feature in our future regression plans.

Thanks!

[cmacknz]

@jen-huang can someone answer the QA team's questions at the end of the comment above?

[jen-huang]

Tagging @hop-dev for reviewing these Qs

[hop-dev]

What is the use case of this feature.

Input packages will allow the user to customise their ingest a lot more than a traditional 'integration' package. For example, an integration package defines the inputs (where the data comes from) and the data streams (how the data is stored in elastic) whereas input packages only define the input, allowing us to create the data stream dynamically and give the user more configuration options.

As we are only at phase q a lot of this configuration is not available yet, (see the fleet parent issue for more info on where we are heading with this) This is why at the moment the testing focuses on input packages working the same as the existing integration packages, but behind the scenes the implementation has quite a few differences.

More reading:

• Meta issue.
• Fleet parent issue

Could this feature be done on Cloud environment as well.

As there are no input only packages released on the package registry I dont believe we can test this on cloud yet

Do we need to add this feature in our future regression plans.

Yes I believe so, however it may be premature to add this until an input only package is released? The test setup at the moment is quite labour intensive. @jen-huang do you agree?

[hop-dev]

@prachigupta-qasource it looks like your elasticsearch was unhealthy from the screenshots, I would recommend wiping your elasticsearch and trying again.

Happy to jump on a zoom call and debug some time? so sorry about the delayed reply!

[ghost]

Hi @hop-dev,

We are blocked to test this ticket due to #92485.

We will test this ticket once the blocker issue is fixed.

Thanks!

[amolnater-qasource]

Hi @hop-dev
Thank you for sharing all the details, it's very helpful.

We have revalidated this feature on latest 8.6 BC10 kibana self-managed environment and found had below observations.

• We are successfully able to install Custom logs integration.
• On following below steps:
  • Add the custom logs integration to an agent policy with the agent assigned
  • set log file to /tmp/test.log
  • set tags to tag1
  • set dataset to "mydataset"
  • GET /logs-mydataset-default/_search
• We have received data on running GET command.

Build details:
BUILD: 58852
COMMIT: d3a625e

Screen Recording:

Amol-Chas_Ubuntu-20.-.Google.Chrome.2023-01-09.12-28-50.mp4

Hence, marking this as QA:Validated.

Please let us know if anything else is required from our end.

Thanks

[amolnater-qasource]

Hi @hop-dev
As we are now successfully able to complete the test for this feature and we have the clear steps available.

Could you please confirm if we need to add this feature for testing in our regression cycle.

Thank you.

[hop-dev]

Hi @amolnater-qasource, I think we do want to add it yes as going forward input packages will be quite important to the fleet user experience, however I understand not having input packages readily available makes the testing of this feature a bit more involved.

@kpollich do you agree we should add this to the regression tests?

[jen-huang]

Yes, please add to regression tests.

[amolnater-qasource]

Hi @hop-dev @jen-huang

Thank you for confirming.
We have created 02 testcases for this feature under our Fleet test suite at links:

• [Self-Managed]: Setup environment for validating Input Packages.
• [Self-Managed]: Validate data for custom logs integration for input packages.

Please let us know if any other scenario needs to be covered.
Thanks!