Convert Journald integration to input type

[chemamartinez]

What does this PR do?

Convert the Journald integration to an input type package according to the specs located here.

It has also been moved to GA by bumping its version to 1.0.0.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Related issues

• Closes [Journald] Convert to input package and make GA #5750

Screenshots

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-04-20T15:52:51.271+0000

• Duration: 14 min 55 sec

Test stats 🧪

Test	Results
Failed	0
Passed	1
Skipped	1
Total	2

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[P1llus]

@jsoriano @hop-dev I think we need to quickly disucss the current spec/format for input packages before we merge this one. I want to see if we can clarify a few things we have issues with.

1. Is the format for system test folders the same now without datastreams?
2. How do we generate docs?
3. Where should we put build.yml?

Looking at the log integration package, it is missing several crucial pieces, so we cant use that as a reference.

I thought it would be quickest to discuss this quickly on slack, so we can get a good understanding of the intended format.

[jsoriano]

@jsoriano @hop-dev I think we need to quickly disucss the current spec/format for input packages before we merge this one. I want to see if we can clarify a few things we have issues with.

1. Is the format for system test folders the same now without datastreams?

It is mostly the same, yes. Regarding the spec intended format, you can see an input package as a single datastream.

2. How do we generate docs?

In principle it is also the same, with files under the docs directory. Though current helpers for data streams probably won't work now. I think we haven't added any input with documented fields yet, this would be something to improve if it doesn't work now.

3. Where should we put build.yml?

According to the spec, in the same place as in integration packages, in _dev/build/build.yml.

Looking at the log integration package, it is missing several crucial pieces, so we cant use that as a reference.

Are you missing other pieces apart of the ones you mention here?

I thought it would be quickest to discuss this quickly on slack, so we can get a good understanding of the intended format.

Maybe we can schedule a meeting, including also @hop-dev and @ishleenk17.

[ishleenk17]

Datastream name needs to be taken up as input from the user in your journald.yml.hbs file

[ishleenk17]

For input packages, we let the user do all the fields mappings through Kibana.
We don't do any mapping of the processed data under fields (like done in input.yml).
Only very generic fields should be mapped here.

[ishleenk17]

2. How do we generate docs?

We don't get the sample event in the docs using the helpers, unlike in integrations.
It doesn't follow the auto generations format. It has to be done manually.

These are some of the input packages already implemented:
Jolokia Input, SQL, Statsd.

All of them might not have system tests though.

[ishleenk17]

Maybe we can schedule a meeting, including also @hop-dev and @ishleenk17.

Happy to do that.
I have shared some of my initial review comments. Let me know if a meeting is required as well.

[P1llus]

@ishleenk17

Datastream name needs to be taken up as input from the user in your journald.yml.hbs file

Shouldn't we let elastic-package or fleet handle this? We already inject the UI elements to configure this, might as well also inject it into the config? Its easy to forget, because we have to remove the datastream settings from the manifest.yml, but keep it in the *.yml.hbs files.

[ishleenk17]

@ishleenk17

Datastream name needs to be taken up as input from the user in your journald.yml.hbs file

Shouldn't we let elastic-package or fleet handle this? We already inject the UI elements to configure this, might as well also inject it into the config? Its easy to forget, because we have to remove the datastream settings from the manifest.yml, but keep it in the *.yml.hbs files.

Fleet might be the right place to handle this for the input package.
But, currently, it's not supported. So you might want to add it for now and raise this request with Fleet.

[chemamartinez]

For input packages, we let the user do all the fields mappings through Kibana.
We don't do any mapping of the processed data under fields (like done in input.yml).
Only very generic fields should be mapped here.

@ishleenk17 I have some doubts regarding this suggestion. I understand we want let the users do the mapping of non-generic fields. However there are certain fields in input.yml that always appear in events (e.g. journald.pid) that might be interesting to keep.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[P1llus]

LGTM. Let's do the planned manual tests first, then we can merge if its all working as intended.

[chemamartinez]

Solved an issue with the mapping of event.created field.

Fixed at 892a487

[pierrehilbert]

If you can just add the link to your E2E tests here it would be perfect!
Other than that, LGTM!

[ishleenk17]

Looks good!

[elasticmachine]

Package journald - 1.0.0 containing this change is available at https://epr.elastic.co/search?package=journald