Skip to content

Create input packages #1424

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 10 commits into from
Aug 31, 2023
Merged

Create input packages #1424

merged 10 commits into from
Aug 31, 2023

Conversation

@mrodm
Copy link
Contributor

@mrodm mrodm commented Aug 31, 2023

Fixes #1361

This PR adds to elastic-package create package the required questions to the survey to be able to create input packages.

Two new questions have been added:

  • Package type: input or integration (default integration)
  • Input Data Stream type: logs or metrics (default logs)
    • Depending on this value, policy_templates map in manifest.yml and agent config template would be updated accordingly

This would create an input package with these files and folders:

 $ tree myinput/
myinput/
├── agent
│   └── stream
│       └── input.yml.hbs
├── changelog.yml
├── docs
│   └── README.md
├── fields
│   └── base-fields.yml
├── img
│   ├── sample-logo.svg
│   └── sample-screenshot.png
├── LICENSE.txt
└── manifest.yml

Example of the survey shown:

 $ elastic-package create package
Create a new package
? Package type: input
? Package name: myinput
? Version: 0.0.1
? License: Elastic-2.0
? Package title: New Package
? Description: This is a new package.
? Categories: custom
? Kibana version constraint: ^8.8.2
? Required Elastic subscription: basic
? Github owner: elastic/integrations
? Input Data Stream type: logs
New package has been created: myinput
Done

@mrodm mrodm self-assigned this Aug 31, 2023
@mrodm mrodm requested a review from a team August 31, 2023 11:07
@mrodm mrodm marked this pull request as ready for review August 31, 2023 11:07
jsoriano
jsoriano approved these changes Aug 31, 2023
View reviewed changes
Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

internal/packages/archetype/input_agent_config.go Outdated
package archetype

const inputAgentConfigTemplate = `data_stream:
dataset: ` + "{{`{{data_stream.dataset}}`}}" + `
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cannot we avoid these concatenations, they make these templates difficult to read.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could be changed to:

Suggested change
dataset: ` + "{{`{{data_stream.dataset}}`}}" + `
dataset: {{ "{{data_stream.dataset}}" }}

It looks at least somehow better to read, no ? @jsoriano .

I could update the other templates to use this way of writing those strings.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just updated all the templates to use that syntax to write strings with {{}}

Moreover, all these templates and resources (logo and screenshot) have been moved to a _static folder and they are loaded now using embed

internal/packages/archetype/package.go
return fmt.Errorf("can't render sample screenshot: %w", err)
}

if packageDescriptor.Manifest.Type == "input" {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a test case in package_test.go?

@mrodm mrodm requested a review from jsoriano August 31, 2023 15:22
mrodm
mrodm commented Aug 31, 2023
View reviewed changes
internal/packages/archetype/_static/package-manifest.yml.tmpl
package archetype

const packageManifestTemplate = `format_version: 2.8.0
format_version: 2.10.0
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to use the latest package-spec version published

mrodm
mrodm commented Aug 31, 2023
View reviewed changes
internal/packages/archetype/package_test.go
Comment on lines +34 to 39
t.Run("input-pacakge", func(t *testing.T) {
pd := createPackageDescriptorForTest("input")

err := createAndCheckPackage(t, pd)
require.Error(t, err)
})
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added test for input package

jsoriano
jsoriano approved these changes Aug 31, 2023
View reviewed changes
Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

internal/packages/archetype/resources.go
var dataStreamElasticsearchIngestPipelineTemplate string

//go:embed _static/dataStream-manifest.yml.tmpl
var dataStreamManifestTemplate string
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Much better 👍

We could have placed these definitions in the files where they are used, but it is fine like this too.

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

History

cc @mrodm

@mrodm mrodm merged commit 9be83fe into elastic:main Aug 31, 2023
@mrodm mrodm deleted the create_input_packages branch August 31, 2023 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jsoriano jsoriano jsoriano approved these changes

Assignees

@mrodm mrodm

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Support input packages in elastic-package create

3 participants

@mrodm @elasticmachine @jsoriano