Skip to content

Conversation

@ruflin
Copy link
Contributor

@ruflin ruflin commented Sep 8, 2023

The dataset value in the manifest.yml file can be overwritten to allow dataset names which are not {package}.{dirName}. One example of this can be found here: elastic/integrations#7670 This is already in use today and is used by some packages. Fleet installs all the assets as expected.

When working on elastic/integrations#7670, CI is failing. The reason is that the pipeline check did not take the custom dataset config into account. The reason why this didn't show up earlier is because the check for the template already has a similar condition in place. Likely all previous packages that used this feature did only have templates and no pipelines.

The dataset value in the manifest.yml file can be overwritten to allow dataset names which are not `{package}.{dirName}`. One example of this can be found here: elastic/integrations#7670 This is already in use today and is used by some packages. Fleet installs all the assets as expected.

When working on elastic/integrations#7670, CI is failing. The reason is that the pipeline check did not take the custom dataset config into account. The reason why this didn't show up earlier is because the check for the template already has a similar condition in place. Likely all previous packages that used this feature did only have templates and no pipelines.
Copy link

@tommyers-elastic tommyers-elastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ruflin
Copy link
Contributor Author

ruflin commented Sep 8, 2023

I'm wondering if the failure that CI hit is in any way related?

Error: error running package system tests: could not complete test run: could not find hits in metrics-aws.ec2_metrics-ep data stream

The following can be seen before in the logs which makes me suspicious:

WARN[0000] The "AWS_SESSION_TOKEN" variable is not set. Defaulting to a blank string.
WARN[0000] The "AWS_PROFILE" variable is not set. Defaulting to a blank string.

What does elastic-package exactly test around aws.ec2 metrics?

@ruflin
Copy link
Contributor Author

ruflin commented Sep 8, 2023

I did rerun the job and seems it was just flaky.

@ruflin
Copy link
Contributor Author

ruflin commented Sep 11, 2023

This PR currently blocks elastic/integrations#7670 @jsoriano As soon as this PR is in, will CI in elastic/integrations#7670 pick up the new version or will we need to first release a new version?

@mrodm
Copy link
Contributor

mrodm commented Sep 11, 2023

This PR currently blocks elastic/integrations#7670 @jsoriano As soon as this PR is in, will CI in elastic/integrations#7670 pick up the new version or will we need to first release a new version?

It's needed a new elastic-package release in order to be used by integrations. Once created the release, dependabot creates the required PR.

Copy link
Contributor

@mrodm mrodm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Could you update the test package awsfirehouse (test/packages/parallel/awsfirehose) to use that custom dataset as in the integrations PR ? In that way, this change could also be tested here in elastic-package @ruflin

@ruflin
Copy link
Contributor Author

ruflin commented Sep 11, 2023

@mrodm Good idea, I'll update it.

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

History

cc @ruflin

Copy link
Contributor

@mrodm mrodm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@ruflin ruflin merged commit 64e31c1 into elastic:main Sep 11, 2023
@ruflin ruflin deleted the pipline-check-fix branch September 11, 2023 14:04
bhapas pushed a commit to bhapas/elastic-package that referenced this pull request Sep 21, 2023
The dataset value in the manifest.yml file can be overwritten to allow dataset names which are not `{package}.{dirName}`. One example of this can be found here: elastic/integrations#7670 This is already in use today and is used by some packages. Fleet installs all the assets as expected.

When working on elastic/integrations#7670, CI is failing. The reason is that the pipeline check did not take the custom dataset config into account. The reason why this didn't show up earlier is because the check for the template already has a similar condition in place. Likely all previous packages that used this feature did only have templates and no pipelines.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants