Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update (most) obs-cloud-monitoring owned packages to format_version 3.0.0 #8041

Conversation

tommyers-elastic
Copy link
Contributor

@tommyers-elastic tommyers-elastic commented Oct 2, 2023

This PR updates all obs-cloud-monitoring owned packages, except azure and azure_metrics, to package spec v3.

These changes are mostly autogenerated in three stages.

  1. Firstly we run elastic-package format on everything. This commit can be excluded from review to make the diff cleaner.

  2. Then the following yq commands are run across all the affected packages:

find $1/data_stream -type f -path "*fields*" -name "*.yml" -exec yq e -i "del .[].release" {} \;
find $1/data_stream -type f -path "*fields*" -name "ecs.yml" -exec yq e -i "unique_by(.name)" {} \;
yq e -i 'del .release' manifest.yml
yq e -i "with(select(.license != null); .conditions.elastic.subscription = .license) | del .license" manifest.yml
  1. Next the ecs-update tool is run as follows:
go run github.com/andrewkroh/go-examples/ecs-update@main -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type -owner elastic/obs-cloudnative-monitoring packages/*

Any remaining issues were fixed manually. Even though most of the changes here were generated by tooling, this still needs careful review (hide whitespace changes to make your life easier).

There should be no functional changes to any of the packages here.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/aws_logs
The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/awsfargate
The format_version in the package manifest changed from 2.9.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/awsfirehose
The format_version in the package manifest changed from 2.9.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/azure_application_insights
The format_version in the package manifest changed from 2.9.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/azure_billing
The format_version in the package manifest changed from 2.0.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/gcp_metrics
The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/azure_metrics
@tommyers-elastic tommyers-elastic added the enhancement New feature or request label Oct 2, 2023
@tommyers-elastic tommyers-elastic requested a review from a team as a code owner October 2, 2023 12:25
@elasticmachine
Copy link

elasticmachine commented Oct 2, 2023

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-10-13T15:44:02.024+0000

  • Duration: 25 min 58 sec

Test stats 🧪

Test Results
Failed 0
Passed 37
Skipped 0
Total 37

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link

elasticmachine commented Oct 2, 2023

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (1/1) 💚
Files 100.0% (1/1) 💚
Classes 100.0% (1/1) 💚
Methods 66.667% (14/21) 👍 3.252
Lines 100.0% (135/135) 💚 0.137
Conditionals 100.0% (0/0) 💚

@tommyers-elastic tommyers-elastic changed the title Update (most) obs-cloud-monitoring owned packages to format version 3 Update (most) obs-cloud-monitoring owned packages to format_version 3.0.0 Oct 2, 2023
Copy link
Contributor

@zmoog zmoog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@tommyers-elastic, do you plan to address the platformlogs' _dev/test/pipeline/test-platformlogs-invalid-raw.log malformed JSON issues on in a different PR?

IIRC, the test fails on stack versions > 8.6 because the json processor in Elasticsearch has become stricter by default starting from 8.7.

More on this issue at zmoog/public-notes#54

@tommyers-elastic
Copy link
Contributor Author

hey @zmoog yeh i have another PR just for azure because it contains quite a few more changes. i haven't addressed the malformed log issue there yet, but i will get round to it soon.

@zmoog
Copy link
Contributor

zmoog commented Oct 5, 2023

@tommyers-elastic, I mentioned it so we do not forget it. Happy to help with a review!

@tommyers-elastic tommyers-elastic force-pushed the update-cloud-packages-to-format-version-3 branch 4 times, most recently from 462fc6a to ed1b1fc Compare October 13, 2023 13:50
@tommyers-elastic tommyers-elastic force-pushed the update-cloud-packages-to-format-version-3 branch from ed1b1fc to f98d4e6 Compare October 13, 2023 14:13
@tommyers-elastic tommyers-elastic merged commit c9e4188 into elastic:main Oct 16, 2023
4 checks passed
@elasticmachine
Copy link

Package aws_logs - 0.6.0 containing this change is available at https://epr.elastic.co/search?package=aws_logs

@elasticmachine
Copy link

Package awsfargate - 0.4.0 containing this change is available at https://epr.elastic.co/search?package=awsfargate

@elasticmachine
Copy link

Package awsfirehose - 0.5.0 containing this change is available at https://epr.elastic.co/search?package=awsfirehose

@elasticmachine
Copy link

Package azure_application_insights - 1.2.0 containing this change is available at https://epr.elastic.co/search?package=azure_application_insights

@elasticmachine
Copy link

Package azure_billing - 1.3.0 containing this change is available at https://epr.elastic.co/search?package=azure_billing

@elasticmachine
Copy link

Package gcp_metrics - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=gcp_metrics

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants