Add new validations to manage object_type and metric_type fields #527

mrodm · 2023-05-17T11:35:35Z

What does this PR do?

This PR adds new validations to include the cases where object_type and/or metric_type are set in field definitions.

If a field defines an object_type (new):
- the only possible value for type is object.
If a field defines a metric_type and an object_type
- currently, aggregate_metric_double not allowed.
- object_type just could have a different value of possible values:
  - histogram
  - long
  - integer
  - short
  - byte
  - double
  - float
  - half_float
  - scaled_float
  - unsigned_long
If a field defines a metric_type but it does not define an object_type
- type just could have these allowed values:
  - histogram
  - aggregate_metric_double
  - long
  - integer
  - short
  - byte
  - double
  - float
  - half_float
  - scaled_float
  - unsigned_long

This PR allows field definitions like this:

- name: prometheus.*.value
  type: object
  object_type: double
  object_type_mapping_type: "*"
  metric_type: gauge
  descriptoin: Prometheus gauge metric

Previously, it was failing with this error:

file "/home/mariorodriguez/Coding/work/integrations/build/packages/aws-1.46.6.zip/data_stream/cloudwatch_metrics/fields/package-fields.yml" is invalid: field 1.type: 1.type must be one of the following: "histogram", "aggregate_metric_double", "long", "integer", "short", "byte", "double", "float", "half_float", "scaled_float", "unsigned_long"

Why is it important?

This new validation will check that the types of the fields will be the right ones when metric_type is set for a field (gauge or counter). Not all types are valid when this is enabled. Based on this doc

Checklist

I have added test packages to test/packages that prove my change is effective.
I have added an entry in spec/changelog.yml.
Added JSON Patches for previous versions.

Related issues

Relates [Fleet] TSDB: need metric_type mapping for dynamic templates kibana#155004

elasticmachine · 2023-05-17T11:44:38Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-05-17T11:52:24.571+0000
Duration: 8 min 49 sec

Test stats 🧪

Test	Results
Failed	0
Passed	842
Skipped	0
Total	842

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-05-17T11:44:46Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`8/8`)	💚
Files	68.966% (`20/29`)	👍
Classes	73.171% (`30/41`)	👍
Methods	56.154% (`73/130`)	👍
Lines	43.029% (`679/1578`)	👍
Conditionals	100.0% (`0/0`)	💚

mrodm · 2023-07-13T11:50:37Z

test integrations

elasticmachine · 2023-07-13T11:53:26Z

Created or updated PR in integrations repostiory to test this vesrion. Check elastic/integrations#6948

mrodm · 2023-07-13T14:48:19Z

Latest run with integrations, the following packages have failed their check:

azure_application_insights
azure_billing
hashicorp_vault

These are the errors for each packages:

azure_application_insights

[2023-07-13T12:42:38.477Z] Error: checking package failed: linting package failed: found 4 validation errors:
[2023-07-13T12:42:38.477Z]    1. file "/var/lib/jenkins/workspace/est-manager_integrations_PR-6948/src/github.com/elastic/integrations/packages/azure_application_insights/data_stream/app_insights/fields/package-fields.yml" is invalid: field 0.fields.1.fields.4.type: 0.fields.1.fields.4.type must be one of the following: "object"
[2023-07-13T12:42:38.477Z]    2. file "/var/lib/jenkins/workspace/est-manager_integrations_PR-6948/src/github.com/elastic/integrations/packages/azure_application_insights/data_stream/app_insights/fields/package-fields.yml" is invalid: field 0.fields.5.type: 0.fields.5.type must be one of the following: "object"
[2023-07-13T12:42:38.477Z]    3. file "/var/lib/jenkins/workspace/est-manager_integrations_PR-6948/src/github.com/elastic/integrations/packages/azure_application_insights/data_stream/app_state/fields/package-fields.yml" is invalid: field 0.fields.1.fields.4.type: 0.fields.1.fields.4.type must be one of the following: "object"
[2023-07-13T12:42:38.477Z]    4. file "/var/lib/jenkins/workspace/est-manager_integrations_PR-6948/src/github.com/elastic/integrations/packages/azure_application_insights/data_stream/app_state/fields/package-fields.yml" is invalid: field 0.fields.5.type: 0.fields.5.type must be one of the following: "object"

azure_billing

[2023-07-13T12:42:37.010Z] Error: checking package failed: linting package failed: found 2 validation errors:
[2023-07-13T12:42:37.010Z]    1. file "/var/lib/jenkins/workspace/est-manager_integrations_PR-6948/src/github.com/elastic/integrations/packages/azure_billing/data_stream/billing/fields/package-fields.yml" is invalid: field 0.fields.1.fields.4.type: 0.fields.1.fields.4.type must be one of the following: "object"
[2023-07-13T12:42:37.010Z]    2. file "/var/lib/jenkins/workspace/est-manager_integrations_PR-6948/src/github.com/elastic/integrations/packages/azure_billing/data_stream/billing/fields/package-fields.yml" is invalid: field 0.fields.5.type: 0.fields.5.type must be one of the following: "object"

hashicorp_vault

[2023-07-13T12:40:24.777Z] Error: checking package failed: linting package failed: found 1 validation error:
[2023-07-13T12:40:24.777Z]    1. file "/var/lib/jenkins/workspace/est-manager_integrations_PR-6948/src/github.com/elastic/integrations/packages/hashicorp_vault/data_stream/audit/fields/fields.yml" is invalid: field 0.fields.0.fields.0.fields.10.type: 0.fields.0.fields.0.fields.10.type must be one of the following: "object"

elasticmachine · 2023-07-17T13:25:47Z

💚 Build Succeeded

Buildkite Build
Commit: 7f62361

History

💚 Build #57 succeeded d922d07

cc @mrodm

mrodm · 2023-07-17T13:42:58Z

test integrations

elasticmachine · 2023-07-17T13:46:54Z

Created or updated PR in integrations repostiory to test this vesrion. Check elastic/integrations#6948

mrodm · 2023-07-18T10:47:02Z

Adding the JSON Patches to remove these validation in previous versions , packages failing running "test integrations" are not due to elastic-package check -v command failing. And those 3 packages (azure_application_insights, azure_billing and hashicorp_vault) are run successfully.

https://fleet-ci.elastic.co/blue/organizations/jenkins/Ingest-manager%2Fintegrations/detail/PR-6948/3/
https://fleet-ci.elastic.co/blue/organizations/jenkins/Ingest-manager%2Fintegrations/detail/PR-6948/4/

jsoriano

👍

tetianakravchenko · 2023-08-07T14:03:30Z

Hey @mrodm when is it planned to get this changes available in elastic-package?

jsoriano · 2023-08-07T14:54:16Z

Hey @mrodm when is it planned to get this changes available in elastic-package?

We will prepare releases of package-spec and elastic-package today or tomorrow.

Add new checks to manage object_type and metric_type fields

0233791

mrodm self-assigned this May 17, 2023

Add changelog entry

3bfe8a5

mrodm mentioned this pull request May 17, 2023

Update elastic-package - new validation check object_type metric_type elastic/integrations#6238

Closed

Merge remote upstream/main into add_validation_object_type_metric_type

d922d07

elasticmachine mentioned this pull request Jul 13, 2023

Test branch package-spec#527 - DO NOT MERGE elastic/integrations#6948

Closed

This was referenced Jul 17, 2023

Fix mappings of flattened fields in azure billing and application insights packages elastic/integrations#6975

Merged

[hashicorp_vault] Fix mappings of granting policies elastic/integrations#6976

Closed

Add JSON patches to remove ifs

7f62361

mrodm changed the title ~~Add new checks to manage object_type and metric_type fields~~ Add new validations to manage object_type and metric_type fields Jul 18, 2023

mrodm marked this pull request as ready for review July 18, 2023 13:41

mrodm requested a review from a team as a code owner July 18, 2023 13:41

jsoriano approved these changes Jul 18, 2023

View reviewed changes

mrodm merged commit 7990807 into elastic:main Jul 18, 2023
3 checks passed

mrodm deleted the add_validation_object_type_metric_type branch July 18, 2023 16:23

This was referenced Aug 7, 2023

[Azure][compute_vm] Add dimensions, metric_type to the compute_vm datastream; remove dupl… elastic/integrations#7293

Merged

[Fleet] TSDB: need metric_type mapping for dynamic templates elastic/kibana#155004

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new validations to manage object_type and metric_type fields #527

Add new validations to manage object_type and metric_type fields #527

mrodm commented May 17, 2023 •

edited

elasticmachine commented May 17, 2023 •

edited

Build stats

Test stats 🧪

elasticmachine commented May 17, 2023

mrodm commented Jul 13, 2023

elasticmachine commented Jul 13, 2023

mrodm commented Jul 13, 2023

elasticmachine commented Jul 17, 2023

mrodm commented Jul 17, 2023

elasticmachine commented Jul 17, 2023

mrodm commented Jul 18, 2023 •

edited

jsoriano left a comment

tetianakravchenko commented Aug 7, 2023

jsoriano commented Aug 7, 2023

Add new validations to manage object_type and metric_type fields #527

Add new validations to manage object_type and metric_type fields #527

Conversation

mrodm commented May 17, 2023 • edited

What does this PR do?

Why is it important?

Checklist

Related issues

elasticmachine commented May 17, 2023 • edited

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented May 17, 2023

🌐 Coverage report

mrodm commented Jul 13, 2023

elasticmachine commented Jul 13, 2023

mrodm commented Jul 13, 2023

elasticmachine commented Jul 17, 2023

💚 Build Succeeded

History

mrodm commented Jul 17, 2023

elasticmachine commented Jul 17, 2023

mrodm commented Jul 18, 2023 • edited

jsoriano left a comment

Choose a reason for hiding this comment

tetianakravchenko commented Aug 7, 2023

jsoriano commented Aug 7, 2023

mrodm commented May 17, 2023 •

edited

elasticmachine commented May 17, 2023 •

edited

mrodm commented Jul 18, 2023 •

edited