Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Hashicorp Vault] Added dimensions mapping and added missing label fields to metrics datastream #8232

Merged
merged 6 commits into from
Oct 25, 2023

Conversation

agithomas
Copy link
Contributor

@agithomas agithomas commented Oct 18, 2023

  • Enhancement

Proposed commit message

  • WHAT: Added dimensions mapping to metrics datastream. Missing fields added to annotate as dimensions.
  • WHY : To benefit from the TSDB enablement of the package.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

  • elastic-package build
  • elastic-package stack up -v -d --services package-registry
  • TSDB-migration-test-kit ran to check record count validation. Test results shared in the comments.

Related issues

Screenshots

No related dashboards found

@agithomas agithomas changed the title Added dimensions mapping to metrics datastream of vault [Hashicorp Vault] Added dimensions mapping to metrics datastream of vault Oct 18, 2023
@agithomas agithomas changed the title [Hashicorp Vault] Added dimensions mapping to metrics datastream of vault [Hashicorp Vault] Added dimensions mapping to metrics datastream Oct 18, 2023
@agithomas
Copy link
Contributor Author

TSDB Migration Tool Kit Execution Run Details - No missing data observed.

Run output
You're testing with version 8.10.0.

Testing data stream metrics-hashicorp_vault.metrics-default.
Index being used for the documents is .ds-metrics-hashicorp_vault.metrics-default-2023.10.18-000001.
Index being used for the settings and mappings is .ds-metrics-hashicorp_vault.metrics-default-2023.10.18-000002.

The time series fields for the TSDB index are: 
	- dimension (27 fields):
		- agent.id
		- cloud.account.id
		- cloud.availability_zone
		- cloud.instance.id
		- cloud.provider
		- cloud.region
		- container.id
		- host.name
		- labels.auth_method
		- labels.cluster
		- labels.creation_ttl
		- labels.expiring
		- labels.gauge
		- labels.host
		- labels.instance
		- labels.job
		- labels.local
		- labels.mount_point
		- labels.namespace
		- labels.policy
		- labels.quantile
		- labels.queue_id
		- labels.term
		- labels.token_type
		- labels.type
		- labels.version
		- service.address
	- counter (74 fields):
		- hashicorp_vault.metrics.go_gc_duration_seconds_count.counter
		- hashicorp_vault.metrics.go_gc_duration_seconds_sum.counter
		- hashicorp_vault.metrics.go_memstats_alloc_bytes_total.counter
		- hashicorp_vault.metrics.go_memstats_frees_total.counter
		- hashicorp_vault.metrics.go_memstats_lookups_total.counter
		- hashicorp_vault.metrics.go_memstats_mallocs_total.counter
		- hashicorp_vault.metrics.process_cpu_seconds_total.counter
		- hashicorp_vault.metrics.vault_barrier_delete_count.counter
		- hashicorp_vault.metrics.vault_barrier_delete_sum.counter
		- hashicorp_vault.metrics.vault_barrier_estimated_encryptions.counter
		- hashicorp_vault.metrics.vault_barrier_get_count.counter
		- hashicorp_vault.metrics.vault_barrier_get_sum.counter
		- hashicorp_vault.metrics.vault_barrier_list_count.counter
		- hashicorp_vault.metrics.vault_barrier_list_sum.counter
		- hashicorp_vault.metrics.vault_barrier_put_count.counter
		- hashicorp_vault.metrics.vault_barrier_put_sum.counter
		- hashicorp_vault.metrics.vault_cache_hit.counter
		- hashicorp_vault.metrics.vault_cache_write.counter
		- hashicorp_vault.metrics.vault_core_activity_segment_write_count.counter
		- hashicorp_vault.metrics.vault_core_activity_segment_write_sum.counter
		- hashicorp_vault.metrics.vault_metrics_collection_count.counter
		- hashicorp_vault.metrics.vault_metrics_collection_interval_count.counter
		- hashicorp_vault.metrics.vault_metrics_collection_interval_sum.counter
		- hashicorp_vault.metrics.vault_metrics_collection_sum.counter
		- hashicorp_vault.metrics.vault_rollback_attempt_auth_approle__count.counter
		- hashicorp_vault.metrics.vault_rollback_attempt_auth_approle__sum.counter
		- hashicorp_vault.metrics.vault_rollback_attempt_auth_my_login__count.counter
		- hashicorp_vault.metrics.vault_rollback_attempt_auth_my_login__sum.counter
		- hashicorp_vault.metrics.vault_rollback_attempt_auth_token__count.counter
		- hashicorp_vault.metrics.vault_rollback_attempt_auth_token__sum.counter
		- hashicorp_vault.metrics.vault_rollback_attempt_auth_userpass__count.counter
		- hashicorp_vault.metrics.vault_rollback_attempt_auth_userpass__sum.counter
		- hashicorp_vault.metrics.vault_rollback_attempt_aws__count.counter
		- hashicorp_vault.metrics.vault_rollback_attempt_aws__sum.counter
		- hashicorp_vault.metrics.vault_rollback_attempt_cubbyhole__count.counter
		- hashicorp_vault.metrics.vault_rollback_attempt_cubbyhole__sum.counter
		- hashicorp_vault.metrics.vault_rollback_attempt_db_prod__count.counter
		- hashicorp_vault.metrics.vault_rollback_attempt_db_prod__sum.counter
		- hashicorp_vault.metrics.vault_rollback_attempt_identity__count.counter
		- hashicorp_vault.metrics.vault_rollback_attempt_identity__sum.counter
		- hashicorp_vault.metrics.vault_rollback_attempt_kv__count.counter
		- hashicorp_vault.metrics.vault_rollback_attempt_kv__sum.counter
		- hashicorp_vault.metrics.vault_rollback_attempt_secret__count.counter
		- hashicorp_vault.metrics.vault_rollback_attempt_secret__sum.counter
		- hashicorp_vault.metrics.vault_rollback_attempt_sys__count.counter
		- hashicorp_vault.metrics.vault_rollback_attempt_sys__sum.counter
		- hashicorp_vault.metrics.vault_route_list_kv__count.counter
		- hashicorp_vault.metrics.vault_route_list_kv__sum.counter
		- hashicorp_vault.metrics.vault_route_list_secret__count.counter
		- hashicorp_vault.metrics.vault_route_list_secret__sum.counter
		- hashicorp_vault.metrics.vault_route_rollback_auth_approle__count.counter
		- hashicorp_vault.metrics.vault_route_rollback_auth_approle__sum.counter
		- hashicorp_vault.metrics.vault_route_rollback_auth_my_login__count.counter
		- hashicorp_vault.metrics.vault_route_rollback_auth_my_login__sum.counter
		- hashicorp_vault.metrics.vault_route_rollback_auth_token__count.counter
		- hashicorp_vault.metrics.vault_route_rollback_auth_token__sum.counter
		- hashicorp_vault.metrics.vault_route_rollback_auth_userpass__count.counter
		- hashicorp_vault.metrics.vault_route_rollback_auth_userpass__sum.counter
		- hashicorp_vault.metrics.vault_route_rollback_aws__count.counter
		- hashicorp_vault.metrics.vault_route_rollback_aws__sum.counter
		- hashicorp_vault.metrics.vault_route_rollback_cubbyhole__count.counter
		- hashicorp_vault.metrics.vault_route_rollback_cubbyhole__sum.counter
		- hashicorp_vault.metrics.vault_route_rollback_db_prod__count.counter
		- hashicorp_vault.metrics.vault_route_rollback_db_prod__sum.counter
		- hashicorp_vault.metrics.vault_route_rollback_identity__count.counter
		- hashicorp_vault.metrics.vault_route_rollback_identity__sum.counter
		- hashicorp_vault.metrics.vault_route_rollback_kv__count.counter
		- hashicorp_vault.metrics.vault_route_rollback_kv__sum.counter
		- hashicorp_vault.metrics.vault_route_rollback_secret__count.counter
		- hashicorp_vault.metrics.vault_route_rollback_secret__sum.counter
		- hashicorp_vault.metrics.vault_route_rollback_sys__count.counter
		- hashicorp_vault.metrics.vault_route_rollback_sys__sum.counter
		- hashicorp_vault.metrics.vault_runtime_gc_pause_ns_count.counter
		- hashicorp_vault.metrics.vault_runtime_gc_pause_ns_sum.counter
	- gauge (163 fields):
		- hashicorp_vault.metrics.go_gc_duration_seconds.value
		- hashicorp_vault.metrics.go_gc_duration_seconds_count.rate
		- hashicorp_vault.metrics.go_gc_duration_seconds_sum.rate
		- hashicorp_vault.metrics.go_goroutines.value
		- hashicorp_vault.metrics.go_info.value
		- hashicorp_vault.metrics.go_memstats_alloc_bytes.value
		- hashicorp_vault.metrics.go_memstats_alloc_bytes_total.rate
		- hashicorp_vault.metrics.go_memstats_buck_hash_sys_bytes.value
		- hashicorp_vault.metrics.go_memstats_frees_total.rate
		- hashicorp_vault.metrics.go_memstats_gc_cpu_fraction.value
		- hashicorp_vault.metrics.go_memstats_gc_sys_bytes.value
		- hashicorp_vault.metrics.go_memstats_heap_alloc_bytes.value
		- hashicorp_vault.metrics.go_memstats_heap_idle_bytes.value
		- hashicorp_vault.metrics.go_memstats_heap_inuse_bytes.value
		- hashicorp_vault.metrics.go_memstats_heap_objects.value
		- hashicorp_vault.metrics.go_memstats_heap_released_bytes.value
		- hashicorp_vault.metrics.go_memstats_heap_sys_bytes.value
		- hashicorp_vault.metrics.go_memstats_last_gc_time_seconds.value
		- hashicorp_vault.metrics.go_memstats_lookups_total.rate
		- hashicorp_vault.metrics.go_memstats_mallocs_total.rate
		- hashicorp_vault.metrics.go_memstats_mcache_inuse_bytes.value
		- hashicorp_vault.metrics.go_memstats_mcache_sys_bytes.value
		- hashicorp_vault.metrics.go_memstats_mspan_inuse_bytes.value
		- hashicorp_vault.metrics.go_memstats_mspan_sys_bytes.value
		- hashicorp_vault.metrics.go_memstats_next_gc_bytes.value
		- hashicorp_vault.metrics.go_memstats_other_sys_bytes.value
		- hashicorp_vault.metrics.go_memstats_stack_inuse_bytes.value
		- hashicorp_vault.metrics.go_memstats_stack_sys_bytes.value
		- hashicorp_vault.metrics.go_memstats_sys_bytes.value
		- hashicorp_vault.metrics.go_threads.value
		- hashicorp_vault.metrics.process_cpu_seconds_total.rate
		- hashicorp_vault.metrics.process_max_fds.value
		- hashicorp_vault.metrics.process_open_fds.value
		- hashicorp_vault.metrics.process_resident_memory_bytes.value
		- hashicorp_vault.metrics.process_start_time_seconds.value
		- hashicorp_vault.metrics.process_virtual_memory_bytes.value
		- hashicorp_vault.metrics.process_virtual_memory_max_bytes.value
		- hashicorp_vault.metrics.up.value
		- hashicorp_vault.metrics.vault_barrier_delete_count.rate
		- hashicorp_vault.metrics.vault_barrier_delete_sum.rate
		- hashicorp_vault.metrics.vault_barrier_estimated_encryptions.rate
		- hashicorp_vault.metrics.vault_barrier_get.value
		- hashicorp_vault.metrics.vault_barrier_get_count.rate
		- hashicorp_vault.metrics.vault_barrier_get_sum.rate
		- hashicorp_vault.metrics.vault_barrier_list.value
		- hashicorp_vault.metrics.vault_barrier_list_count.rate
		- hashicorp_vault.metrics.vault_barrier_list_sum.rate
		- hashicorp_vault.metrics.vault_barrier_put_count.rate
		- hashicorp_vault.metrics.vault_barrier_put_sum.rate
		- hashicorp_vault.metrics.vault_cache_hit.rate
		- hashicorp_vault.metrics.vault_cache_write.rate
		- hashicorp_vault.metrics.vault_core_active.value
		- hashicorp_vault.metrics.vault_core_activity_segment_write_count.rate
		- hashicorp_vault.metrics.vault_core_activity_segment_write_sum.rate
		- hashicorp_vault.metrics.vault_core_in_flight_requests.value
		- hashicorp_vault.metrics.vault_core_mount_table_num_entries.value
		- hashicorp_vault.metrics.vault_core_mount_table_size.value
		- hashicorp_vault.metrics.vault_core_performance_standby.value
		- hashicorp_vault.metrics.vault_core_replication_dr_primary.value
		- hashicorp_vault.metrics.vault_core_replication_dr_secondary.value
		- hashicorp_vault.metrics.vault_core_replication_performance_primary.value
		- hashicorp_vault.metrics.vault_core_replication_performance_secondary.value
		- hashicorp_vault.metrics.vault_core_unsealed.value
		- hashicorp_vault.metrics.vault_expire_leases_by_expiration.value
		- hashicorp_vault.metrics.vault_expire_num_irrevocable_leases.value
		- hashicorp_vault.metrics.vault_expire_num_leases.value
		- hashicorp_vault.metrics.vault_identity_entity_alias_count.value
		- hashicorp_vault.metrics.vault_identity_entity_count.value
		- hashicorp_vault.metrics.vault_identity_num_entities.value
		- hashicorp_vault.metrics.vault_metrics_collection.value
		- hashicorp_vault.metrics.vault_metrics_collection_count.rate
		- hashicorp_vault.metrics.vault_metrics_collection_interval.value
		- hashicorp_vault.metrics.vault_metrics_collection_interval_count.rate
		- hashicorp_vault.metrics.vault_metrics_collection_interval_sum.rate
		- hashicorp_vault.metrics.vault_metrics_collection_sum.rate
		- hashicorp_vault.metrics.vault_rollback_attempt_auth_approle_.value
		- hashicorp_vault.metrics.vault_rollback_attempt_auth_approle__count.rate
		- hashicorp_vault.metrics.vault_rollback_attempt_auth_approle__sum.rate
		- hashicorp_vault.metrics.vault_rollback_attempt_auth_my_login_.value
		- hashicorp_vault.metrics.vault_rollback_attempt_auth_my_login__count.rate
		- hashicorp_vault.metrics.vault_rollback_attempt_auth_my_login__sum.rate
		- hashicorp_vault.metrics.vault_rollback_attempt_auth_token_.value
		- hashicorp_vault.metrics.vault_rollback_attempt_auth_token__count.rate
		- hashicorp_vault.metrics.vault_rollback_attempt_auth_token__sum.rate
		- hashicorp_vault.metrics.vault_rollback_attempt_auth_userpass_.value
		- hashicorp_vault.metrics.vault_rollback_attempt_auth_userpass__count.rate
		- hashicorp_vault.metrics.vault_rollback_attempt_auth_userpass__sum.rate
		- hashicorp_vault.metrics.vault_rollback_attempt_aws_.value
		- hashicorp_vault.metrics.vault_rollback_attempt_aws__count.rate
		- hashicorp_vault.metrics.vault_rollback_attempt_aws__sum.rate
		- hashicorp_vault.metrics.vault_rollback_attempt_cubbyhole_.value
		- hashicorp_vault.metrics.vault_rollback_attempt_cubbyhole__count.rate
		- hashicorp_vault.metrics.vault_rollback_attempt_cubbyhole__sum.rate
		- hashicorp_vault.metrics.vault_rollback_attempt_db_prod_.value
		- hashicorp_vault.metrics.vault_rollback_attempt_db_prod__count.rate
		- hashicorp_vault.metrics.vault_rollback_attempt_db_prod__sum.rate
		- hashicorp_vault.metrics.vault_rollback_attempt_identity_.value
		- hashicorp_vault.metrics.vault_rollback_attempt_identity__count.rate
		- hashicorp_vault.metrics.vault_rollback_attempt_identity__sum.rate
		- hashicorp_vault.metrics.vault_rollback_attempt_kv_.value
		- hashicorp_vault.metrics.vault_rollback_attempt_kv__count.rate
		- hashicorp_vault.metrics.vault_rollback_attempt_kv__sum.rate
		- hashicorp_vault.metrics.vault_rollback_attempt_secret_.value
		- hashicorp_vault.metrics.vault_rollback_attempt_secret__count.rate
		- hashicorp_vault.metrics.vault_rollback_attempt_secret__sum.rate
		- hashicorp_vault.metrics.vault_rollback_attempt_sys_.value
		- hashicorp_vault.metrics.vault_rollback_attempt_sys__count.rate
		- hashicorp_vault.metrics.vault_rollback_attempt_sys__sum.rate
		- hashicorp_vault.metrics.vault_route_list_kv_.value
		- hashicorp_vault.metrics.vault_route_list_kv__count.rate
		- hashicorp_vault.metrics.vault_route_list_kv__sum.rate
		- hashicorp_vault.metrics.vault_route_list_secret_.value
		- hashicorp_vault.metrics.vault_route_list_secret__count.rate
		- hashicorp_vault.metrics.vault_route_list_secret__sum.rate
		- hashicorp_vault.metrics.vault_route_rollback_auth_approle_.value
		- hashicorp_vault.metrics.vault_route_rollback_auth_approle__count.rate
		- hashicorp_vault.metrics.vault_route_rollback_auth_approle__sum.rate
		- hashicorp_vault.metrics.vault_route_rollback_auth_my_login_.value
		- hashicorp_vault.metrics.vault_route_rollback_auth_my_login__count.rate
		- hashicorp_vault.metrics.vault_route_rollback_auth_my_login__sum.rate
		- hashicorp_vault.metrics.vault_route_rollback_auth_token_.value
		- hashicorp_vault.metrics.vault_route_rollback_auth_token__count.rate
		- hashicorp_vault.metrics.vault_route_rollback_auth_token__sum.rate
		- hashicorp_vault.metrics.vault_route_rollback_auth_userpass_.value
		- hashicorp_vault.metrics.vault_route_rollback_auth_userpass__count.rate
		- hashicorp_vault.metrics.vault_route_rollback_auth_userpass__sum.rate
		- hashicorp_vault.metrics.vault_route_rollback_aws_.value
		- hashicorp_vault.metrics.vault_route_rollback_aws__count.rate
		- hashicorp_vault.metrics.vault_route_rollback_aws__sum.rate
		- hashicorp_vault.metrics.vault_route_rollback_cubbyhole_.value
		- hashicorp_vault.metrics.vault_route_rollback_cubbyhole__count.rate
		- hashicorp_vault.metrics.vault_route_rollback_cubbyhole__sum.rate
		- hashicorp_vault.metrics.vault_route_rollback_db_prod_.value
		- hashicorp_vault.metrics.vault_route_rollback_db_prod__count.rate
		- hashicorp_vault.metrics.vault_route_rollback_db_prod__sum.rate
		- hashicorp_vault.metrics.vault_route_rollback_identity_.value
		- hashicorp_vault.metrics.vault_route_rollback_identity__count.rate
		- hashicorp_vault.metrics.vault_route_rollback_identity__sum.rate
		- hashicorp_vault.metrics.vault_route_rollback_kv_.value
		- hashicorp_vault.metrics.vault_route_rollback_kv__count.rate
		- hashicorp_vault.metrics.vault_route_rollback_kv__sum.rate
		- hashicorp_vault.metrics.vault_route_rollback_secret_.value
		- hashicorp_vault.metrics.vault_route_rollback_secret__count.rate
		- hashicorp_vault.metrics.vault_route_rollback_secret__sum.rate
		- hashicorp_vault.metrics.vault_route_rollback_sys_.value
		- hashicorp_vault.metrics.vault_route_rollback_sys__count.rate
		- hashicorp_vault.metrics.vault_route_rollback_sys__sum.rate
		- hashicorp_vault.metrics.vault_runtime_alloc_bytes.value
		- hashicorp_vault.metrics.vault_runtime_free_count.value
		- hashicorp_vault.metrics.vault_runtime_gc_pause_ns.value
		- hashicorp_vault.metrics.vault_runtime_gc_pause_ns_count.rate
		- hashicorp_vault.metrics.vault_runtime_gc_pause_ns_sum.rate
		- hashicorp_vault.metrics.vault_runtime_heap_objects.value
		- hashicorp_vault.metrics.vault_runtime_malloc_count.value
		- hashicorp_vault.metrics.vault_runtime_num_goroutines.value
		- hashicorp_vault.metrics.vault_runtime_sys_bytes.value
		- hashicorp_vault.metrics.vault_runtime_total_gc_pause_ns.value
		- hashicorp_vault.metrics.vault_runtime_total_gc_runs.value
		- hashicorp_vault.metrics.vault_secret_kv_count.value
		- hashicorp_vault.metrics.vault_token_count.value
		- hashicorp_vault.metrics.vault_token_count_by_auth.value
		- hashicorp_vault.metrics.vault_token_count_by_policy.value
		- hashicorp_vault.metrics.vault_token_count_by_ttl.value
	- routing_path (27 fields):
		- agent.id
		- cloud.account.id
		- cloud.availability_zone
		- cloud.instance.id
		- cloud.provider
		- cloud.region
		- container.id
		- host.name
		- labels.auth_method
		- labels.cluster
		- labels.creation_ttl
		- labels.expiring
		- labels.gauge
		- labels.host
		- labels.instance
		- labels.job
		- labels.local
		- labels.mount_point
		- labels.namespace
		- labels.policy
		- labels.quantile
		- labels.queue_id
		- labels.term
		- labels.token_type
		- labels.type
		- labels.version
		- service.address

Index tsdb-index-enabled successfully created.

Copying documents from .ds-metrics-hashicorp_vault.metrics-default-2023.10.18-000001 to tsdb-index-enabled...
All 3093 documents taken from index .ds-metrics-hashicorp_vault.metrics-default-2023.10.18-000001 were successfully placed to index tsdb-index-enabled.

@agithomas agithomas marked this pull request as ready for review October 18, 2023 08:41
@agithomas agithomas requested a review from a team as a code owner October 18, 2023 08:41
@elasticmachine
Copy link

elasticmachine commented Oct 18, 2023

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-10-25T10:51:31.398+0000

  • Duration: 18 min 48 sec

Test stats 🧪

Test Results
Failed 0
Passed 18
Skipped 0
Total 18

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link

elasticmachine commented Oct 18, 2023

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (2/2) 💚
Files 100.0% (3/3) 💚 8.333
Classes 100.0% (3/3) 💚 8.333
Methods 92.593% (25/27) 👍 7.436
Lines 87.85% (188/214) 👎 -3.7
Conditionals 100.0% (0/0) 💚

Copy link
Contributor

@ritalwar ritalwar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@agithomas agithomas changed the title [Hashicorp Vault] Added dimensions mapping to metrics datastream [Hashicorp Vault] Added dimensions mapping and added missing label fields to metrics datastream Oct 19, 2023
Copy link
Contributor

@kcreddy kcreddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving since its required as a code owner for SEI.

@agithomas agithomas merged commit 51a09b1 into elastic:main Oct 25, 2023
4 checks passed
@agithomas agithomas mentioned this pull request Oct 25, 2023
5 tasks
@elasticmachine
Copy link

Package hashicorp_vault - 1.20.0 containing this change is available at https://epr.elastic.co/search?package=hashicorp_vault

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants