Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Kubernetes] Add state namespace datastream #8329

Merged
merged 10 commits into from
Oct 31, 2023

Conversation

constanca-m
Copy link
Contributor

@constanca-m constanca-m commented Oct 30, 2023

What

Add state namespace data stream.

The metricset was added in version 8.11.0.

Dimensions for TSDB:

  • kubernetes.namespace is unique, so it is a dimension.
  • agent.id is a common dimension among data streams.
  • orchestrator.cluster.url is a dimension for all kubernetes data streams.
Results from running TSDB testing kit:
The time series fields for the TSDB index are: 
	- dimension (3 fields):
		- agent.id
		- kubernetes.namespace
		- orchestrator.cluster.url
	- gauge (1 fields):
		- kubernetes.state_namespace.created.sec
	- routing_path (3 fields):
		- agent.id
		- kubernetes.namespace
		- orchestrator.cluster.url

Index tsdb-index-enabled successfully created.

Copying documents from .ds-metrics-kubernetes.state_namespace-default-2023.10.30-000001 to tsdb-index-enabled...
All 210 documents taken from index .ds-metrics-kubernetes.state_namespace-default-2023.10.30-000001 were successfully placed to index tsdb-index-enabled.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

  1. Clone this branch.
  2. Build the package.
  3. Update the service registry.
  4. Add Kubernetes to a policy and check the new data stream.

Related issues

Relates to elastic/elastic-agent#3100.

Screenshots

In Discover:
image

In the integration itself:
image

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
@constanca-m constanca-m added Integration:kubernetes Kubernetes Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team [elastic/obs-cloudnative-monitoring] labels Oct 30, 2023
@constanca-m constanca-m self-assigned this Oct 30, 2023
@constanca-m constanca-m requested a review from a team as a code owner October 30, 2023 10:57
Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
@elasticmachine
Copy link

elasticmachine commented Oct 30, 2023

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-10-31T16:41:46.770+0000

  • Duration: 77 min 5 sec

Test stats 🧪

Test Results
Failed 0
Passed 97
Skipped 0
Total 97

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link

elasticmachine commented Oct 30, 2023

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (1/1) 💚
Files 100.0% (1/1) 💚
Classes 100.0% (1/1) 💚
Methods 96.386% (80/83)
Lines 100.0% (22/22) 💚
Conditionals 100.0% (0/0) 💚

constanca-m and others added 2 commits October 31, 2023 09:19
Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
@gizas
Copy link
Contributor

gizas commented Oct 31, 2023

Another idea is to consider if we need to add this info of namespace in our dashboards. To add it in Overview or in Pods?

@MichaelKatsoulis , @tetianakravchenko , @ChrsMark suggestions?

Kubernetes namespace metrics.

fields:
- name: created.sec
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should here be defined metric_type ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since TSDB won't be enabled, then the metric type does not need to be defined.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would not be enabled?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The advantages of TSDB are related to the metric types. This data stream only has one metric type, so there wouldn't be any. We followed a similar approach for other data streams that only had one metric, such as AWS Billing or GCP Billing @gizas

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After TSDB meeting discussion, we are enabling TSDB in all metric data streams after all. I updated the commits and will update the description now.

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
Enable TSDB.

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
@constanca-m constanca-m merged commit ca6dc49 into elastic:main Oct 31, 2023
4 checks passed
@constanca-m constanca-m deleted the add-state-namespace branch October 31, 2023 19:00
@elasticmachine
Copy link

Package kubernetes - 1.52.0 containing this change is available at https://epr.elastic.co/search?package=kubernetes

@elasticmachine
Copy link

Package kubernetes - 1.55.0 containing this change is available at https://epr.elastic.co/search?package=kubernetes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Integration:kubernetes Kubernetes Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team [elastic/obs-cloudnative-monitoring]
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants