Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Cluster filter on Kubernetes Overview dashboard #1413

Merged
merged 4 commits into from
Aug 4, 2021
Merged

Add Cluster filter on Kubernetes Overview dashboard #1413

merged 4 commits into from
Aug 4, 2021

Conversation

akshay-saraswat
Copy link
Contributor

What does this PR do?

Added a Cluster name selector on the [Metrics Kubernetes] Overview dashboard. This selector will be used as a filter for the dashboard. Also, added a blurb for this selector on the Kubernetes package guide to explain how this selector works and where it sources the data from.

Why is it important?

We have added a new Cluster name field in the Kubernetes package. This field will not be useful unless users could discover it in some form
in UI.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have added an entry to my package's changelog.yml file.

Screen Shot 2021-07-30 at 9 49 11 AM

@akshay-saraswat akshay-saraswat added bug Something isn't working, use only for issues Team:Integrations Label for the Integrations team Integration:kubernetes Kubernetes 7.14 candidate labels Jul 30, 2021
@elasticmachine
Copy link

Pinging @elastic/integrations (Team:Integrations)

@elasticmachine
Copy link

elasticmachine commented Jul 30, 2021

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-07-30T15:09:27.001+0000

  • Duration: 7 min 48 sec

  • Commit: 2582c4883dd580da3fa0cb7f2138ce5624d5ecc1

Trends 🧪

Image of Build Times

Steps errors 2

Expand to view the steps failures

Check integration: kubernetes
  • Took 0 min 0 sec . View more details on here
  • Description: ../../build/elastic-package check -v
Google Storage Download
  • Took 0 min 0 sec . View more details on here

Log output

Expand to view the last 100 lines of log output

[2021-07-30T15:15:19.527Z] go: downloading github.com/go-logr/logr v0.4.0
[2021-07-30T15:15:19.527Z] go: downloading k8s.io/utils v0.0.0-20201110183641-67b214c5f920
[2021-07-30T15:15:19.527Z] go: downloading k8s.io/kube-openapi v0.0.0-20210305001622-591a79e4bda7
[2021-07-30T15:15:19.527Z] go: downloading github.com/davecgh/go-spew v1.1.1
[2021-07-30T15:15:19.527Z] go: downloading golang.org/x/time v0.0.0-20210220033141-f8bda1e9f3ba
[2021-07-30T15:15:19.527Z] go: downloading github.com/google/btree v1.0.0
[2021-07-30T15:15:19.527Z] go: downloading gopkg.in/inf.v0 v0.9.1
[2021-07-30T15:15:19.527Z] go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
[2021-07-30T15:15:19.527Z] go: downloading google.golang.org/protobuf v1.26.0
[2021-07-30T15:15:19.788Z] go: downloading github.com/hashicorp/golang-lru v0.5.1
[2021-07-30T15:15:19.788Z] go: downloading github.com/go-openapi/spec v0.19.5
[2021-07-30T15:15:19.788Z] go: downloading github.com/exponent-io/jsonpath v0.0.0-20151013193312-d6023ce2651d
[2021-07-30T15:15:19.788Z] go: downloading k8s.io/component-base v0.21.0
[2021-07-30T15:15:19.788Z] go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510
[2021-07-30T15:15:19.788Z] go: downloading github.com/go-errors/errors v1.0.1
[2021-07-30T15:15:19.788Z] go: downloading github.com/MakeNowJust/heredoc v0.0.0-20170808103936-bb23615498cd
[2021-07-30T15:15:19.788Z] go: downloading github.com/russross/blackfriday v1.5.2
[2021-07-30T15:15:19.788Z] go: downloading github.com/go-stack/stack v1.8.0
[2021-07-30T15:15:19.788Z] go: downloading github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00
[2021-07-30T15:15:19.788Z] go: downloading github.com/stretchr/testify v1.7.0
[2021-07-30T15:15:19.788Z] go: downloading github.com/xlab/treeprint v0.0.0-20181112141820-a009c3971eca
[2021-07-30T15:15:19.788Z] go: downloading github.com/mitchellh/go-wordwrap v1.0.0
[2021-07-30T15:15:19.788Z] go: downloading github.com/moby/term v0.0.0-20201216013528-df9cb8a40635
[2021-07-30T15:15:19.788Z] go: downloading go.starlark.net v0.0.0-20200306205701-8dd3e2ee1dd5
[2021-07-30T15:15:19.788Z] go: downloading github.com/moby/spdystream v0.2.0
[2021-07-30T15:15:19.788Z] go: downloading github.com/pmezard/go-difflib v1.0.0
[2021-07-30T15:15:20.049Z] go: downloading github.com/go-openapi/swag v0.19.5
[2021-07-30T15:15:20.049Z] go: downloading github.com/go-openapi/jsonpointer v0.19.3
[2021-07-30T15:15:20.049Z] go: downloading github.com/go-openapi/jsonreference v0.19.3
[2021-07-30T15:15:20.049Z] go: downloading github.com/PuerkitoBio/purell v1.1.1
[2021-07-30T15:15:20.049Z] go: downloading github.com/mailru/easyjson v0.7.0
[2021-07-30T15:15:20.049Z] go: downloading github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578
[2021-07-30T15:16:06.824Z] Running in /var/lib/jenkins/workspace/est-manager_integrations_PR-1413/src/github.com/elastic/integrations/packages/kubernetes
[2021-07-30T15:16:07.112Z] + ../../build/elastic-package check -v
[2021-07-30T15:16:07.112Z] elastic-package has been installed.
[2021-07-30T15:16:07.112Z] 2021/07/30 15:16:06 DEBUG Enable verbose logging
[2021-07-30T15:16:07.112Z] 2021/07/30 15:16:06 DEBUG Distribution built without a version tag, can't determine release chronology. Please consider using official releases at https://github.com/elastic/elastic-package/releases
[2021-07-30T15:16:07.112Z] Format the package
[2021-07-30T15:16:07.112Z] Error: checking package failed: formatting the integration failed (path: /var/lib/jenkins/workspace/est-manager_integrations_PR-1413/src/github.com/elastic/integrations/packages/kubernetes, failFast: true): walking through the integration files failed: formatting file failed (path: /var/lib/jenkins/workspace/est-manager_integrations_PR-1413/src/github.com/elastic/integrations/packages/kubernetes/kibana/dashboard/kubernetes-AV4RGUqo5NkDleZmzKuZ.json): file is not formatted (path: /var/lib/jenkins/workspace/est-manager_integrations_PR-1413/src/github.com/elastic/integrations/packages/kubernetes/kibana/dashboard/kubernetes-AV4RGUqo5NkDleZmzKuZ.json)
[2021-07-30T15:16:07.140Z] Running in /var/lib/jenkins/workspace/est-manager_integrations_PR-1413/src/github.com/elastic/integrations
[2021-07-30T15:16:07.157Z] Archiving artifacts
[2021-07-30T15:16:07.479Z] ‘build/test-results/*.xml’ doesn’t match anything: ‘build’ exists but not ‘build/test-results/*.xml’
[2021-07-30T15:16:07.479Z] No artifacts found that match the file pattern "build/test-results/*.xml". Configuration error?
[2021-07-30T15:16:07.490Z] Recording test results
[2021-07-30T15:16:07.758Z] None of the test reports contained any result
[2021-07-30T15:16:07.758Z] [Checks API] No suitable checks publisher found.
[2021-07-30T15:16:08.067Z] + build/elastic-package stack dump -v --output build/elastic-stack-dump/latest/kubernetes
[2021-07-30T15:16:08.067Z] 2021/07/30 15:16:07  WARN CommitHash is undefined, in both /var/lib/jenkins/workspace/est-manager_integrations_PR-1413/.elastic-package/version and the compiled binary, config may be out of date.
[2021-07-30T15:16:08.067Z] 2021/07/30 15:16:07 DEBUG Enable verbose logging
[2021-07-30T15:16:08.067Z] 2021/07/30 15:16:07 DEBUG Distribution built without a version tag, can't determine release chronology. Please consider using official releases at https://github.com/elastic/elastic-package/releases
[2021-07-30T15:16:08.067Z] 2021/07/30 15:16:07 DEBUG Dump Elastic stack data
[2021-07-30T15:16:08.067Z] 2021/07/30 15:16:07 DEBUG Dump stack logs
[2021-07-30T15:16:08.067Z] 2021/07/30 15:16:07 DEBUG Recreate the output location (path: build/elastic-stack-dump/latest/kubernetes)
[2021-07-30T15:16:08.067Z] 2021/07/30 15:16:07 DEBUG Dump stack logs for elasticsearch
[2021-07-30T15:16:08.067Z] 2021/07/30 15:16:07 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/est-manager_integrations_PR-1413/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs elasticsearch
[2021-07-30T15:16:09.003Z] 2021/07/30 15:16:08 DEBUG Dump stack logs for elastic-agent
[2021-07-30T15:16:09.003Z] 2021/07/30 15:16:08 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/est-manager_integrations_PR-1413/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs elastic-agent
[2021-07-30T15:16:09.570Z] 2021/07/30 15:16:09 DEBUG Dump stack logs for fleet-server
[2021-07-30T15:16:09.570Z] 2021/07/30 15:16:09 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/est-manager_integrations_PR-1413/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs fleet-server
[2021-07-30T15:16:10.506Z] 2021/07/30 15:16:10 DEBUG Dump stack logs for kibana
[2021-07-30T15:16:10.506Z] 2021/07/30 15:16:10 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/est-manager_integrations_PR-1413/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs kibana
[2021-07-30T15:16:11.074Z] 2021/07/30 15:16:11 DEBUG Dump stack logs for package-registry
[2021-07-30T15:16:11.074Z] 2021/07/30 15:16:11 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/est-manager_integrations_PR-1413/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs package-registry
[2021-07-30T15:16:12.017Z] Path to stack dump: build/elastic-stack-dump/latest/kubernetes
[2021-07-30T15:16:12.017Z] Done
[2021-07-30T15:16:12.037Z] Archiving artifacts
[2021-07-30T15:16:12.412Z] + build/elastic-package stack down -v
[2021-07-30T15:16:12.412Z] 2021/07/30 15:16:12  WARN CommitHash is undefined, in both /var/lib/jenkins/workspace/est-manager_integrations_PR-1413/.elastic-package/version and the compiled binary, config may be out of date.
[2021-07-30T15:16:12.412Z] 2021/07/30 15:16:12 DEBUG Enable verbose logging
[2021-07-30T15:16:12.412Z] 2021/07/30 15:16:12 DEBUG Distribution built without a version tag, can't determine release chronology. Please consider using official releases at https://github.com/elastic/elastic-package/releases
[2021-07-30T15:16:12.412Z] Take down the Elastic stack
[2021-07-30T15:16:12.412Z] 2021/07/30 15:16:12 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/est-manager_integrations_PR-1413/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack down --volumes
[2021-07-30T15:16:12.980Z] The ELASTICSEARCH_IMAGE_REF variable is not set. Defaulting to a blank string.
[2021-07-30T15:16:12.980Z] The KIBANA_IMAGE_REF variable is not set. Defaulting to a blank string.
[2021-07-30T15:16:12.980Z] The STACK_PATH variable is not set. Defaulting to a blank string.
[2021-07-30T15:16:12.980Z] The PROFILE_NAME variable is not set. Defaulting to a blank string.
[2021-07-30T15:16:12.980Z] The ELASTIC_AGENT_IMAGE_REF variable is not set. Defaulting to a blank string.
[2021-07-30T15:16:12.980Z] Removing network elastic-package-stack_default
[2021-07-30T15:16:12.980Z] Network elastic-package-stack_default not found.
[2021-07-30T15:16:13.239Z] Done
[2021-07-30T15:16:13.542Z] + ls build/test-coverage/*.xml
[2021-07-30T15:16:13.542Z] ls: cannot access 'build/test-coverage/*.xml': No such file or directory
[2021-07-30T15:16:13.554Z] isCoverageReportPresent: coverage files not found, report won't be stashed
[2021-07-30T15:16:13.584Z] Failed in branch kubernetes
[2021-07-30T15:16:13.649Z] Running in /var/lib/jenkins/workspace/est-manager_integrations_PR-1413/src/github.com/elastic/integrations
[2021-07-30T15:16:13.669Z] Terminated
[2021-07-30T15:16:13.687Z] Terminated
[2021-07-30T15:16:13.697Z] Terminated
[2021-07-30T15:16:13.767Z] Error when executing always post condition:
[2021-07-30T15:16:13.767Z] java.lang.NullPointerException
[2021-07-30T15:16:13.767Z] 
[2021-07-30T15:16:14.088Z] Running on Jenkins in /var/lib/jenkins/workspace/est-manager_integrations_PR-1413
[2021-07-30T15:16:14.169Z] [INFO] getVaultSecret: Getting secrets
[2021-07-30T15:16:14.202Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-07-30T15:16:15.234Z] + chmod 755 generate-build-data.sh
[2021-07-30T15:16:15.234Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-1413/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-1413/runs/2 FAILURE 407967
[2021-07-30T15:16:15.234Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-1413/runs/2/steps/?limit=10000 -o steps-info.json
[2021-07-30T15:16:15.785Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-1413/runs/2/tests/?status=FAILED -o tests-errors.json
[2021-07-30T15:16:15.785Z] Retry 1/3 exited 22, retrying in 1 seconds...
[2021-07-30T15:16:17.128Z] Retry 2/3 exited 22, retrying in 2 seconds...

Copy link
Member

@ChrsMark ChrsMark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for submitting this. I left some comments. Also I think that we miss the updated screenshot for the docs.

packages/kubernetes/changelog.yml Outdated Show resolved Hide resolved
@@ -132,7 +132,7 @@ An example event for `event` looks as following:
| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | |
| host.os.version | Operating system version as a raw string. | keyword | |
| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | |
| kubernetes.annotations.* | Kubernetes annotations map | object | |
| kubernetes.annotations.\* | Kubernetes annotations map | object | |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these backslashes intentional?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes were generated by the elastic-package build. Don't know why these are required.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mtojek do you know if this is normal?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran elastic-package build after elastic-package format and don't see this change introduced anymore.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this escaping is needed to properly visualize them in READMEs (potential conflict with italics ).

@MichaelKatsoulis
Copy link
Contributor

@akshay-saraswat thanks for this. I think you should run elastic-package format to format the new dashboard. Also you need to update the version in the manifest.yml

@akshay-saraswat
Copy link
Contributor Author

@akshay-saraswat thanks for this. I think you should run elastic-package format to format the new dashboard. Also you need to update the version in the manifest.yml

@elasticmachine
Copy link

elasticmachine commented Aug 2, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-08-04T10:17:59.910+0000

  • Duration: 28 min 1 sec

  • Commit: cf7a056

Test stats 🧪

Test Results
Failed 0
Passed 112
Skipped 0
Total 112

Trends 🧪

Image of Build Times

Image of Tests

@akshay-saraswat
Copy link
Contributor Author

Ran following commands in package/kubernetes

  1. elastic-package export dashboard -v
  2. elastic-package format
  3. elastic-package build

Added a screenshot too to the updated commit.

Copy link
Member

@ChrsMark ChrsMark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only one addition is needed, but overall it looks good to me.

packages/kubernetes/changelog.yml Show resolved Hide resolved
@ChrsMark
Copy link
Member

ChrsMark commented Aug 3, 2021

@akshay-saraswat
Copy link
Contributor Author

Tests seem to fail to due to format failures: https://beats-ci.elastic.co/blue/organizations/jenkins/Ingest-manager%2Fintegrations/detail/PR-1413/5/pipeline

  • Moved README changes from docs to _dev/build/docs.
  • Ran elastic-package format
  • Ran elastic-package lint
  • Ran elastic-package build

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>
@ChrsMark
Copy link
Member

ChrsMark commented Aug 3, 2021

@mtojek tests seem to fail in CI while locally ep check does not complain. I wonder what can be wrong here, I restarted the the CI job several times but still 🤔 . Any ideas?

@ChrsMark ChrsMark merged commit 8086eb5 into elastic:master Aug 4, 2021
eyalkraft pushed a commit to build-security/integrations that referenced this pull request Mar 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
7.14 candidate bug Something isn't working, use only for issues Integration:kubernetes Kubernetes Team:Integrations Label for the Integrations team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants