Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

index_pattern under visState.params cannot be empty "" #18865

Merged
merged 8 commits into from
Jun 3, 2020
Merged

index_pattern under visState.params cannot be empty "" #18865

merged 8 commits into from
Jun 3, 2020

Conversation

kaiyan-sheng
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng commented May 29, 2020

This PR is to add check to make sure index_pattern under visState.params not be empty "". For example in metricbeat dashboard/visualization, when default index is not metricbeat-* in Kibana and index_pattern is set to "", visualizations will take the default index pattern by default. If the default index pattern is not metricbeat-*, then these dashboards will fail and show no data.

This PR is to not only fix the rest of the dashboards having this issue but also adding the check in mage/check.go.

Related PR: #18698
Discuss Forum Link: https://discuss.elastic.co/t/metricbeats-dashboards-for-kubernetes-controller-manager/233402/4

This PR also improved the error message on which visualization failed the test:

>> Dashboard format - module/kubernetes/_meta/kibana/7/dashboard/Metricbeat-kubernetes-controller-manager.json:
   expected index pattern reference 'metricbeat-*': unexpected index pattern reference found in visualization state params: `` in visualization `Controller Process [Metricbeat Kubernetes]`
>> Dashboard format - module/kubernetes/_meta/kibana/7/dashboard/Metricbeat-kubernetes-proxy.json:
   expected index pattern reference 'metricbeat-*': unexpected index pattern reference found in visualization state params: `` in visualization `Proxy CPU [Metricbeat Kubernetes]`
   expected index pattern reference 'metricbeat-*': unexpected index pattern reference found in visualization state params: `` in visualization `Proxy memory [Metricbeat Kubernetes]`

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 29, 2020
@kaiyan-sheng kaiyan-sheng self-assigned this May 29, 2020
@kaiyan-sheng kaiyan-sheng added needs_backport PR is waiting to be backported to other branches. Team:Platforms Label for the Integrations - Platforms team labels May 29, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 29, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented May 29, 2020

❕ Build Aborted

Either there was a build timeout or someone aborted the build.'}

Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: [Branch indexing]

  • Start Time: 2020-06-02T13:40:40.732+0000

  • Duration: 123 min 14 sec

Test stats 🧪

Test Results
Failed 0
Passed 8259
Skipped 1543
Total 9802

Steps errors

Expand to view the steps failures

  • Name: Fix permissions

    • Description:

    • Duration: 0 min 6 sec

    • Start Time: 2020-06-02T14:47:03.150+0000

    • log

  • Name: Install docker-compose 1.21.0

    • Description: .ci/scripts/install-docker-compose.sh

    • Duration: 1 min 32 sec

    • Start Time: 2020-06-02T15:29:21.941+0000

    • log

  • Name: Install Go 1.13.10

    • Description: .ci/scripts/install-go.sh

    • Duration: 1 min 57 sec

    • Start Time: 2020-06-02T14:09:11.000+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-06-02T15:43:37.197Z] + TERRAFORM_VERSION=0.12.24
[2020-06-02T15:43:37.197Z] + HOME=/var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865
[2020-06-02T15:43:37.197Z] + TERRAFORM_CMD=/var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/bin/terraform
[2020-06-02T15:43:37.197Z] ++ uname -s
[2020-06-02T15:43:37.197Z] ++ tr '[:upper:]' '[:lower:]'
[2020-06-02T15:43:37.197Z] + OS=linux
[2020-06-02T15:43:37.197Z] + mkdir -p /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/bin
[2020-06-02T15:43:37.197Z] + curl -sSLo - https://releases.hashicorp.com/terraform/0.12.24/terraform_0.12.24_linux_amd64.zip
[2020-06-02T15:43:37.769Z] ++ dirname /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/bin/terraform
[2020-06-02T15:43:37.769Z] + unzip -o /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/bin/terraform.zip -d /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/bin
[2020-06-02T15:43:37.769Z] Archive:  /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/bin/terraform.zip
[2020-06-02T15:43:38.340Z]   inflating: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/bin/terraform  
[2020-06-02T15:43:38.340Z] + rm /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/bin/terraform.zip
[2020-06-02T15:43:38.340Z] + chmod +x /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/bin/terraform
[2020-06-02T15:43:38.694Z] + make mage
[2020-06-02T15:43:38.694Z] Installing mage v1.9.0 from vendor dir.
[2020-06-02T15:43:40.077Z] /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/.magefile cleaned
[2020-06-02T15:43:40.423Z] + git config --get user.email
[2020-06-02T15:43:40.423Z] + [ -z  ]
[2020-06-02T15:43:40.423Z] + git config user.email beatsmachine@users.noreply.github.com
[2020-06-02T15:43:40.423Z] + git config user.name beatsmachine
[2020-06-02T15:43:40.881Z] + .ci/scripts/terraform-cleanup.sh x-pack/metricbeat
[2020-06-02T15:43:40.881Z] + DIRECTORY=x-pack/metricbeat
[2020-06-02T15:43:40.881Z] + FAILED=0
[2020-06-02T15:43:40.881Z] ++ find x-pack/metricbeat -name terraform.tfstate
[2020-06-02T15:43:40.881Z] + exit 0
[2020-06-02T15:43:41.384Z] + curl -sSLo codecov https://codecov.io/bash
[2020-06-02T15:43:41.648Z] + FILE=auditbeat/build/coverage/full.cov
[2020-06-02T15:43:41.648Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-06-02T15:43:41.648Z] + FILE=filebeat/build/coverage/full.cov
[2020-06-02T15:43:41.648Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-06-02T15:43:41.648Z] + FILE=heartbeat/build/coverage/full.cov
[2020-06-02T15:43:41.648Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-06-02T15:43:41.648Z] + FILE=libbeat/build/coverage/full.cov
[2020-06-02T15:43:41.648Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-06-02T15:43:41.648Z] + FILE=metricbeat/build/coverage/full.cov
[2020-06-02T15:43:41.648Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-06-02T15:43:41.648Z] + FILE=packetbeat/build/coverage/full.cov
[2020-06-02T15:43:41.649Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-06-02T15:43:41.649Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-06-02T15:43:41.649Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-06-02T15:43:41.649Z] + FILE=journalbeat/build/coverage/full.cov
[2020-06-02T15:43:41.649Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-06-02T15:43:42.439Z] Failed in branch Metricbeat x-pack
[2020-06-02T15:43:42.706Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats
[2020-06-02T15:43:43.046Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-06-02T15:43:43.070Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Lint
[2020-06-02T15:43:43.216Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-06-02T15:43:43.332Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-06-02T15:43:43.457Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Dockerlogbeat
[2020-06-02T15:43:43.600Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Journalbeat-oss
[2020-06-02T15:43:43.756Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-06-02T15:43:43.866Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Winlogbeat-oss
[2020-06-02T15:43:43.975Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-06-02T15:43:44.080Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-06-02T15:43:44.193Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-06-02T15:43:44.296Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Heartbeat-oss
[2020-06-02T15:43:44.404Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-06-02T15:43:44.543Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-06-02T15:43:44.667Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Libbeat-x-pack
[2020-06-02T15:43:44.794Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-06-02T15:43:44.927Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-06-02T15:43:45.035Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-06-02T15:43:45.166Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Filebeat-Windows
[2020-06-02T15:43:45.295Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-06-02T15:43:45.424Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-06-02T15:43:45.682Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Packetbeat-oss
[2020-06-02T15:43:45.866Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-06-02T15:43:46.050Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Metricbeat-Windows
[2020-06-02T15:43:46.226Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Filebeat-x-pack
[2020-06-02T15:43:46.342Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-06-02T15:43:46.494Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Filebeat-oss
[2020-06-02T15:43:46.615Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-06-02T15:43:46.744Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-06-02T15:43:46.881Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-06-02T15:43:47.058Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-06-02T15:43:47.174Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-06-02T15:43:47.389Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-06-02T15:43:47.627Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Libbeat-oss
[2020-06-02T15:43:47.747Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-06-02T15:43:47.883Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-06-02T15:43:48.037Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-06-02T15:43:48.174Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Libbeat-crosscompile
[2020-06-02T15:43:48.283Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Libbeat-stress-tests
[2020-06-02T15:43:48.416Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-06-02T15:43:48.528Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-06-02T15:43:48.953Z] + cat
[2020-06-02T15:43:48.953Z] + /usr/local/bin/runbld ./runbld-script
[2020-06-02T15:43:48.953Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-06-02T15:43:53.619Z] Body did not finish within grace period; terminating with extreme prejudice
[2020-06-02T15:43:53.678Z] ERROR: runbld post build action failed.
[2020-06-02T15:43:53.678Z] Timeout has been exceeded
[2020-06-02T15:43:53.894Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18865
[2020-06-02T15:43:54.021Z] [INFO] getVaultSecret: Getting secrets
[2020-06-02T15:43:54.083Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-06-02T15:43:54.790Z] + chmod 755 generate-build-data.sh
[2020-06-02T15:43:54.790Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18865/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18865/runs/9 ABORTED 7393789
[2020-06-02T15:43:54.790Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18865/runs/9/steps/?limit=10000 -o steps-info.json
[2020-06-02T15:43:57.273Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18865/runs/9/tests/?status=FAILED -o tests-errors.json
[2020-06-02T15:43:59.134Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18865/runs/9/log/ -o pipeline-log.txt

@kaiyan-sheng kaiyan-sheng requested a review from a team as a code owner May 30, 2020 13:13
@@ -270,13 +270,14 @@ type dashboardObject struct {
Title string `json:"title"`
KibanaSavedObjectMeta *struct {
SearchSourceJSON struct {
Index string `json:"index"`
Index *string `json:"index"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need a pointer here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a pointer here because I need to distinguish if this field exists or the value of this field is empty string "". Without the pointer, even if VisState.Params doesn't have IndexPattern field, it will return "".

Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Looking at the affected dashboards, they seem to be some of the most recent ones, I wonder if something has changed in Kibana or the dashboards exporter that provokes this difference.
Maybe we can modify the dashboards exporter to fill these parameters with correct values.

@kaiyan-sheng kaiyan-sheng merged commit f6710e3 into elastic:master Jun 3, 2020
@kaiyan-sheng kaiyan-sheng deleted the empty_index_pattern branch June 3, 2020 13:50
@kaiyan-sheng kaiyan-sheng removed the needs_backport PR is waiting to be backported to other branches. label Jun 3, 2020
kaiyan-sheng added a commit that referenced this pull request Jun 4, 2020
…8942)

* `index_pattern` under visState.params cannot be empty ""

(cherry picked from commit f6710e3)
kaiyan-sheng added a commit that referenced this pull request Jun 4, 2020
…8947)

* `index_pattern` under visState.params cannot be empty ""

(cherry picked from commit f6710e3)
kaiyan-sheng added a commit that referenced this pull request Jun 5, 2020
…8943)

* `index_pattern` under visState.params cannot be empty ""

(cherry picked from commit f6710e3)
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
* `index_pattern` under visState.params cannot be empty ""
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
…) (elastic#18943)

* `index_pattern` under visState.params cannot be empty ""

(cherry picked from commit fbe67a0)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
review Team:Platforms Label for the Integrations - Platforms team v7.7.2 v7.8.0 v7.9.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants