Follow up changes in dashboards in mage check && fix minor issue #27553

kvch · 2021-08-23T14:00:46Z

What does this PR do?

This PR follows up the missing dashboard checks in mage check. It also addresses a minor dashboard decoding issue in -decode flag of export dashboard.

Furthermore, assets from the Saved Objects API are separated into different JSON files. We went back to JSONs to make it more developer-friendly. By sticking with the more human-readable JSONs it helps people to review changes a bit better.

When exporting dashboards I suggest you use the -folder flag. You have to specify a root folder e.g. a module name and the Beat/exporter will create a {foldername}/_meta/kibana/7 folder for you. The assets will be separated into different directories and the name of them will be the ID of the asset.

$ ./filebeat export dashboard --id Metricbeat-host-overview --folder system
$ tree metricbeat/module/system/_meta/kibana/7
metricbeat/module/system/_meta/kibana/7
├── dashboard
│   ├── 79ffd6e0-faa0-11e6-947f-177f697178b8-ecs.json
│   ├── c431f410-f9ac-11e9-90e8-1fb18e796788.json
│   ├── CPU-slash-Memory-per-container-ecs.json
│   └── Metricbeat-system-overview-ecs.json
└── visualization
    ├── 089b85d0-1b16-11e7-b09e-037021c4f8df-ecs.json
    ├── 19e123b0-4d5a-11e7-aee5-fdc812cc3bec-ecs.json
    ├── 1aae9140-1b93-11e7-8ada-3df93aab833e-ecs.json
    ├── 26732e20-1b91-11e7-bec4-a5e9ec5cab8b-ecs.json
    ├── 2e224660-1b19-11e7-b09e-037021c4f8df-ecs.json
    ├── 34f97ee0-1b96-11e7-8ada-3df93aab833e-ecs.json
    ├── 3d65d450-a9c3-11e7-af20-67db8aecb295-ecs.json
    ├── 4b254630-f998-11e9-90e8-1fb18e796788.json
    ├── 4d546850-1b15-11e7-b09e-037021c4f8df-ecs.json
    ├── 4e4bb1e0-1b1b-11e7-b09e-037021c4f8df-ecs.json
    ├── 522ee670-1b92-11e7-bec4-a5e9ec5cab8b-ecs.json
    ├── 590a60f0-5d87-11e7-8884-1bb4c3b890e4-ecs.json
    ├── 6b7b9a40-faa1-11e6-86b1-cd7735ff7e23-ecs.json
    ├── 7cdb1330-4d1a-11e7-a196-69b9a7a020a9-ecs.json
    ├── 825fdb80-4d1d-11e7-b5f2-2b7c1895bf32-ecs.json
    ├── 83e12df0-1b91-11e7-bec4-a5e9ec5cab8b-ecs.json
    ├── 855899e0-1b1c-11e7-b09e-037021c4f8df-ecs.json
    ├── 8c071e20-f999-11e9-90e8-1fb18e796788.json
    ├── 96976150-4d5d-11e7-aa29-87a97a796de6-ecs.json
    ├── 99381c80-4d60-11e7-9a4c-ed99bbcaa42b-ecs.json
    ├── 9c69cad0-f9b0-11e9-90e8-1fb18e796788.json
    ├── a30871f0-f98f-11e9-90e8-1fb18e796788.json
    ├── ab2d1e90-1b1a-11e7-b09e-037021c4f8df-ecs.json
    ├── bb3a8720-f991-11e9-90e8-1fb18e796788.json
    ├── bfa5e400-1b16-11e7-b09e-037021c4f8df-ecs.json
    ├── c5e3cf90-4d60-11e7-9a4c-ed99bbcaa42b-ecs.json
    ├── c6f2ffd0-4d17-11e7-a196-69b9a7a020a9-ecs.json
    ├── Container-Block-IO-ecs.json
    ├── Container-CPU-usage-ecs.json
    ├── Container-Memory-stats-ecs.json
    ├── d2e80340-4d5c-11e7-aa29-87a97a796de6-ecs.json
    ├── d3166e80-1b91-11e7-bec4-a5e9ec5cab8b-ecs.json
    ├── d3f51850-f9b6-11e9-90e8-1fb18e796788.json
    ├── e0f001c0-1b18-11e7-b09e-037021c4f8df-ecs.json
    ├── e6e639e0-f992-11e9-90e8-1fb18e796788.json
    ├── fe064790-1b1f-11e7-bec4-a5e9ec5cab8b-ecs.json
    └── System-Navigation-ecs.json

Why is it important?

It adds extra checks that make sure dashboards are formatted correctly.

It also makes dashboard exporting more developer friendly.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~- [ ] I have made corresponding changes to the documentation~~
~~- [ ] I have made corresponding change to the default configuration files~~
~~- [ ] I have added tests that prove my fix is effective or that my feature works~~
~~- [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.~~

mergify · 2021-08-23T14:00:51Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fix-libbeat-dashboards-must-be-decoded upstream/fix-libbeat-dashboards-must-be-decoded
git merge upstream/master
git push upstream fix-libbeat-dashboards-must-be-decoded

elasticmachine · 2021-08-23T14:01:00Z

Pinging @elastic/agent (Team:Agent)

elasticmachine · 2021-08-23T15:17:35Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-08-26T16:27:12.340+0000
Duration: 162 min 27 sec
Commit: a84f99d

Test stats 🧪

Test	Results
Failed	0
Passed	53711
Skipped	5325
Total	59036

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	53711
Skipped	5325
Total	59036

jsoriano · 2021-08-23T15:17:38Z

auditbeat/module/auditd/_meta/kibana/7/dashboard/auditbeat-kernel-executions.ndjson

+{"attributes":{"description":"Command executions","kibanaSavedObjectMeta":{"searchSourceJSON":{"filter":[],"query":{"language":"kuery","query":""}}},"savedSearchRefName":"search_0","title":"Error Codes [Auditbeat Auditd] ECS","uiStateJSON":{},"version":1,"visState":{"aggs":[{"enabled":true,"id":"1","params":{},"schema":"metric","type":"count"},{"enabled":true,"id":"2","params":{"exclude":"0","field":"auditd.data.exit","order":"desc","orderBy":"1","size":10},"schema":"segment","type":"terms"}],"params":{"addLegend":true,"addTooltip":true,"distinctColors":true,"isDonut":true,"legendPosition":"right","palette":{"name":"kibana_palette","type":"palette"},"type":"pie"},"title":"Error Codes [Auditbeat Auditd] ECS","type":"pie"}},"coreMigrationVersion":"8.0.0","id":"20a8e8d0-c1c8-11e7-8995-936807a28b16-ecs","migrationVersion":{"visualization":"7.14.0"},"references":[{"id":"d382f5b0-c1c6-11e7-8995-936807a28b16-ecs","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2021-08-04T16:35:57.797Z","version":"WzQ5NjYsMV0="}
+{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":{"filter":[],"indexRefName":"kibanaSavedObjectMeta.searchSourceJSON.index","query":{"language":"kuery","query":""}}},"title":"Primary Username Tag Cloud [Auditbeat Auditd] ECS","uiStateJSON":{},"version":1,"visState":{"aggs":[{"enabled":true,"id":"1","params":{},"schema":"metric","type":"count"},{"enabled":true,"id":"2","params":{"field":"auditd.summary.actor.primary","order":"desc","orderBy":"1","size":10},"schema":"segment","type":"terms"}],"params":{"maxFontSize":45,"minFontSize":18,"orientation":"single","palette":{"name":"kibana_palette","type":"palette"},"scale":"linear"},"title":"Primary Username Tag Cloud [Auditbeat Auditd] ECS","type":"tagcloud"}},"coreMigrationVersion":"8.0.0","id":"f81a6de0-c1c1-11e7-8995-936807a28b16-ecs","migrationVersion":{"visualization":"7.14.0"},"references":[{"id":"auditbeat-*","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"visualization","updated_at":"2021-08-04T16:35:57.797Z","version":"WzQ5NjcsMV0="}
+{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":{"filter":[],"query":{"language":"kuery","query":""}}},"savedSearchRefName":"search_0","title":"Exe Name Tag Cloud [Auditbeat Auditd] ECS","uiStateJSON":{},"version":1,"visState":{"aggs":[{"enabled":true,"id":"1","params":{},"schema":"metric","type":"count"},{"enabled":true,"id":"2","params":{"field":"process.executable","order":"desc","orderBy":"1","size":10},"schema":"segment","type":"terms"}],"params":{"maxFontSize":45,"minFontSize":14,"orientation":"single","palette":{"name":"kibana_palette","type":"palette"},"scale":"linear"},"title":"Exe Name Tag Cloud [Auditbeat Auditd] ECS","type":"tagcloud"}},"coreMigrationVersion":"8.0.0","id":"2efac370-c1ca-11e7-8995-936807a28b16-ecs","migrationVersion":{"visualization":"7.14.0"},"references":[{"id":"d382f5b0-c1c6-11e7-8995-936807a28b16-ecs","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2021-08-04T16:35:57.797Z","version":"WzQ5NjgsMV0="}
+{"attributes":{"description":"Overview of kernel executions","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":{"filter":[],"highlightAll":true,"query":{"language":"kuery","query":""},"version":true}},"optionsJSON":{"darkTheme":false,"useMargins":false},"panelsJSON":[{"embeddableConfig":{"enhancements":{}},"gridData":{"h":12,"i":"1","w":16,"x":16,"y":0},"panelIndex":"1","panelRefName":"panel_1","type":"visualization","version":"7.3.0"},{"embeddableConfig":{"enhancements":{}},"gridData":{"h":12,"i":"3","w":16,"x":32,"y":0},"panelIndex":"3","panelRefName":"panel_3","type":"visualization","version":"7.3.0"},{"embeddableConfig":{"enhancements":{}},"gridData":{"h":12,"i":"5","w":16,"x":0,"y":0},"panelIndex":"5","panelRefName":"panel_5","type":"visualization","version":"7.3.0"},{"embeddableConfig":{"enhancements":{}},"gridData":{"h":20,"i":"6","w":48,"x":0,"y":12},"panelIndex":"6","panelRefName":"panel_6","type":"search","version":"7.3.0"}],"timeRestore":false,"title":"[Auditbeat Auditd] Executions ECS","version":1},"coreMigrationVersion":"8.0.0","id":"7de391b0-c1ca-11e7-8995-936807a28b16-ecs","migrationVersion":{"dashboard":"7.14.0"},"references":[{"id":"20a8e8d0-c1c8-11e7-8995-936807a28b16-ecs","name":"1:panel_1","type":"visualization"},{"id":"f81a6de0-c1c1-11e7-8995-936807a28b16-ecs","name":"3:panel_3","type":"visualization"},{"id":"2efac370-c1ca-11e7-8995-936807a28b16-ecs","name":"5:panel_5","type":"visualization"},{"id":"d382f5b0-c1c6-11e7-8995-936807a28b16-ecs","name":"6:panel_6","type":"search"}],"type":"dashboard","updated_at":"2021-08-04T16:35:57.797Z","version":"WzQ5NzAsMV0="}


Decoding the embeded JSON objects is nice, what do you think about storing them with indentation as they were before?
They wouldn't be strictly NDJSON files... but this would make them to be easier to review. setup would take care of fixing the format as expected by the API.

So the JSONs would no longer be delimited by a new line. What else would you use instead as a delimiter? My first idea would be putting every asset into a separate file and putting each dashboard into a different folder so we know which asset belongs to each dashboard. If you have anything simpler, maybe returning to JSONs, please let me know.

In general, I do not like the idea but if this helps module developers, I can be persuaded.

So the JSONs would no longer be delimited by a new line. What else would you use instead as a delimiter?

It could still be delimited by a new line, we could decode one object after the other using a json decoder. We would lose though the possibility of continuing to decode objects after one is invalid, but in that sense it would be the same as with dashboards in beats before 7.15.

My first idea would be putting every asset into a separate file and putting each dashboard into a different folder so we know which asset belongs to each dashboard. If you have anything simpler, maybe returning to JSONs, please let me know.

In integrations every asset is stored in a different file now https://github.com/elastic/integrations/tree/master/packages/apache/kibana, so this could be definitely an option, with the benefit of using the same format in both places.

In general, I do not like the idea but if this helps module developers, I can be persuaded.

In my opinion it really helps 🙂 Compare the changes in dashboards in these two PRs:

https://github.com/elastic/beats/pull/27520/files

https://github.com/elastic/beats/pull/27119/files

Similar changes, but in the second one they are much easier to see.

dev-tools/mage/check.go

jsoriano

Using one file per object sounds good to me, this way this is consistent with elastic-package.

dev-tools/cmd/dashboards/export_dashboards.go

dev-tools/mage/check.go

dev-tools/mage/dashboard.go

libbeat/dashboards/decode.go

libbeat/dashboards/export.go

libbeat/tests/system/test_dashboard.py

dev-tools/cmd/dashboards/export_dashboards.go

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

jsoriano

Thanks for all these changes! I think we may still need to review some things with test_dashboards, but let's check after merging this to avoid problems and conflicts with the module dashboards.

jsoriano · 2021-08-26T14:47:25Z

libbeat/dashboards/decode.go

+		_, err = result.WriteRune('\n')
+		if err != nil {
+			return exported
+		}


Nit. Writes to bytes.Buffer always return nil according to the docs, so no need to handle errors here. But better safe than sorry, so as you prefer 🙂

) This PR follows up the missing dashboard checks in `mage check`. It also addresses a minor dashboard decoding issue in `-decode` flag of `export dashboard`. Furthermore, assets from the Saved Objects API are separated into different JSON files. We went back to JSONs to make it more developer-friendly. By sticking with the more human-readable JSONs it helps people to review changes a bit better. When exporting dashboards I suggest you use the `-folder` flag. You have to specify a root folder e.g. a module name and the Beat/exporter will create a `{foldername}/_meta/kibana/7` folder for you. The assets will be separated into different directories and the name of them will be the ID of the asset. ```sh $ ./filebeat export dashboard --id Metricbeat-host-overview --folder system $ tree metricbeat/module/system/_meta/kibana/7 metricbeat/module/system/_meta/kibana/7 ├── dashboard │ ├── 79ffd6e0-faa0-11e6-947f-177f697178b8-ecs.json │ ├── c431f410-f9ac-11e9-90e8-1fb18e796788.json │ ├── CPU-slash-Memory-per-container-ecs.json │ └── Metricbeat-system-overview-ecs.json └── visualization ├── 089b85d0-1b16-11e7-b09e-037021c4f8df-ecs.json ├── 19e123b0-4d5a-11e7-aee5-fdc812cc3bec-ecs.json ├── 1aae9140-1b93-11e7-8ada-3df93aab833e-ecs.json ├── 26732e20-1b91-11e7-bec4-a5e9ec5cab8b-ecs.json ├── 2e224660-1b19-11e7-b09e-037021c4f8df-ecs.json ├── 34f97ee0-1b96-11e7-8ada-3df93aab833e-ecs.json ├── 3d65d450-a9c3-11e7-af20-67db8aecb295-ecs.json ├── 4b254630-f998-11e9-90e8-1fb18e796788.json ├── 4d546850-1b15-11e7-b09e-037021c4f8df-ecs.json ├── 4e4bb1e0-1b1b-11e7-b09e-037021c4f8df-ecs.json ├── 522ee670-1b92-11e7-bec4-a5e9ec5cab8b-ecs.json ├── 590a60f0-5d87-11e7-8884-1bb4c3b890e4-ecs.json ├── 6b7b9a40-faa1-11e6-86b1-cd7735ff7e23-ecs.json ├── 7cdb1330-4d1a-11e7-a196-69b9a7a020a9-ecs.json ├── 825fdb80-4d1d-11e7-b5f2-2b7c1895bf32-ecs.json ├── 83e12df0-1b91-11e7-bec4-a5e9ec5cab8b-ecs.json ├── 855899e0-1b1c-11e7-b09e-037021c4f8df-ecs.json ├── 8c071e20-f999-11e9-90e8-1fb18e796788.json ├── 96976150-4d5d-11e7-aa29-87a97a796de6-ecs.json ├── 99381c80-4d60-11e7-9a4c-ed99bbcaa42b-ecs.json ├── 9c69cad0-f9b0-11e9-90e8-1fb18e796788.json ├── a30871f0-f98f-11e9-90e8-1fb18e796788.json ├── ab2d1e90-1b1a-11e7-b09e-037021c4f8df-ecs.json ├── bb3a8720-f991-11e9-90e8-1fb18e796788.json ├── bfa5e400-1b16-11e7-b09e-037021c4f8df-ecs.json ├── c5e3cf90-4d60-11e7-9a4c-ed99bbcaa42b-ecs.json ├── c6f2ffd0-4d17-11e7-a196-69b9a7a020a9-ecs.json ├── Container-Block-IO-ecs.json ├── Container-CPU-usage-ecs.json ├── Container-Memory-stats-ecs.json ├── d2e80340-4d5c-11e7-aa29-87a97a796de6-ecs.json ├── d3166e80-1b91-11e7-bec4-a5e9ec5cab8b-ecs.json ├── d3f51850-f9b6-11e9-90e8-1fb18e796788.json ├── e0f001c0-1b18-11e7-b09e-037021c4f8df-ecs.json ├── e6e639e0-f992-11e9-90e8-1fb18e796788.json ├── fe064790-1b1f-11e7-bec4-a5e9ec5cab8b-ecs.json └── System-Navigation-ecs.json ``` It adds extra checks that make sure dashboards are formatted correctly. It also makes dashboard exporting more developer friendly. (cherry picked from commit c017771)

) (#27636) This PR follows up the missing dashboard checks in `mage check`. It also addresses a minor dashboard decoding issue in `-decode` flag of `export dashboard`. Furthermore, assets from the Saved Objects API are separated into different JSON files. We went back to JSONs to make it more developer-friendly. By sticking with the more human-readable JSONs it helps people to review changes a bit better. When exporting dashboards I suggest you use the `-folder` flag. You have to specify a root folder e.g. a module name and the Beat/exporter will create a `{foldername}/_meta/kibana/7` folder for you. The assets will be separated into different directories and the name of them will be the ID of the asset. ```sh $ ./filebeat export dashboard --id Metricbeat-host-overview --folder system $ tree metricbeat/module/system/_meta/kibana/7 metricbeat/module/system/_meta/kibana/7 ├── dashboard │ ├── 79ffd6e0-faa0-11e6-947f-177f697178b8-ecs.json │ ├── c431f410-f9ac-11e9-90e8-1fb18e796788.json │ ├── CPU-slash-Memory-per-container-ecs.json │ └── Metricbeat-system-overview-ecs.json └── visualization ├── 089b85d0-1b16-11e7-b09e-037021c4f8df-ecs.json ├── 19e123b0-4d5a-11e7-aee5-fdc812cc3bec-ecs.json ├── 1aae9140-1b93-11e7-8ada-3df93aab833e-ecs.json ├── 26732e20-1b91-11e7-bec4-a5e9ec5cab8b-ecs.json ├── 2e224660-1b19-11e7-b09e-037021c4f8df-ecs.json ├── 34f97ee0-1b96-11e7-8ada-3df93aab833e-ecs.json ├── 3d65d450-a9c3-11e7-af20-67db8aecb295-ecs.json ├── 4b254630-f998-11e9-90e8-1fb18e796788.json ├── 4d546850-1b15-11e7-b09e-037021c4f8df-ecs.json ├── 4e4bb1e0-1b1b-11e7-b09e-037021c4f8df-ecs.json ├── 522ee670-1b92-11e7-bec4-a5e9ec5cab8b-ecs.json ├── 590a60f0-5d87-11e7-8884-1bb4c3b890e4-ecs.json ├── 6b7b9a40-faa1-11e6-86b1-cd7735ff7e23-ecs.json ├── 7cdb1330-4d1a-11e7-a196-69b9a7a020a9-ecs.json ├── 825fdb80-4d1d-11e7-b5f2-2b7c1895bf32-ecs.json ├── 83e12df0-1b91-11e7-bec4-a5e9ec5cab8b-ecs.json ├── 855899e0-1b1c-11e7-b09e-037021c4f8df-ecs.json ├── 8c071e20-f999-11e9-90e8-1fb18e796788.json ├── 96976150-4d5d-11e7-aa29-87a97a796de6-ecs.json ├── 99381c80-4d60-11e7-9a4c-ed99bbcaa42b-ecs.json ├── 9c69cad0-f9b0-11e9-90e8-1fb18e796788.json ├── a30871f0-f98f-11e9-90e8-1fb18e796788.json ├── ab2d1e90-1b1a-11e7-b09e-037021c4f8df-ecs.json ├── bb3a8720-f991-11e9-90e8-1fb18e796788.json ├── bfa5e400-1b16-11e7-b09e-037021c4f8df-ecs.json ├── c5e3cf90-4d60-11e7-9a4c-ed99bbcaa42b-ecs.json ├── c6f2ffd0-4d17-11e7-a196-69b9a7a020a9-ecs.json ├── Container-Block-IO-ecs.json ├── Container-CPU-usage-ecs.json ├── Container-Memory-stats-ecs.json ├── d2e80340-4d5c-11e7-aa29-87a97a796de6-ecs.json ├── d3166e80-1b91-11e7-bec4-a5e9ec5cab8b-ecs.json ├── d3f51850-f9b6-11e9-90e8-1fb18e796788.json ├── e0f001c0-1b18-11e7-b09e-037021c4f8df-ecs.json ├── e6e639e0-f992-11e9-90e8-1fb18e796788.json ├── fe064790-1b1f-11e7-bec4-a5e9ec5cab8b-ecs.json └── System-Navigation-ecs.json ``` It adds extra checks that make sure dashboards are formatted correctly. It also makes dashboard exporting more developer friendly. (cherry picked from commit c017771) Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>

) (#27637) This PR follows up the missing dashboard checks in `mage check`. It also addresses a minor dashboard decoding issue in `-decode` flag of `export dashboard`. Furthermore, assets from the Saved Objects API are separated into different JSON files. We went back to JSONs to make it more developer-friendly. By sticking with the more human-readable JSONs it helps people to review changes a bit better. When exporting dashboards I suggest you use the `-folder` flag. You have to specify a root folder e.g. a module name and the Beat/exporter will create a `{foldername}/_meta/kibana/7` folder for you. The assets will be separated into different directories and the name of them will be the ID of the asset. ```sh $ ./filebeat export dashboard --id Metricbeat-host-overview --folder system $ tree metricbeat/module/system/_meta/kibana/7 metricbeat/module/system/_meta/kibana/7 ├── dashboard │ ├── 79ffd6e0-faa0-11e6-947f-177f697178b8-ecs.json │ ├── c431f410-f9ac-11e9-90e8-1fb18e796788.json │ ├── CPU-slash-Memory-per-container-ecs.json │ └── Metricbeat-system-overview-ecs.json └── visualization ├── 089b85d0-1b16-11e7-b09e-037021c4f8df-ecs.json ├── 19e123b0-4d5a-11e7-aee5-fdc812cc3bec-ecs.json ├── 1aae9140-1b93-11e7-8ada-3df93aab833e-ecs.json ├── 26732e20-1b91-11e7-bec4-a5e9ec5cab8b-ecs.json ├── 2e224660-1b19-11e7-b09e-037021c4f8df-ecs.json ├── 34f97ee0-1b96-11e7-8ada-3df93aab833e-ecs.json ├── 3d65d450-a9c3-11e7-af20-67db8aecb295-ecs.json ├── 4b254630-f998-11e9-90e8-1fb18e796788.json ├── 4d546850-1b15-11e7-b09e-037021c4f8df-ecs.json ├── 4e4bb1e0-1b1b-11e7-b09e-037021c4f8df-ecs.json ├── 522ee670-1b92-11e7-bec4-a5e9ec5cab8b-ecs.json ├── 590a60f0-5d87-11e7-8884-1bb4c3b890e4-ecs.json ├── 6b7b9a40-faa1-11e6-86b1-cd7735ff7e23-ecs.json ├── 7cdb1330-4d1a-11e7-a196-69b9a7a020a9-ecs.json ├── 825fdb80-4d1d-11e7-b5f2-2b7c1895bf32-ecs.json ├── 83e12df0-1b91-11e7-bec4-a5e9ec5cab8b-ecs.json ├── 855899e0-1b1c-11e7-b09e-037021c4f8df-ecs.json ├── 8c071e20-f999-11e9-90e8-1fb18e796788.json ├── 96976150-4d5d-11e7-aa29-87a97a796de6-ecs.json ├── 99381c80-4d60-11e7-9a4c-ed99bbcaa42b-ecs.json ├── 9c69cad0-f9b0-11e9-90e8-1fb18e796788.json ├── a30871f0-f98f-11e9-90e8-1fb18e796788.json ├── ab2d1e90-1b1a-11e7-b09e-037021c4f8df-ecs.json ├── bb3a8720-f991-11e9-90e8-1fb18e796788.json ├── bfa5e400-1b16-11e7-b09e-037021c4f8df-ecs.json ├── c5e3cf90-4d60-11e7-9a4c-ed99bbcaa42b-ecs.json ├── c6f2ffd0-4d17-11e7-a196-69b9a7a020a9-ecs.json ├── Container-Block-IO-ecs.json ├── Container-CPU-usage-ecs.json ├── Container-Memory-stats-ecs.json ├── d2e80340-4d5c-11e7-aa29-87a97a796de6-ecs.json ├── d3166e80-1b91-11e7-bec4-a5e9ec5cab8b-ecs.json ├── d3f51850-f9b6-11e9-90e8-1fb18e796788.json ├── e0f001c0-1b18-11e7-b09e-037021c4f8df-ecs.json ├── e6e639e0-f992-11e9-90e8-1fb18e796788.json ├── fe064790-1b1f-11e7-bec4-a5e9ec5cab8b-ecs.json └── System-Navigation-ecs.json ``` It adds extra checks that make sure dashboards are formatted correctly. It also makes dashboard exporting more developer friendly. (cherry picked from commit c017771) Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>

* master: Forward port 7.14.1 changelog to master (elastic#27687) Addressing multiple dashboard issues: deps loading once, field conversion, etc. (elastic#27669) Remove adaptive queue sizes from agent's spec files (elastic#27653) Osquerybeat: Improve testability and unit test coverage (elastic#27591) Osquerybeat: lockdown flagsfile, prevent global defaults (elastic#27611) Import the references of dashboard assets using the Saved Objects API (elastic#27647) Fix bug with override path in cgroups (elastic#27620) Allow Kibana client to authorize with Elasticsearch API key (elastic#27540) Filebeat auditd: Fix Top Exec Commands dashboard visualization (elastic#27638) [elastic-agent] Fix docker tar.gz generation for complete image (elastic#27621) Follow up changes in dashboards in mage check && fix minor issue (elastic#27553) [Heartbeat] Fix bug where `enabled: false` is ignored. (elastic#27615) Support kube_state_metrics v2.0.0 (elastic#27552)

…stic#27553) ## What does this PR do? This PR follows up the missing dashboard checks in `mage check`. It also addresses a minor dashboard decoding issue in `-decode` flag of `export dashboard`. Furthermore, assets from the Saved Objects API are separated into different JSON files. We went back to JSONs to make it more developer-friendly. By sticking with the more human-readable JSONs it helps people to review changes a bit better. When exporting dashboards I suggest you use the `-folder` flag. You have to specify a root folder e.g. a module name and the Beat/exporter will create a `{foldername}/_meta/kibana/7` folder for you. The assets will be separated into different directories and the name of them will be the ID of the asset. ```sh $ ./filebeat export dashboard --id Metricbeat-host-overview --folder system $ tree metricbeat/module/system/_meta/kibana/7 metricbeat/module/system/_meta/kibana/7 ├── dashboard │ ├── 79ffd6e0-faa0-11e6-947f-177f697178b8-ecs.json │ ├── c431f410-f9ac-11e9-90e8-1fb18e796788.json │ ├── CPU-slash-Memory-per-container-ecs.json │ └── Metricbeat-system-overview-ecs.json └── visualization ├── 089b85d0-1b16-11e7-b09e-037021c4f8df-ecs.json ├── 19e123b0-4d5a-11e7-aee5-fdc812cc3bec-ecs.json ├── 1aae9140-1b93-11e7-8ada-3df93aab833e-ecs.json ├── 26732e20-1b91-11e7-bec4-a5e9ec5cab8b-ecs.json ├── 2e224660-1b19-11e7-b09e-037021c4f8df-ecs.json ├── 34f97ee0-1b96-11e7-8ada-3df93aab833e-ecs.json ├── 3d65d450-a9c3-11e7-af20-67db8aecb295-ecs.json ├── 4b254630-f998-11e9-90e8-1fb18e796788.json ├── 4d546850-1b15-11e7-b09e-037021c4f8df-ecs.json ├── 4e4bb1e0-1b1b-11e7-b09e-037021c4f8df-ecs.json ├── 522ee670-1b92-11e7-bec4-a5e9ec5cab8b-ecs.json ├── 590a60f0-5d87-11e7-8884-1bb4c3b890e4-ecs.json ├── 6b7b9a40-faa1-11e6-86b1-cd7735ff7e23-ecs.json ├── 7cdb1330-4d1a-11e7-a196-69b9a7a020a9-ecs.json ├── 825fdb80-4d1d-11e7-b5f2-2b7c1895bf32-ecs.json ├── 83e12df0-1b91-11e7-bec4-a5e9ec5cab8b-ecs.json ├── 855899e0-1b1c-11e7-b09e-037021c4f8df-ecs.json ├── 8c071e20-f999-11e9-90e8-1fb18e796788.json ├── 96976150-4d5d-11e7-aa29-87a97a796de6-ecs.json ├── 99381c80-4d60-11e7-9a4c-ed99bbcaa42b-ecs.json ├── 9c69cad0-f9b0-11e9-90e8-1fb18e796788.json ├── a30871f0-f98f-11e9-90e8-1fb18e796788.json ├── ab2d1e90-1b1a-11e7-b09e-037021c4f8df-ecs.json ├── bb3a8720-f991-11e9-90e8-1fb18e796788.json ├── bfa5e400-1b16-11e7-b09e-037021c4f8df-ecs.json ├── c5e3cf90-4d60-11e7-9a4c-ed99bbcaa42b-ecs.json ├── c6f2ffd0-4d17-11e7-a196-69b9a7a020a9-ecs.json ├── Container-Block-IO-ecs.json ├── Container-CPU-usage-ecs.json ├── Container-Memory-stats-ecs.json ├── d2e80340-4d5c-11e7-aa29-87a97a796de6-ecs.json ├── d3166e80-1b91-11e7-bec4-a5e9ec5cab8b-ecs.json ├── d3f51850-f9b6-11e9-90e8-1fb18e796788.json ├── e0f001c0-1b18-11e7-b09e-037021c4f8df-ecs.json ├── e6e639e0-f992-11e9-90e8-1fb18e796788.json ├── fe064790-1b1f-11e7-bec4-a5e9ec5cab8b-ecs.json └── System-Navigation-ecs.json ``` ## Why is it important? It adds extra checks that make sure dashboards are formatted correctly. It also makes dashboard exporting more developer friendly.

kvch requested a review from jsoriano August 23, 2021 14:00

kvch requested a review from a team as a code owner August 23, 2021 14:00

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 23, 2021

kvch added the Team:Elastic-Agent Label for the Agent team label Aug 23, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 23, 2021

kvch added 2 commits August 23, 2021 16:01

Follow up dashboard changes in mage check

57e67ea

Decode dashboards correctly

97a2faf

kvch force-pushed the fix-libbeat-dashboards-must-be-decoded branch from eb6beca to 97a2faf Compare August 23, 2021 14:02

jsoriano reviewed Aug 23, 2021

View reviewed changes

kvch added 4 commits August 24, 2021 16:20

tmp

3aea58f

tmp

11382a4

follow up in tests

99a79ef

add tests and follow up in module files

1b00b80

jsoriano reviewed Aug 26, 2021

View reviewed changes

kvch and others added 9 commits August 26, 2021 14:46

More elegant way to write bytes

48e7fe4

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

address review notes

4c45ace

fix lint

8a68cfb

one more commit

96173a5

always decode

057b5dc

add error message when using -output

be50fdf

add error message

6e54774

make -folder required

96a71b2

correct correct extension

a84f99d

jsoriano approved these changes Aug 26, 2021

View reviewed changes

kvch added backport-v7.15.0 Automated backport with mergify backport-v7.16.0 Automated backport with mergify labels Aug 30, 2021

kvch merged commit c017771 into elastic:master Aug 30, 2021

mergify bot mentioned this pull request Aug 30, 2021

[7.x](backport #27553) Follow up changes in dashboards in mage check && fix minor issue #27636

Merged

mergify bot mentioned this pull request Aug 30, 2021

[7.15](backport #27553) Follow up changes in dashboards in mage check && fix minor issue #27637

Merged

jsoriano mentioned this pull request Aug 30, 2021

Support decoded saved objects when importing from Beats elastic/integrations#1559

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Follow up changes in dashboards in mage check && fix minor issue #27553

Follow up changes in dashboards in mage check && fix minor issue #27553

kvch commented Aug 23, 2021 •

edited

Loading

mergify bot commented Aug 23, 2021

elasticmachine commented Aug 23, 2021

elasticmachine commented Aug 23, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

jsoriano Aug 23, 2021

kvch Aug 23, 2021

jsoriano Aug 23, 2021 •

edited

Loading

jsoriano left a comment

jsoriano left a comment

jsoriano Aug 26, 2021

Follow up changes in dashboards in mage check && fix minor issue #27553

Follow up changes in dashboards in mage check && fix minor issue #27553

Conversation

kvch commented Aug 23, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

mergify bot commented Aug 23, 2021

elasticmachine commented Aug 23, 2021

elasticmachine commented Aug 23, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

jsoriano Aug 23, 2021

Choose a reason for hiding this comment

kvch Aug 23, 2021

Choose a reason for hiding this comment

jsoriano Aug 23, 2021 • edited Loading

Choose a reason for hiding this comment

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano Aug 26, 2021

Choose a reason for hiding this comment

kvch commented Aug 23, 2021 •

edited

Loading

elasticmachine commented Aug 23, 2021 •

edited by jenkins-beats-ci bot

Loading

jsoriano Aug 23, 2021 •

edited

Loading