Support Kibana Spaces #8045

ycombinator · 2018-08-21T23:03:51Z

Resolves #7942.

Kibana is implementing a new feature called Spaces, in which Kibana saved objects (such as dashboards) and advanced settings can be restricted to a user-defined namespace. Spaces are identified by a unique ID, e.g. my-space. There is also a notion of a Default space which corresponds to how Kibana worked up until the Spaces feature was introduced.

Beats have the ability to import dashboards into Kibana as well as export dashboards out of Kibana. With Spaces, dashboards may belong to a specific space. So Beats must learn to accept an optional Space ID and operate against it when importing or exporting dashboards. This PR teaches Beats to do just that.

Concretely, if a user wishes to import or export dashboards from a specific space, say with ID = my-space, they must either:

Edit <beat>.yml and set setup.kibana.space.id: my-space, or
Run <beat> setup or <beat> export dashboard along with the -E setup.kibana.space.id=my-space option.

Similarly, if a Beat developer wishes to export dashboards from a specific space, say with ID = my-space, they must run:

go run dev-tools/cmd/dashboards/export_dashboards.go -space-id my-space [other options]

ycombinator · 2018-08-21T23:06:08Z

Note that there is currently a bug in the Kibana Spaces implementation which allows a non-existent space ID to be specified without causing any errors: elastic/kibana#21408 (comment). Once this bug is resolved, I'd like to re-test this PR with a non-existent space ID to make sure we fail with a good error message.

ruflin · 2018-08-22T08:57:47Z

I think it would be worth to add a system test for this to make sure it works. There are already tests loading dashboards which could be copied and adjusted.

ruflin

I suggest to add the system tests directly as part of this PR? Code looks good but would be nice if we have automated tests around it so we also know in case functionality breaks in the Kibana side.

ycombinator · 2018-08-23T13:27:46Z

Yes, that’s my plan. I just haven’t got to it yet since your previous comment because of support live fire followed by being on holiday now. I plan to get to it next week when I’m back from holiday.

…

On Thu, Aug 23, 2018 at 07:25 Nicolas Ruflin ***@***.***> wrote: ***@***.**** requested changes on this pull request. I suggest to add the system tests directly as part of this PR? Code looks good but would be nice if we have automated tests around it so we also know in case functionality breaks in the Kibana side. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#8045 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADHdXv_WAJJ7B4xxSET2WgTC_oYblXCks5uTpEYgaJpZM4WGuzK> .

houndci-bot · 2018-08-27T17:57:57Z

dev-tools/cmd/dashboards/export_dashboards.go

@@ -48,11 +49,14 @@ func makeURL(url, path string, params url.Values) string {
 	return strings.Join([]string{url, path, "?", params.Encode()}, "")
 }

-func Export(client *http.Client, conn string, dashboard string, out string) error {
+func Export(client *http.Client, conn string, spaceID string, dashboard string, out string) error {


exported function Export should have comment or be unexported

alexfrancoeur · 2018-08-28T12:41:53Z

@ycombinator @ruflin I'm pretty familiar with the Kibana dev environment but this is the first time I've been asked to be a reviewer for any beats changes 😄 What's the best way to go about testing this PR? Does this seem about right? Also, does make start the beat? How would I test with <beat> setup?

mkdir -p ${GOPATH}/src/github.com/elastic
cd ${GOPATH}/src/github.com/elastic
git clone https://github.com/elastic/beats.git
git checkout -b ycombinator-support-kibana-spaces master
git pull git://github.com/ycombinator/beats.git support-kibana-spaces
cd beats/packetbeat
make

Sorry for the n00b questions

ycombinator · 2018-08-28T12:57:57Z

@alexfrancoeur Ahhhh, sorry about that, but the steps you outlined are 👌. After the last step (make), you can then proceed to test this PR as follows.

Edit packetbeat.yml and find the setup.kibana section. Under that, you should see a space.id setting, commented out. Uncomment it and set it's value to some space ID in Kibana (note that we currently have this bug that's relevant to this step: [Spaces] - Accessing non-existent Spaces kibana#22385)
(If you are on a Mac) Also in packetbeat.yml, set packetbeat.interfaces.device: lo0 (because the default value of any for that setting doesn't work on Macs).

Run the command to import Packetbeat dashboards into Kibana:

./packetbeat setup --dashboards -E setup.dashboards.directory=_meta/kibana.generated/

If successful, you should see the following output:

Loading dashboards (Kibana must be running and reachable)
Loaded dashboards

As an alternative to step 1, you should be able to specify -E setup.kibana.space.id=XXXX with the command in step 3.

ruflin

Code changes overall look good to me.

Is CI not happy yet because the Kibana PR is not merged yet? Please ping me as soon as Kibana PR is in so I can manually test this against master.

ycombinator · 2018-08-29T22:33:33Z

@ruflin Yeah, CI is failing on this PR at the moment because the Kibana Spaces PR (elastic/kibana#21408) hasn't been merged into master yet. I checked with @legrego about the Kibana Spaces PR and it's expected to get reviewed and then merged about 2-6 weeks from now.

So I'm going to leave this PR here as-is until then and resolve any conflicts, etc. all at once after the Kibana Spaces PR has landed in master.

ruflin · 2018-08-30T07:00:59Z

@legrego I would strongly prefer if we could get this rather sooner then later into master so we have enough time on our side to test the feature and integrate with it.

legrego · 2018-08-30T14:08:27Z

@ruflin

@legrego I would strongly prefer if we could get this rather sooner then later into master so we have enough time on our side to test the feature and integrate with it.

Understood. We'd love to get it in as soon as possible, but we are also trying to be realistic with our timeline. It's a large PR that will likely have a long review cycle, but we will do everything we can to get it into master in a reasonable timeframe.

In the meantime, is there anything we can do to facilitate testing?

ruflin · 2018-08-31T11:43:34Z

@legrego Thanks for the details. I think so far we are covered as we manually test against the feature branch.

…ce ID

ycombinator · 2018-10-01T14:20:49Z

Now that the Kibana Spaces PR is merged, I've rebased this PR on master.

ycombinator · 2018-10-01T18:35:25Z

CI is failing because the latest snapshot of Kibana being used by system tests is actually 3+ days old and therefore does not yet contain Spaces in it.

ycombinator · 2018-10-01T19:13:24Z

Jenkins, test this.

ycombinator · 2018-10-01T21:28:52Z

Jenkins, test this.

ycombinator · 2018-10-02T01:46:26Z

@ruflin CI is green now, after Kibana Spaces PR was merged into master and a few more changes to this PR around system tests. Ready for your review again. Thanks.

ruflin

Did not test it locally but code and system tests LGTM.

ruflin · 2018-10-03T09:40:59Z

libbeat/tests/system/test_dashboard.py

+        }
+
+        headers = {
+            "kbn-xsrf": "1"


out of curiosity: Why is this needed?

It's for protecting against CSRF attacks: https://www.owasp.org/index.php/Cross-Site_Request_Forgery_(CSRF)_Prevention_Cheat_Sheet#Protecting_REST_Services:_Use_of_Custom_Request_Headers

* Support Kibana Spaces via setup.kibana.space.id setting * Adding CHANGELOG entry * Add space-id option to export_dashboards script * Add integration tests for importing and exporting dashboards with space ID * Adding -space-id option for export_dashboards.go to documentation * Adding entry to dev CHANGELOG * Prefix space path with / * Create Kibana space before trying to use it * Make spaces tests conditional on version * Add semver to requirements.txt * Add kbn-xsrf header to POST request * Don't recreate space in second space test (cherry picked from commit 2a04e2c)

Cherry-pick of PR #8045 to 6.x branch. Original message: Resolves #7942. Kibana is implementing a new feature called Spaces, in which Kibana saved objects (such as dashboards) and advanced settings can be restricted to a user-defined namespace. Spaces are identified by a unique ID, e.g. `my-space`. There is also a notion of a Default space which corresponds to how Kibana worked up until the Spaces feature was introduced. Beats have the ability to import dashboards into Kibana as well as export dashboards out of Kibana. With Spaces, dashboards may belong to a specific space. So Beats must learn to accept an optional Space ID and operate against it when importing or exporting dashboards. This PR teaches Beats to do just that. Concretely, if a user wishes to import or export dashboards from a specific space, say with ID = `my-space`, they must either: * Edit `<beat>.yml` and set `setup.kibana.space.id: my-space`, _or_ * Run `<beat> setup` or `<beat> export dashboard` along with the `-E setup.kibana.space.id=my-space` option. Similarly, if a Beat _developer_ wishes to export dashboards from a specific space, say with ID = `my-space`, they must run: ```sh go run dev-tools/cmd/dashboards/export_dashboards.go -space-id my-space [other options] ```

The docs missing from #8045

The docs missing from elastic#8045

ycombinator added review libbeat v7.0.0-alpha1 v6.5.0 labels Aug 21, 2018

ycombinator requested a review from alexfrancoeur August 21, 2018 23:06

ycombinator requested a review from ruflin August 22, 2018 12:08

ruflin requested changes Aug 23, 2018

View reviewed changes

houndci-bot reviewed Aug 27, 2018

View reviewed changes

ycombinator changed the title ~~Support Kibana Spaces via setup.kibana.space.id setting~~ Support Kibana Spaces Aug 27, 2018

ycombinator force-pushed the support-kibana-spaces branch from 45598ca to f8260de Compare August 27, 2018 18:15

ycombinator force-pushed the support-kibana-spaces branch from b10d78d to 69f53b7 Compare August 28, 2018 13:34

ruflin reviewed Aug 29, 2018

View reviewed changes

chrisronline mentioned this pull request Sep 13, 2018

[Beats] Add space.id to all filebeat and metricbeat tutorials elastic/kibana#22998

Merged

ycombinator mentioned this pull request Oct 1, 2018

Support for Kibana Spaces in Beats modules #7942

Closed

ycombinator added 7 commits October 1, 2018 07:19

Support Kibana Spaces via setup.kibana.space.id setting

3ad9745

Adding CHANGELOG entry

287915d

Add space-id option to export_dashboards script

47e9986

Add integration tests for importing and exporting dashboards with spa…

9ec545f

…ce ID

Adding -space-id option for export_dashboards.go to documentation

820bcf8

Adding entry to dev CHANGELOG

5dc34d2

Prefix space path with /

a4d4d84

ycombinator force-pushed the support-kibana-spaces branch from 69f53b7 to a4d4d84 Compare October 1, 2018 14:20

ycombinator force-pushed the support-kibana-spaces branch 2 times, most recently from 4e38414 to 0f6feb8 Compare October 1, 2018 15:33

Create Kibana space before trying to use it

30ed549

ycombinator force-pushed the support-kibana-spaces branch from 0f6feb8 to 30ed549 Compare October 1, 2018 16:12

ycombinator added 2 commits October 1, 2018 10:12

Make spaces tests conditional on version

1f3e578

Add semver to requirements.txt

c90c541

ycombinator added 2 commits October 1, 2018 17:18

Add kbn-xsrf header to POST request

80b1b99

Don't recreate space in second space test

89dbafd

ycombinator mentioned this pull request Oct 2, 2018

[6.x] [Beats] Add space.id to all filebeat and metricbeat tutorials (#22998) elastic/kibana#23689

Merged

ruflin approved these changes Oct 3, 2018

View reviewed changes

ycombinator merged commit 2a04e2c into elastic:master Oct 3, 2018

ycombinator mentioned this pull request Oct 3, 2018

Cherry-pick #8045 to 6.x: Support Kibana Spaces #8545

Merged

This was referenced Oct 25, 2019

Document about setup.kibana.space.id is incomplete #14251

Closed

space.id #13934

Closed

ycombinator deleted the support-kibana-spaces branch December 25, 2019 11:12

immon added a commit that referenced this pull request Feb 26, 2020

[DOCS] document setup.kibana.space.id

61e1759

The docs missing from #8045

immon mentioned this pull request Feb 26, 2020

[DOCS] document setup.kibana.space.id #16604

Closed

leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023

[DOCS] document setup.kibana.space.id

b3dec87

The docs missing from elastic#8045

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Kibana Spaces #8045

Support Kibana Spaces #8045

ycombinator commented Aug 21, 2018 •

edited

ycombinator commented Aug 21, 2018

ruflin commented Aug 22, 2018

ruflin left a comment

ycombinator commented Aug 23, 2018 via email

houndci-bot Aug 27, 2018

alexfrancoeur commented Aug 28, 2018

ycombinator commented Aug 28, 2018 •

edited

ruflin left a comment

ycombinator commented Aug 29, 2018 •

edited

ruflin commented Aug 30, 2018

legrego commented Aug 30, 2018

ruflin commented Aug 31, 2018

ycombinator commented Oct 1, 2018

ycombinator commented Oct 1, 2018

ycombinator commented Oct 1, 2018

ycombinator commented Oct 1, 2018

ycombinator commented Oct 2, 2018

ruflin left a comment

ruflin Oct 3, 2018

ycombinator Oct 3, 2018

Support Kibana Spaces #8045

Support Kibana Spaces #8045

Conversation

ycombinator commented Aug 21, 2018 • edited

ycombinator commented Aug 21, 2018

ruflin commented Aug 22, 2018

ruflin left a comment

Choose a reason for hiding this comment

ycombinator commented Aug 23, 2018 via email

houndci-bot Aug 27, 2018

Choose a reason for hiding this comment

alexfrancoeur commented Aug 28, 2018

ycombinator commented Aug 28, 2018 • edited

ruflin left a comment

Choose a reason for hiding this comment

ycombinator commented Aug 29, 2018 • edited

ruflin commented Aug 30, 2018

legrego commented Aug 30, 2018

ruflin commented Aug 31, 2018

ycombinator commented Oct 1, 2018

ycombinator commented Oct 1, 2018

ycombinator commented Oct 1, 2018

ycombinator commented Oct 1, 2018

ycombinator commented Oct 2, 2018

ruflin left a comment

Choose a reason for hiding this comment

ruflin Oct 3, 2018

Choose a reason for hiding this comment

ycombinator Oct 3, 2018

Choose a reason for hiding this comment

ycombinator commented Aug 21, 2018 •

edited

ycombinator commented Aug 28, 2018 •

edited

ycombinator commented Aug 29, 2018 •

edited