New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Import / Export API for Dashboards #10858

Merged
merged 33 commits into from Jun 6, 2017

Conversation

@simianhacker
Member

simianhacker commented Mar 22, 2017

This PR creates an import / export API for dashboards. The current implementation under Mangement > Saved Objects allows you to only export everything or individual dashboards (with out the supporting saved objects like searches, visualizations or index patterns). This creates 2 API endpoints at /api/kibana/dashboards/export and /api/kibana/dashboards/import. The user can POST to the export end point an array of dashboard IDs. The response will contain ALL (searches, index patterns, visualizations, etc) the objects needed for that dashboard in an object with a version. To import a dashboard (an all of it's supporting objects) the user can POST the same object returned from the export endpoint to the import endpoint. The import endpoint will check the version and try and upgrade the objects if they are from a previous version; and import them into the .kibana index; skipping existing objects.

@tbragin tbragin added the :Management label Mar 27, 2017

@simianhacker simianhacker requested review from epixa and spalger Mar 29, 2017

@simianhacker simianhacker changed the title from [WIP] Import / Export API for Dashboards to Import / Export API for Dashboards Mar 29, 2017

@epixa

This comment has been minimized.

Show comment
Hide comment
@epixa

epixa Mar 29, 2017

Member

Does the export request actually result in any changes internally? If not, why not just do a GET instead?

Member

epixa commented Mar 29, 2017

Does the export request actually result in any changes internally? If not, why not just do a GET instead?

@simianhacker

This comment has been minimized.

Show comment
Hide comment
@simianhacker

simianhacker Mar 29, 2017

Member

No reason other then I was just trying to provide the maximum options so you could either use a GET or POST depending on which HTTP religion you subscribed to :D

Member

simianhacker commented Mar 29, 2017

No reason other then I was just trying to provide the maximum options so you could either use a GET or POST depending on which HTTP religion you subscribed to :D

@xycloud

This comment has been minimized.

Show comment
Hide comment
@xycloud

xycloud commented Mar 30, 2017

mark

@simianhacker

This comment has been minimized.

Show comment
Hide comment
@simianhacker

simianhacker Mar 30, 2017

Member

Hamill

Member

simianhacker commented Mar 30, 2017

Hamill

@epixa

This comment has been minimized.

Show comment
Hide comment
@epixa

epixa Mar 30, 2017

Member

@xycloud If you just want to see updates for a given issue, please use the "subscribe" link on the right side of posts rather than leaving a generic comment

Member

epixa commented Mar 30, 2017

@xycloud If you just want to see updates for a given issue, please use the "subscribe" link on the right side of posts rather than leaving a generic comment

@spalger spalger added :Platform and removed :Management labels Apr 11, 2017

@kimjoar kimjoar self-assigned this Apr 26, 2017

@simianhacker

This comment has been minimized.

Show comment
Hide comment
@simianhacker

simianhacker Apr 28, 2017

Member

@monicasarbu I added two query arguments to the import api:

  • ?exclude=<type> - For excluding certain types from being imported (aka index-patterns) and an option for
  • ?force=true - To force the import to overwrite existing objects.
Member

simianhacker commented Apr 28, 2017

@monicasarbu I added two query arguments to the import api:

  • ?exclude=<type> - For excluding certain types from being imported (aka index-patterns) and an option for
  • ?force=true - To force the import to overwrite existing objects.

@spalger spalger requested review from kimjoar and removed request for epixa and spalger May 2, 2017

Dreadnoth added a commit to Dreadnoth/kibana that referenced this pull request Aug 8, 2017

Import / Export API for Dashboards (#10858)
* Initial implementation

* Deduping final data array. Adding index patterns from saved searches

* Adding import

* Finishing import implimentation

* Adding tests for export

* Adding tests for import

* Filtering out bad ids

* Adding options for exclude and force

* Fixes per request by PR reviewers

* Adding a check for empty ids

* Use SavedObject API

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

* Omits missed objects, adds tests

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

* Moves import to SavedObjectsClient

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

* Fixing a bug with missing index patterns

* Fixing spacing issues (because the format is weird on this array which is too much for eslint to handle)

* Fixing tests and renaming file

* Changing paths to /api/kibana/dashboards/(export|import); adding validation to export

* Single call signature and use async/await

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

* Adding try/catch to JSON parses; Removing payload from export;

* removing redundent code

* Changing test to only use query arguments

* Removing bad panel from test data

* Return errors for bulkCreate objects

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

* Changing everything to named imports; removing deps from everything; fixing tests to reflect named imports

* Refactoring to use async/await pattern
@dkirrane

This comment has been minimized.

Show comment
Hide comment
@dkirrane

dkirrane Sep 7, 2017

Any documentation or Swagger to describe the usage of these import/export APIs?

dkirrane commented Sep 7, 2017

Any documentation or Swagger to describe the usage of these import/export APIs?

@hogarthj

This comment has been minimized.

Show comment
Hide comment
@hogarthj

hogarthj Sep 7, 2017

@dkirrane Hopefully better documented when the 6.0 release happens ;)

I read through the javascript API code to work out the appropriate behaviours... this is an excerpt of an ansible role you might be able to use as a base for dashboard behaviour:

- name: check if "{{ dashboard.name }}" exists
  uri:
    url: "http://localhost/api/kibana/dashboards/export?dashboard={{ dashboard.id }}"
    body_format: json
    user: "{{ kibana_es_username }}"
    password: "{{ kibana_es_password }}"
    force_basic_auth: "{{ 'yes' if kibana_es_username != '' else 'no' }}"
    status_code: [200,404]
  register: dashboard_status

- name: extract dashboard to tmp "{{ dashboard.name }}" if requested
  delegate_to: localhost
  template:
    src: kibana_dashboard.j2
    dest: "/tmp/{{ dashboard.file }}"
  when:
    - dashboard_status.json.objects is defined
    - dashboard_status.json.objects | length != 0
    - (kibana_dashboard_extract | bool) == true

- name: import dashboard "{{ dashboard.name }}"
  vars:
    force_import: "{{ (dashboard.force|default('false')|bool) or (kibana_dashboard_force|bool) }}"
    exclude_type: "{{ dashboard.exclude|default('index-pattern') }}"
  uri:
    headers:
      kbn-version: "{{ kibana_version }}"
    url: "http://localhost/api/kibana/dashboards/import?force={{ force_import | lower }}{{ '&exclude=' + exclude_type if exclude_type != '' else '' }}"
    body: "{{ lookup('file', dashboard.file ) | from_json | to_json }}"
    body_format: json
    method: POST
    user: "{{ kibana_es_username }}"
    password: "{{ kibana_es_password }}"
    force_basic_auth: "{{ 'yes' if kibana_es_username != '' else 'no' }}"
  when: ((dashboard_status.json.objects is defined) and (dashboard_status.json.objects | length == 0)) or ((dashboard.force|default('false')|bool) == true) or ((kibana_dashboard_force|bool) == true )
  changed_when: true

This is compatible with an x-pack install (just set the user credentials matching the vars and it will work) and expects a dict called dashboard with keys of name and file... where file is a json file that has been exported.

The way I've been using this is to design the dashboard in the Kibana UI and copy the ID from the URL.

Then to run this play with the variable kibana_dashboard_extract set to true to save the dashboard in /tmp

and then copy that saved json file to the files directory in the playbook for future import to other kibana instances.

hogarthj commented Sep 7, 2017

@dkirrane Hopefully better documented when the 6.0 release happens ;)

I read through the javascript API code to work out the appropriate behaviours... this is an excerpt of an ansible role you might be able to use as a base for dashboard behaviour:

- name: check if "{{ dashboard.name }}" exists
  uri:
    url: "http://localhost/api/kibana/dashboards/export?dashboard={{ dashboard.id }}"
    body_format: json
    user: "{{ kibana_es_username }}"
    password: "{{ kibana_es_password }}"
    force_basic_auth: "{{ 'yes' if kibana_es_username != '' else 'no' }}"
    status_code: [200,404]
  register: dashboard_status

- name: extract dashboard to tmp "{{ dashboard.name }}" if requested
  delegate_to: localhost
  template:
    src: kibana_dashboard.j2
    dest: "/tmp/{{ dashboard.file }}"
  when:
    - dashboard_status.json.objects is defined
    - dashboard_status.json.objects | length != 0
    - (kibana_dashboard_extract | bool) == true

- name: import dashboard "{{ dashboard.name }}"
  vars:
    force_import: "{{ (dashboard.force|default('false')|bool) or (kibana_dashboard_force|bool) }}"
    exclude_type: "{{ dashboard.exclude|default('index-pattern') }}"
  uri:
    headers:
      kbn-version: "{{ kibana_version }}"
    url: "http://localhost/api/kibana/dashboards/import?force={{ force_import | lower }}{{ '&exclude=' + exclude_type if exclude_type != '' else '' }}"
    body: "{{ lookup('file', dashboard.file ) | from_json | to_json }}"
    body_format: json
    method: POST
    user: "{{ kibana_es_username }}"
    password: "{{ kibana_es_password }}"
    force_basic_auth: "{{ 'yes' if kibana_es_username != '' else 'no' }}"
  when: ((dashboard_status.json.objects is defined) and (dashboard_status.json.objects | length == 0)) or ((dashboard.force|default('false')|bool) == true) or ((kibana_dashboard_force|bool) == true )
  changed_when: true

This is compatible with an x-pack install (just set the user credentials matching the vars and it will work) and expects a dict called dashboard with keys of name and file... where file is a json file that has been exported.

The way I've been using this is to design the dashboard in the Kibana UI and copy the ID from the URL.

Then to run this play with the variable kibana_dashboard_extract set to true to save the dashboard in /tmp

and then copy that saved json file to the files directory in the playbook for future import to other kibana instances.

@epixa

This comment has been minimized.

Show comment
Hide comment
@epixa

epixa Sep 7, 2017

Member

There's no documentation for these endpoints yet as they are still experimental, but if you like to live on the bleeding edge, you can feel free to give them a whirl. We are using these APIs for beats dashboard setup in 6.0 onward, so they are probably going to remain pretty stable despite being experimental.

In the absence of documentation, I recommend setting up a Kibana dashboard and then using the export API (GET /api/kibana/dashboards/export?dashboard=<id of dashboard>). The response body of the export API should be a valid request body for the import API (POST /api/kibana/dashboards/import). @simianhacker Please correct me if I'm wrong...

Member

epixa commented Sep 7, 2017

There's no documentation for these endpoints yet as they are still experimental, but if you like to live on the bleeding edge, you can feel free to give them a whirl. We are using these APIs for beats dashboard setup in 6.0 onward, so they are probably going to remain pretty stable despite being experimental.

In the absence of documentation, I recommend setting up a Kibana dashboard and then using the export API (GET /api/kibana/dashboards/export?dashboard=<id of dashboard>). The response body of the export API should be a valid request body for the import API (POST /api/kibana/dashboards/import). @simianhacker Please correct me if I'm wrong...

@hogarthj

This comment has been minimized.

Show comment
Hide comment
@hogarthj

hogarthj Sep 7, 2017

@epixa it is valid ... it's that response body I'm grabbing with the ansible above (from my dev instance) and then import directly to the test and prod instances.

But you can't just /api/kibana/dashboards/export AFAIK ... you need the id of the dashboard you want to grab or it errors stating it's missing a dashboard.id

hogarthj commented Sep 7, 2017

@epixa it is valid ... it's that response body I'm grabbing with the ansible above (from my dev instance) and then import directly to the test and prod instances.

But you can't just /api/kibana/dashboards/export AFAIK ... you need the id of the dashboard you want to grab or it errors stating it's missing a dashboard.id

@epixa

This comment has been minimized.

Show comment
Hide comment
@epixa

epixa Sep 7, 2017

Member

@hogarthj Thanks for the clarification, and yes, you're correct. You must specify dashboard as a query parameter, as you did in your ansible script. I've updated my comment.

Member

epixa commented Sep 7, 2017

@hogarthj Thanks for the clarification, and yes, you're correct. You must specify dashboard as a query parameter, as you did in your ansible script. I've updated my comment.

@simianhacker

This comment has been minimized.

Show comment
Hide comment
@simianhacker

simianhacker Sep 7, 2017

Member

@epixa and @hogarthj Yes... the response from the export is the body for the import. You are required to provide dashboard arguments for each dashboard id you're trying to export. Here is a what a typical dashboard export will look like (with X-Pack):

curl --user <username>:<password> -XGET localhost:5601/api/kibana/dashboards/export?dashboard=<some-dashboard-uuid> > my-dashboards.json

To import it looks like

curl --user <username>:<password> -XPOST localhost:5601/api/kibana/dashboards/import -H 'kbn-xsrf:true' -H 'Content-type:application/json' -d @./my-dashboards.json

If you are NOT using x-pack you can omit the --user <username>:<password> portion.

Member

simianhacker commented Sep 7, 2017

@epixa and @hogarthj Yes... the response from the export is the body for the import. You are required to provide dashboard arguments for each dashboard id you're trying to export. Here is a what a typical dashboard export will look like (with X-Pack):

curl --user <username>:<password> -XGET localhost:5601/api/kibana/dashboards/export?dashboard=<some-dashboard-uuid> > my-dashboards.json

To import it looks like

curl --user <username>:<password> -XPOST localhost:5601/api/kibana/dashboards/import -H 'kbn-xsrf:true' -H 'Content-type:application/json' -d @./my-dashboards.json

If you are NOT using x-pack you can omit the --user <username>:<password> portion.

@dkirrane

This comment has been minimized.

Show comment
Hide comment
@dkirrane

dkirrane Sep 12, 2017

curl -XGET http://localhost:5601/app/kibana/dashboards/export?dashboard=33797130-97ae-11e7-aa10-79a79f2e360b
{"statusCode":404,"error":"Not Found"}

The dashboard _id I used is the one returned in id field by this API call
curl -s http://localhost:9200/.kibana/dashboard/_search?pretty

...
      {
        "_index" : ".kibana",
        "_type" : "dashboard",
        "_id" : "33797130-97ae-11e7-aa10-79a79f2e360b",
...

dkirrane commented Sep 12, 2017

curl -XGET http://localhost:5601/app/kibana/dashboards/export?dashboard=33797130-97ae-11e7-aa10-79a79f2e360b
{"statusCode":404,"error":"Not Found"}

The dashboard _id I used is the one returned in id field by this API call
curl -s http://localhost:9200/.kibana/dashboard/_search?pretty

...
      {
        "_index" : ".kibana",
        "_type" : "dashboard",
        "_id" : "33797130-97ae-11e7-aa10-79a79f2e360b",
...
@tylersmalley

This comment has been minimized.

Show comment
Hide comment
@tylersmalley

tylersmalley Sep 12, 2017

Member

@dkirrane, alternatively you can use the Saved Objects API to capture the ID's:

$ curl -s http://localhost:5601/api/saved_objects/dashboard | python -m json.tool
{
    "page": 1,
    "per_page": 20,
    "saved_objects": [
        {
            "attributes": {
                "description": "",
                "hits": 0,
                "kibanaSavedObjectMeta": {
                    "searchSourceJSON": "{\"query\":{\"query\":\"\",\"language\":\"lucene\"},\"filter\":[],\"highlightAll\":true,\"version\":true}"
                },
                "optionsJSON": "{\"darkTheme\":false}",
                "panelsJSON": "[]",
                "timeRestore": false,
                "title": "New Dashboard",
                "uiStateJSON": "{}",
                "version": 1
            },
            "id": "9a8bc880-9805-11e7-aaf0-9134624698e1",
            "type": "dashboard",
            "updated_at": "2017-09-12T21:59:09.319Z",
            "version": 1
        }
    ],
    "total": 1
}
Member

tylersmalley commented Sep 12, 2017

@dkirrane, alternatively you can use the Saved Objects API to capture the ID's:

$ curl -s http://localhost:5601/api/saved_objects/dashboard | python -m json.tool
{
    "page": 1,
    "per_page": 20,
    "saved_objects": [
        {
            "attributes": {
                "description": "",
                "hits": 0,
                "kibanaSavedObjectMeta": {
                    "searchSourceJSON": "{\"query\":{\"query\":\"\",\"language\":\"lucene\"},\"filter\":[],\"highlightAll\":true,\"version\":true}"
                },
                "optionsJSON": "{\"darkTheme\":false}",
                "panelsJSON": "[]",
                "timeRestore": false,
                "title": "New Dashboard",
                "uiStateJSON": "{}",
                "version": 1
            },
            "id": "9a8bc880-9805-11e7-aaf0-9134624698e1",
            "type": "dashboard",
            "updated_at": "2017-09-12T21:59:09.319Z",
            "version": 1
        }
    ],
    "total": 1
}
@dkirrane

This comment has been minimized.

Show comment
Hide comment
@dkirrane

dkirrane Sep 13, 2017

@tylersmalley tried that. Got the id but still returning 404. Tried with both Kibana 5.5.1 and 5.5.2
curl -XGET http://localhost:5601/app/kibana/dashboards/export?dashboard=f2f2a960-97df-11e7-9226-8163619d402b

{
    "error": "Not Found",
    "statusCode": 404
}

dkirrane commented Sep 13, 2017

@tylersmalley tried that. Got the id but still returning 404. Tried with both Kibana 5.5.1 and 5.5.2
curl -XGET http://localhost:5601/app/kibana/dashboards/export?dashboard=f2f2a960-97df-11e7-9226-8163619d402b

{
    "error": "Not Found",
    "statusCode": 404
}
@epixa

This comment has been minimized.

Show comment
Hide comment
@epixa

epixa Sep 13, 2017

Member

Use api in the url rather than app.

Member

epixa commented Sep 13, 2017

Use api in the url rather than app.

@cjauvin

This comment has been minimized.

Show comment
Hide comment
@cjauvin

cjauvin Sep 21, 2017

This API is very handy, but there's something I'm not sure I understand: assuming that the only way to export a dashboard with the UI is to use Management -> Saved Objects -> Export Everything.. then it means that it's not possible to import a UI-exported dashboard via your API, because the two formats are not compatible (in particular, the UI exports a JSON list, while the API requires an object).

Shouldn't the API be accompanied by a UI update which unifies the import/export processes, both from their API and UI perspectives?

cjauvin commented Sep 21, 2017

This API is very handy, but there's something I'm not sure I understand: assuming that the only way to export a dashboard with the UI is to use Management -> Saved Objects -> Export Everything.. then it means that it's not possible to import a UI-exported dashboard via your API, because the two formats are not compatible (in particular, the UI exports a JSON list, while the API requires an object).

Shouldn't the API be accompanied by a UI update which unifies the import/export processes, both from their API and UI perspectives?

@epixa

This comment has been minimized.

Show comment
Hide comment
@epixa

epixa Sep 21, 2017

Member

@cjauvin I suspect the two outputs will converge at some point, but they don't do the same thing at the moment, and we're not yet ready to say for sure whether they will converge or not. "Export everything" is a raw dump of certain types of saved objects, whereas this new import/export API is a more intelligent mechanism to export all related saved objects for a given dashboard.

Out of curiosity, do you have a specific workflow in mind that would benefit from a shared format?

Member

epixa commented Sep 21, 2017

@cjauvin I suspect the two outputs will converge at some point, but they don't do the same thing at the moment, and we're not yet ready to say for sure whether they will converge or not. "Export everything" is a raw dump of certain types of saved objects, whereas this new import/export API is a more intelligent mechanism to export all related saved objects for a given dashboard.

Out of curiosity, do you have a specific workflow in mind that would benefit from a shared format?

@cjauvin

This comment has been minimized.

Show comment
Hide comment
@cjauvin

cjauvin Sep 21, 2017

@epixa Thanks for the quick answer! I do have a specific workflow in mind: we are currently provisioning an ES + Kibana stack running in a Kubernetes cluster, and in this context your API is very handy as a way to automatically provision required dashboards at the cluster creation and setup stage (with the Helm tool, specifically). However, we would like it also to be extra easy for a cluster operator to be able to contribute any dashboard update (say a title change) back into the setup scripts.. and for this simply exporting (from the UI) + copy/pasting would probably be the path of least friction.

cjauvin commented Sep 21, 2017

@epixa Thanks for the quick answer! I do have a specific workflow in mind: we are currently provisioning an ES + Kibana stack running in a Kubernetes cluster, and in this context your API is very handy as a way to automatically provision required dashboards at the cluster creation and setup stage (with the Helm tool, specifically). However, we would like it also to be extra easy for a cluster operator to be able to contribute any dashboard update (say a title change) back into the setup scripts.. and for this simply exporting (from the UI) + copy/pasting would probably be the path of least friction.

@simianhacker

This comment has been minimized.

Show comment
Hide comment
@simianhacker

simianhacker Sep 21, 2017

Member

@epixa I've always envision doing an export from the dashboard listing page, click the checkboxes next to the dashboards you want to export and click the new export button that shows up next to the delete button. And it just redirects you the the API URL with the query arguments for the dashboards. BTW... the export response already has the content-disposition header for treating it as a download. So really all that is left is the UI work. :D

Member

simianhacker commented Sep 21, 2017

@epixa I've always envision doing an export from the dashboard listing page, click the checkboxes next to the dashboards you want to export and click the new export button that shows up next to the delete button. And it just redirects you the the API URL with the query arguments for the dashboards. BTW... the export response already has the content-disposition header for treating it as a download. So really all that is left is the UI work. :D

@epixa

This comment has been minimized.

Show comment
Hide comment
@epixa

epixa Sep 22, 2017

Member

@simianhacker That experience sounds great to me. We can't just jump into it since import needs to accommodate existing json exports, but it sounds like a good goal

Member

epixa commented Sep 22, 2017

@simianhacker That experience sounds great to me. We can't just jump into it since import needs to accommodate existing json exports, but it sounds like a good goal

@yangwong9664

This comment has been minimized.

Show comment
Hide comment
@yangwong9664

yangwong9664 Oct 31, 2017

Great API, really useful for our project, I have no problems with exporting. Do we have any examples of the request body for an import POST?

yangwong9664 commented Oct 31, 2017

Great API, really useful for our project, I have no problems with exporting. Do we have any examples of the request body for an import POST?

@simianhacker

This comment has been minimized.

Show comment
Hide comment
@simianhacker

simianhacker Oct 31, 2017

Member

@yangwong9664 it’s the same as the response from the export.

Member

simianhacker commented Oct 31, 2017

@yangwong9664 it’s the same as the response from the export.

@yangwong9664

This comment has been minimized.

Show comment
Hide comment
@yangwong9664

yangwong9664 commented Oct 31, 2017

@simianhacker Thanks.

@gingerwizard

This comment has been minimized.

Show comment
Hide comment
@gingerwizard

gingerwizard Nov 1, 2017

Is there a way to simply dump all objects through this API? I see the import takes a list of objects.

gingerwizard commented Nov 1, 2017

Is there a way to simply dump all objects through this API? I see the import takes a list of objects.

@gingerwizard

This comment has been minimized.

Show comment
Hide comment
@gingerwizard

gingerwizard commented Nov 1, 2017

ah i see #11632

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment