Update logstash pipeline management to use system index APIs #80405

jaymode · 2020-10-13T19:43:20Z

Summary

This change updates the logstash pipeline management plugin to use
pipeline management APIs in Elasticsearch rather than directly
accessing the .logstash index. In Elasticsearch 8.0, direct access to
system indices will no longer be allowed when using standard APIs.
Given this change, a new set of APIs has been created specifically for
the management of Logstash pipelines and this change makes use of the
APIs.

Relates elastic/elasticsearch#53350

Checklist

Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

This change updates the logstash pipeline management plugin to use pipeline management APIs in Elasticsearch rather than directly accessing the .logstash index. In Elasticsearch 8.0, direct access to system indices will no longer be allowed when using standard APIs. Given this change, a new set of APIs has been created specifically for the management of Logstash pipelines and this change makes use of the APIs. Relates elastic/elasticsearch#53350

kaisecheng · 2020-10-13T20:19:20Z

I have a concern about backward compatibility when this Kibana connects to Elasticsearch version < 7.9. /_logstash doesn't exist in the old version. In Logstash, we maintain both legacy API and the new system indices API for < 7.9 and 7.9+

For the role management, I am not quite sure but I think manage_logstash_pipeline need to add around here https://github.com/elastic/kibana/blob/master/x-pack/plugins/logstash/server/plugin.ts#L47

I am also new to Kibana. It would be great if a Kibana member can have a look.

jaymode · 2020-10-14T15:16:37Z

I have a concern about backward compatibility when this Kibana connects to Elasticsearch version < 7.9. /_logstash doesn't exist in the old version

From the docs:

Running different major version releases of Kibana and Elasticsearch (e.g.
Kibana 5.x and Elasticsearch 2.x) is not supported, nor is running a minor
version of Kibana that is newer than the version of Elasticsearch (e.g. Kibana
5.1 and Elasticsearch 5.0).

Given this moving Kibana to use these APIs is ok. Elasticsearch will have the APIs in 7.10+ and Kibana should always be the same version as Elasticsearch or older within the same minor, which should only be temporary for upgrades.

For the role management, I am not quite sure but I think manage_logstash_pipeline need to add around here

Good catch.

kaisecheng · 2020-10-14T15:41:00Z

@jaymode Will you continue your work on role management? Or you would like me to finish the rest of this PR?

jaymode · 2020-10-14T15:43:42Z

@kaisecheng if you don't mind finishing the PR that would be great! I've added you as a collaborator to my repository.

kaisecheng · 2020-10-15T13:38:40Z

@elasticmachine merge upstream

…a into logstash_system_index_apis

roaksoax · 2020-10-21T22:41:57Z

@elasticmachine merge upstream

roaksoax · 2020-10-28T14:58:09Z

@elasticmachine merge upstream

roaksoax · 2020-11-04T15:25:40Z

@elasticmachine merge upstream

…em_index_apis

mshustov · 2020-11-10T08:15:23Z

x-pack/plugins/logstash/server/routes/pipeline/delete.ts

-          index: INDEX_NAMES.PIPELINES,
-          id: request.params.id,
-          refresh: 'wait_for',
+        await client.callAsCurrentUser('transport.request', {


Logstash plugin needs to be migrated to the new Elasticsearch client https://github.com/elastic/elasticsearch-js as we are going to remove the old one in v8.0. Changelog: https://github.com/elastic/kibana/blob/master/src/core/MIGRATION_EXAMPLES.md#elasticsearch-client

transport.request doesn't provide type safety.

The current implementation doesn't prefix endpoints with _kibana as outlined in the migration path for 7.x Kibana's system indices #81536

In #82716, we are going to provide a separate Elasticsearch client to address 2 & 3 points.
Logstash plugin will:

migrate to provided SystemIndices Elasticsearch client

adopt all usage places to make sure compatibility with the new Elasticsearch client version

migrate to /_logstash/pipeline/ endpoints (as done in this PR)

How soon do you want to migrate to System Indices? Can you wait for #82716?
@jaymode @kaisecheng

Regarding 3, these APIs will not be prefixed with _kibana as they exist in their own plugin within Elasticsearch and not within the Kibana system index plugin.

Ideally, I'd like to see us move the Logstash UI to use the system index APIs sooner rather than later to get more of the system index work into users' hands.

Regarding 3, these APIs will not be prefixed with _kibana as they exist in their own plugin within Elasticsearch and not within the Kibana system index plugin.

Makes sense to merge it then. You just need to address 1st point then. Can be done in the follow-up.

Hopefully we can fix the 2nd problem as well with #80405 (comment)

I will address 1st point as a follow-up issue

@restrry is there a meta issue/transition plan to support the new elasticsearch-js client library (not just for logstash)?

@roaksoax #83910

mshustov · 2020-11-10T08:48:28Z

x-pack/plugins/logstash/server/routes/pipeline/delete.ts

-          id: request.params.id,
-          refresh: 'wait_for',
+        await client.callAsCurrentUser('transport.request', {
+          path: '/_logstash/pipeline/' + encodeURIComponent(request.params.id),


@delvedor I can't find /_logstash support in the new Elasticsearch client. Why it so?
https://github.com/elastic/elasticsearch-js/blob/master/api/kibana.d.ts

I'm not aware of that endpoint, and I wasn't able to find it in the rest-api-spec (and here). Are you sure it's a public API?

I can't see them in the spec either, but they present in the code
https://github.com/elastic/elasticsearch/blob/fb7471b32955c80f1fe850054d68d5b66aca2896/x-pack/plugin/logstash/src/main/java/org/elasticsearch/xpack/logstash/rest/RestGetPipelineAction.java#L32
https://github.com/elastic/elasticsearch/blob/fb7471b32955c80f1fe850054d68d5b66aca2896/x-pack/plugin/logstash/src/main/java/org/elasticsearch/xpack/logstash/rest/RestPutPipelineAction.java#L35
https://github.com/elastic/elasticsearch/blob/fb7471b32955c80f1fe850054d68d5b66aca2896/x-pack/plugin/logstash/src/main/java/org/elasticsearch/xpack/logstash/rest/RestDeletePipelineAction.java#L34

The intent for these Logstash APIs is that they are for communication between the stack components and Elasticsearch so we intentionally did not publish a rest api spec for these. If this is necessary, we can probably add specs for these APIs.

communication between the stack components and Elasticsearch so we intentionally did not publish a rest api spec for these.

Can we add a separate REST API spec for such cases? We need to make sure that all the changes in REST API are reflected in the Kibana code as well - we rely on TypeScript type definitions generated from REST API spec.

++ another vote on this. I'd be nice to leverage the safety net y'all created with the API spec, but for product to product APIs

Sorry for missing an update here, the ES team discussed this and came to the conclusion that we'd add a spec and docs for this API. This work hasn't been started yet.

The API spec files have been merged in elastic/elasticsearch#67788

kaisecheng · 2020-11-11T15:45:34Z

@elasticmachine merge upstream

kibanamachine · 2020-11-11T17:33:33Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: c124c34

Metrics [docs]

Distributable file count

id	before	after	diff
`default`	42778	42776	-2

History

💚 Build #85907 succeeded 70d5435
💔 Build #85835 failed 10a96d6
💛 Build #84304 was flaky fab9a9e
💛 Build #83228 was flaky bf61657
💚 Build #81906 succeeded c50d984

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…#80405) This change updates the logstash pipeline management plugin to use pipeline management APIs in Elasticsearch rather than directly accessing the .logstash index. In Elasticsearch 8.0, direct access to system indices will no longer be allowed when using standard APIs. Given this change, a new set of APIs has been created specifically for the management of Logstash pipelines and this change makes use of the APIs. Co-authored-by: Kaise Cheng <kaise.cheng@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> ES:elastic#53350 LS:elastic#12291

…ts-public * upstream/master: (57 commits) Remove unused asciidoc file (elastic#83228) [Lens] Remove background from lens embeddable (elastic#83061) [Discover] Unskip flaky tests based on discover fixture index pattern (elastic#82991) Removing unnecessary trailing slash in CODEOWNERS Trying to fix CODEOWNERS again, where was a non-existent team prior (elastic#83236) Trying to fix CODEOWERS, missing a starting slash (elastic#83233) skip flaky suite (elastic#83231) Add enzyme rerender test helper (elastic#83208) Move Elasticsearch type definitions out of APM (elastic#83081) [ts/checkTsProjects] produce a more useful error message (elastic#83209) [kbnClient] retry updating config if necessary (elastic#83205) I accidentally removed this line in a recent PR (elastic#83201) Don't make the caller do work the function can do (elastic#83180) [App Search] Update EngineRouter & EngineNav to use EngineLogic (elastic#83138) [Workplace Search] Add routes for Sources (elastic#83125) Update logstash pipeline management to use system index APIs (elastic#80405) [ML] Replace EuiBasicTable with EuiInMemoryTable (elastic#83057) [Metrics UI] Add basic interaction and shell for node details overlay (elastic#82013) [App Search] Added the log retention confirmation modal to the Settings page (elastic#83009) [docs] Fix create map title in import geospatial page (elastic#83172) ...

kibanamachine · 2020-11-13T18:04:39Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 80405 or prevent reminders by adding the backport:skip label.

kibanamachine · 2020-11-16T18:14:39Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 80405 or prevent reminders by adding the backport:skip label.

…#80405) This change updates the logstash pipeline management plugin to use pipeline management APIs in Elasticsearch rather than directly accessing the .logstash index. In Elasticsearch 8.0, direct access to system indices will no longer be allowed when using standard APIs. Given this change, a new set of APIs has been created specifically for the management of Logstash pipelines and this change makes use of the APIs. Co-authored-by: Kaise Cheng <kaise.cheng@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> ES:elastic#53350 LS:elastic#12291

…#83526) This change updates the logstash pipeline management plugin to use pipeline management APIs in Elasticsearch rather than directly accessing the .logstash index. In Elasticsearch 8.0, direct access to system indices will no longer be allowed when using standard APIs. Given this change, a new set of APIs has been created specifically for the management of Logstash pipelines and this change makes use of the APIs. Co-authored-by: Kaise Cheng <kaise.cheng@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> ES:#53350 LS:#12291 Co-authored-by: Jay Modi <jaymode@users.noreply.github.com>

jaymode added the review label Oct 13, 2020

updated privileges. fixed ui tests

8eb4cc2

Merge branch 'master' into logstash_system_index_apis

43d7d14

kaisecheng added the release_note:enhancement label Oct 15, 2020

kaisecheng added 2 commits October 15, 2020 16:32

fix eslint

c4455ba

Merge branch 'logstash_system_index_apis' of github.com:jaymode/kiban…

c50d984

…a into logstash_system_index_apis

kaisecheng mentioned this pull request Oct 15, 2020

[Meta] Logstash use system indices API elastic/logstash#12291

Closed

8 tasks

kaisecheng added the v7.11.0 label Oct 15, 2020

Merge branch 'master' into logstash_system_index_apis

bf61657

Merge branch 'master' into logstash_system_index_apis

fab9a9e

kaisecheng added a commit to kaisecheng/kibana that referenced this pull request Nov 2, 2020

cherry pick from elastic#80405

ca53319

kaisecheng mentioned this pull request Nov 2, 2020

Fix pipeline management metadata versioning #82255

Closed

1 task

kibanamachine and others added 2 commits November 4, 2020 10:25

Merge branch 'master' into logstash_system_index_apis

10a96d6

Merge branch 'master' of github.com:elastic/kibana into logstash_syst…

70d5435

…em_index_apis

mshustov reviewed Nov 10, 2020

View reviewed changes

mshustov approved these changes Nov 11, 2020

View reviewed changes

Merge branch 'master' into logstash_system_index_apis

c124c34

kaisecheng merged commit b460414 into elastic:master Nov 11, 2020

mshustov mentioned this pull request Nov 12, 2020

Failing test: Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/logstash/pipeline_create·js - logstash pipeline create new cancel button discards the pipeline and redirects to the list #83231

Open

kaisecheng mentioned this pull request Nov 12, 2020

fix logstash central pipeline management test #83281

Merged

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Nov 13, 2020

kaisecheng mentioned this pull request Nov 17, 2020

[7.x] Update logstash pipeline management to use system index APIs (#80405) #83526

Merged

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Nov 17, 2020

mshustov mentioned this pull request Nov 18, 2020

Core should expose a dedicated Elasticsearch client for interacting with Kibana system indices #82716

Closed

tylersmalley mentioned this pull request Dec 16, 2020

Failing ES Promotion: logstash pipeline list route add button links to the empty pipeline editor #86038

Closed

jaymode deleted the logstash_system_index_apis branch January 28, 2021 17:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update logstash pipeline management to use system index APIs #80405

Update logstash pipeline management to use system index APIs #80405

jaymode commented Oct 13, 2020

kaisecheng commented Oct 13, 2020

jaymode commented Oct 14, 2020

kaisecheng commented Oct 14, 2020

jaymode commented Oct 14, 2020

kaisecheng commented Oct 15, 2020

roaksoax commented Oct 21, 2020

roaksoax commented Oct 28, 2020

roaksoax commented Nov 4, 2020

mshustov Nov 10, 2020 •

edited

Loading

jaymode Nov 10, 2020

mshustov Nov 11, 2020

kaisecheng Nov 11, 2020

roaksoax Nov 17, 2020 •

edited

Loading

mshustov Nov 26, 2020

mshustov Nov 10, 2020

delvedor Nov 10, 2020 •

edited

Loading

mshustov Nov 10, 2020 •

edited

Loading

jaymode Nov 10, 2020

mshustov Nov 11, 2020

jsvd Dec 14, 2020

jaymode Dec 14, 2020

jaymode Jan 28, 2021

kaisecheng commented Nov 11, 2020

kibanamachine commented Nov 11, 2020

kibanamachine commented Nov 13, 2020

kibanamachine commented Nov 16, 2020

Update logstash pipeline management to use system index APIs #80405

Update logstash pipeline management to use system index APIs #80405

Conversation

jaymode commented Oct 13, 2020

Summary

Checklist

For maintainers

kaisecheng commented Oct 13, 2020

jaymode commented Oct 14, 2020

kaisecheng commented Oct 14, 2020

jaymode commented Oct 14, 2020

kaisecheng commented Oct 15, 2020

roaksoax commented Oct 21, 2020

roaksoax commented Oct 28, 2020

roaksoax commented Nov 4, 2020

mshustov Nov 10, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roaksoax Nov 17, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

delvedor Nov 10, 2020 • edited Loading

Choose a reason for hiding this comment

mshustov Nov 10, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaisecheng commented Nov 11, 2020

kibanamachine commented Nov 11, 2020

💚 Build Succeeded

Metrics [docs]

Distributable file count

History

kibanamachine commented Nov 13, 2020

kibanamachine commented Nov 16, 2020

mshustov Nov 10, 2020 •

edited

Loading

roaksoax Nov 17, 2020 •

edited

Loading

delvedor Nov 10, 2020 •

edited

Loading

mshustov Nov 10, 2020 •

edited

Loading