[ML] Changing all calls to ML endpoints to use internal user #70487

jgowdyelastic · 2020-07-01T17:38:36Z

Changes all calls to ML elasticsearch endpoints so that they are called by kibana's internal user.
All requests to standard es functions, like search are unchanged and use the current user.

All shared function providers now need to supply the cluster client object rather than just the callWithCurrentUser function.

Functions which originally used ml.privilegeCheck to check ML privileges now use the same checks at the routing level.

Calls to these endpoints pass across the original users credentials as a header:
PUT /api/ml/data_frame/analytics/{analyticsId}
POST /api/ml/data_frame/_evaluate
PUT /api/ml/datafeeds/{datafeedId}
POST /api/ml/datafeeds/{datafeedId}/_update
GET /api/ml/datafeeds/{datafeedId}/_preview

When creating a user role, we now have the ability to manage ML capabilities in spaces through the UI:

I believe further refactoring to our shared functions could be done where we would no longer require the cluster client to be passed to us, we could instead scope our own client using the request object passed in on each call to the shared function providers. This work can be done in a follow up PR.

Checklist

Delete any items that are not applicable to this PR.

Unit or functional tests were updated or added to match the most common scenarios
This was checked for breaking API changes and was labeled appropriately

jgowdyelastic · 2020-07-02T16:24:26Z

@elasticmachine merge upstream

…internal-user

…' of github.com:jgowdyelastic/kibana into changing-all-calls-to-ml-endpoints-to-use-internal-user

jgowdyelastic · 2020-07-03T07:33:51Z

@elasticmachine merge upstream

elasticmachine · 2020-07-03T07:33:53Z

merge conflict between base and head

…internal-user

…' of github.com:jgowdyelastic/kibana into changing-all-calls-to-ml-endpoints-to-use-internal-user

jgowdyelastic · 2020-07-05T09:16:49Z

@elasticmachine merge upstream

jgowdyelastic · 2020-07-06T12:52:02Z

@elasticmachine merge upstream

…internal-user

jgowdyelastic · 2020-07-07T14:50:04Z

@elasticmachine merge upstream

elasticmachine · 2020-07-07T14:50:06Z

merge conflict between base and head

darnautov · 2020-07-13T13:30:24Z

x-pack/plugins/ml/public/application/capabilities/check_capabilities.ts

+          window.location.href = '#/jobs';
+          return reject();


I presume there is no need to reject the promise when you perform a hard refresh. Would be better to use navigateToUrl instead,

jgowdyelastic · 2020-07-13T14:30:33Z

@elasticmachine merge upstream

jgowdyelastic · 2020-07-13T14:32:16Z

@elasticmachine merge upstream

rylnd

Detections changes look good! As mentioned offline we'll need to update our telemetry once this goes in, but we look good there as well 👍

jgowdyelastic · 2020-07-13T18:08:40Z

@elasticmachine merge upstream

…internal-user

jgowdyelastic · 2020-07-14T05:46:45Z

retest

jgowdyelastic · 2020-07-14T07:30:12Z

@elasticmachine merge upstream

…internal-user

jgowdyelastic · 2020-07-14T11:40:51Z

@elasticmachine merge upstream

…internal-user

legrego · 2020-07-14T12:47:23Z

x-pack/plugins/ml/common/types/capabilities.ts

+    catalogue: [PLUGIN_ID],
+    savedObject: {
+      all: [],
+      read: [],


As discussed offline, let's grant access to index patterns and saved searches to users of this feature:

Suggested change

read: [],

read: ['index-pattern', 'search'],

added in b150246

legrego

Thanks for the edits, LGTM.

I opened #71647 to address the current requirement on the monitor cluster privilege. It would be great if we could solve for this in a followup shortly after merging

kibanamachine · 2020-07-14T14:39:03Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: b150246

Build metrics

✅ unchanged

History

💚 Build #61337 succeeded b343a28
💔 Build #61328 failed d49b0ad
💚 Build #61284 succeeded 9d236d3
💔 Build #61259 failed 83c507c
💚 Build #61241 succeeded 9d55bf2

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…#70487) * [ML] Changing all calls to ML endpoints to use internal user * updating alerting * updating documentation * [ML] Changing all calls to ML endpoints to use internal user * updating alerting * updating documentation * fixing missed types * adding authorization headers to endpoint calls * correcting has privileges call * updating security tests * odd eslint error * adding auth header to module setup * fixing missing auth argument * fixing delete DFA job permission checks * removing debug test tag * removing additional ml privilege checks * adding authorization header to _evaluate * updating alerting cluster client name * code clean up * changing authorizationHeader name * updating alterting documentation * fixing secondary credentials * adding management links * updating SIEM telemetry * fixing merge conflicts * granting access to index patterns Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* master: (21 commits) [Maps] 7.9 design improvements (elastic#71563) [ML] Changing all calls to ML endpoints to use internal user (elastic#70487) [eventLog] prevent log writing when initialization fails (elastic#71339) [Observability] landing page always being displayed (elastic#71494) [IM] Address data stream copy feedback (elastic#71615) [Logs UI] Anomalies page dataset filtering (elastic#71110) [data.search.aggs] Remove `use_field_mapping` from top hits agg (elastic#71168) [ML] Anomaly swim lane embeddable navigation and filter actions (elastic#71082) Fixes typo in siem_cloudtrail job description (elastic#71569) Require granted API Keys to have a name (elastic#71623) Update getUsageForCollection (elastic#71609) Only fetch saved elements once (elastic#71310) [SecuritySolution][Resolver] Adding siem index and guarding process ancestry (elastic#71570) [APM] Additional data telemetry changes (elastic#71112) [Visualize] Fix export table for table export links (elastic#71249) [Search] Server side search API (elastic#70446) use inclusive language (elastic#71607) [Security Solution] Hide timeline footer when Resolver is open (elastic#71516) [Index template wizard] Remove shadow and use border for components panels (elastic#71606) [ML] Kibana API endpoint for histogram chart data (elastic#70976) ...

…#71667) * [ML] Changing all calls to ML endpoints to use internal user * updating alerting * updating documentation * [ML] Changing all calls to ML endpoints to use internal user * updating alerting * updating documentation * fixing missed types * adding authorization headers to endpoint calls * correcting has privileges call * updating security tests * odd eslint error * adding auth header to module setup * fixing missing auth argument * fixing delete DFA job permission checks * removing debug test tag * removing additional ml privilege checks * adding authorization header to _evaluate * updating alerting cluster client name * code clean up * changing authorizationHeader name * updating alterting documentation * fixing secondary credentials * adding management links * updating SIEM telemetry * fixing merge conflicts * granting access to index patterns Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

jgowdyelastic added 6 commits July 1, 2020 18:36

[ML] Changing all calls to ML endpoints to use internal user

06b3200

updating alerting

678bbda

updating documentation

77dfc92

[ML] Changing all calls to ML endpoints to use internal user

c24c0e2

updating alerting

0aabf8c

updating documentation

4420652

elasticmachine and others added 2 commits July 2, 2020 10:24

Merge branch 'master' into changing-all-calls-to-ml-endpoints-to-use-…

6995887

…internal-user

Merge branch 'changing-all-calls-to-ml-endpoints-to-use-internal-user…

7607a2a

…' of github.com:jgowdyelastic/kibana into changing-all-calls-to-ml-endpoints-to-use-internal-user

jgowdyelastic added 5 commits July 3, 2020 09:11

Merge branch 'master' into changing-all-calls-to-ml-endpoints-to-use-…

898726c

…internal-user

Merge branch 'changing-all-calls-to-ml-endpoints-to-use-internal-user…

b579a26

…' of github.com:jgowdyelastic/kibana into changing-all-calls-to-ml-endpoints-to-use-internal-user

fixing missed types

59e5c04

adding authorization headers to endpoint calls

4dec2d2

correcting has privileges call

899977f

jgowdyelastic mentioned this pull request Jul 4, 2020

[ML] Integration with Kibana Spaces #64172

Closed

26 tasks

jgowdyelastic added 6 commits July 5, 2020 10:43

updating security tests

79c9235

odd eslint error

9e433d0

adding auth header to module setup

071f4fd

fixing missing auth argument

f4b4ef4

fixing delete DFA job permission checks

0722486

removing debug test tag

d40fdd4

elasticmachine and others added 3 commits July 6, 2020 06:52

Merge branch 'master' into changing-all-calls-to-ml-endpoints-to-use-…

c94d504

…internal-user

removing additional ml privilege checks

abcc53c

adding authorization header to _evaluate

0fb8c1b

darnautov reviewed Jul 13, 2020

View reviewed changes

jgowdyelastic added 2 commits July 13, 2020 16:47

fixing secondary credentials

f52200f

adding management links

c820cfd

rylnd approved these changes Jul 13, 2020

View reviewed changes

Merge branch 'master' into changing-all-calls-to-ml-endpoints-to-use-…

9d55bf2

…internal-user

elasticmachine and others added 3 commits July 14, 2020 01:30

Merge branch 'master' into changing-all-calls-to-ml-endpoints-to-use-…

83c507c

…internal-user

updating SIEM telemetry

9d236d3

Merge branch 'master' into changing-all-calls-to-ml-endpoints-to-use-…

5f9d34c

…internal-user

elasticmachine and others added 2 commits July 14, 2020 05:40

Merge branch 'master' into changing-all-calls-to-ml-endpoints-to-use-…

d49b0ad

…internal-user

fixing merge conflicts

b343a28

legrego self-requested a review July 14, 2020 12:31

legrego reviewed Jul 14, 2020

View reviewed changes

granting access to index patterns

b150246

legrego mentioned this pull request Jul 14, 2020

[ML] Machine Learning should not require the monitor cluster privilege to create a data frame job #71647

Closed

qn895 approved these changes Jul 14, 2020

View reviewed changes

legrego approved these changes Jul 14, 2020

View reviewed changes

jgowdyelastic merged commit a1e511a into elastic:master Jul 14, 2020

jgowdyelastic deleted the changing-all-calls-to-ml-endpoints-to-use-internal-user branch July 14, 2020 14:49

jgowdyelastic mentioned this pull request Jul 14, 2020

[7.x] [ML] Changing all calls to ML endpoints to use internal user (#70487) #71667

Merged

lcawl mentioned this pull request Jul 21, 2020

Managing access to machine learning Kibana Spaces elastic/stack-docs#1303

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Changing all calls to ML endpoints to use internal user #70487

[ML] Changing all calls to ML endpoints to use internal user #70487

jgowdyelastic commented Jul 1, 2020 •

edited

Loading

jgowdyelastic commented Jul 2, 2020

jgowdyelastic commented Jul 3, 2020

elasticmachine commented Jul 3, 2020

jgowdyelastic commented Jul 5, 2020

jgowdyelastic commented Jul 6, 2020

jgowdyelastic commented Jul 7, 2020

elasticmachine commented Jul 7, 2020

darnautov Jul 13, 2020

jgowdyelastic commented Jul 13, 2020

jgowdyelastic commented Jul 13, 2020

rylnd left a comment

jgowdyelastic commented Jul 13, 2020

jgowdyelastic commented Jul 14, 2020

jgowdyelastic commented Jul 14, 2020

jgowdyelastic commented Jul 14, 2020

legrego Jul 14, 2020

jgowdyelastic Jul 14, 2020

legrego left a comment

kibanamachine commented Jul 14, 2020

[ML] Changing all calls to ML endpoints to use internal user #70487

[ML] Changing all calls to ML endpoints to use internal user #70487

Conversation

jgowdyelastic commented Jul 1, 2020 • edited Loading

Checklist

jgowdyelastic commented Jul 2, 2020

jgowdyelastic commented Jul 3, 2020

elasticmachine commented Jul 3, 2020

jgowdyelastic commented Jul 5, 2020

jgowdyelastic commented Jul 6, 2020

jgowdyelastic commented Jul 7, 2020

elasticmachine commented Jul 7, 2020

darnautov Jul 13, 2020

Choose a reason for hiding this comment

jgowdyelastic commented Jul 13, 2020

jgowdyelastic commented Jul 13, 2020

rylnd left a comment

Choose a reason for hiding this comment

jgowdyelastic commented Jul 13, 2020

jgowdyelastic commented Jul 14, 2020

jgowdyelastic commented Jul 14, 2020

jgowdyelastic commented Jul 14, 2020

legrego Jul 14, 2020

Choose a reason for hiding this comment

jgowdyelastic Jul 14, 2020

Choose a reason for hiding this comment

legrego left a comment

Choose a reason for hiding this comment

kibanamachine commented Jul 14, 2020

💚 Build Succeeded

Build metrics

History

jgowdyelastic commented Jul 1, 2020 •

edited

Loading