[ML] Explain data frame analytics API #49455

dimitris-athanasiou · 2019-11-21T17:44:39Z

This commit replaces the _estimate_memory_usage API with
a new API, the _explain API.

The API consolidates information that is useful before
creating a data frame analytics job.

It includes:

memory estimation
field selection explanation

Memory estimation is moved here from what was previously
calculated in the _estimate_memory_usage API.

Field selection is a new feature that explains to the user
whether each available field was selected to be included or
not in the analysis. In the case it was not included, it also
explains the reason why.

elasticmachine · 2019-11-21T17:44:42Z

Pinging @elastic/ml-core (:ml)

dimitris-athanasiou · 2019-11-21T17:45:09Z

docs/reference/ml/df-analytics/apis/dfanalytics-info.asciidoc

+----
+{
+  "field_selection": [
+


I'll come back to add this

I have now added this in

dimitris-athanasiou · 2019-11-21T17:45:48Z

@alvarezmelissa87 pinging you as this is the PR that adds field selection explanation

dimitris-athanasiou · 2019-11-21T17:46:21Z

@szabosteve I'm updating documentation too in this PR, could you please take a look at the docs changes?

benwtrent

Great stuff! Some minor comments.

Also, I think you might need to black list the failure focused yaml tests for the ml with security tests

...high-level/src/test/java/org/elasticsearch/client/documentation/MlClientDocumentationIT.java

...l/src/main/java/org/elasticsearch/xpack/ml/action/TransportDataFrameAnalyticsInfoAction.java

...ml/src/main/java/org/elasticsearch/xpack/ml/dataframe/extractor/ExtractedFieldsDetector.java

...rc/main/java/org/elasticsearch/xpack/ml/rest/dataframe/RestDataFrameAnalyticsInfoAction.java

szabosteve

Thank you @dimitris-athanasiou for documenting this! I left a couple of minor comments, but it LGTM.

docs/reference/ml/df-analytics/apis/dfanalytics-info.asciidoc

x-pack/plugin/src/test/resources/rest-api-spec/test/ml/data_frame_analytics_info.yml

przemekwitek · 2019-11-22T09:34:09Z

...n/java/org/elasticsearch/xpack/ml/dataframe/process/MemoryUsageEstimationProcessManager.java

@@ -100,9 +100,9 @@ private MemoryUsageEstimationResult runJob(String jobId,
        } finally {
            process.consumeAndCloseOutputStream();
            try {
-                LOGGER.info("[{}] Closing process", jobId);
+                LOGGER.debug("[{}] Closing process", jobId);


Should these be kept in sync with log-levels in AnalyticsProcessManager.java (which are currently info)?

As this API is not part of the life-cycle of the job, I'd rather it stays quiet in the info level unless something goes wrong.

...rc/main/java/org/elasticsearch/xpack/ml/rest/dataframe/RestDataFrameAnalyticsInfoAction.java

...l/src/main/java/org/elasticsearch/xpack/ml/action/TransportDataFrameAnalyticsInfoAction.java

przemekwitek · 2019-11-22T10:00:58Z

...rest-high-level/src/main/java/org/elasticsearch/client/ml/dataframe/info/FieldSelection.java

+    FieldSelection(String name, Set<String> mappingTypes, boolean isIncluded, boolean isRequired, @Nullable FeatureType featureType,
+                           @Nullable String reason) {
+        this.name = Objects.requireNonNull(name);
+        this.mappingTypes = Collections.unmodifiableSet(mappingTypes);


Objects.requireNonNull(mappingTypes)
?

Collections.unmodifiableSet throws if it's passed null

przemekwitek · 2019-11-22T10:02:02Z

...high-level/src/test/java/org/elasticsearch/client/ml/dataframe/info/FieldSelectionTests.java

+            .stream().collect(Collectors.toSet());
+        FieldSelection.FeatureType featureType = randomBoolean() ? null : randomFrom(FieldSelection.FeatureType.values());
+        String reason = randomBoolean() ? null : randomAlphaOfLength(20);
+        return new FieldSelection(randomAlphaOfLength(10),


Suggested change

return new FieldSelection(randomAlphaOfLength(10),

return new FieldSelection(

randomAlphaOfLength(10),

przemekwitek · 2019-11-22T10:03:48Z

...plugin/core/src/main/java/org/elasticsearch/xpack/core/ml/dataframe/info/FieldSelection.java

+    FieldSelection(String name, Set<String> mappingTypes, boolean isIncluded, boolean isRequired, @Nullable FeatureType featureType,
+                           @Nullable String reason) {
+        this.name = Objects.requireNonNull(name);
+        this.mappingTypes = Collections.unmodifiableSet(mappingTypes);


Objects.requireNonNull?
Also, should we use ExceptionHelper.requireNonNull instead?

ExceptionsHelper.requireNonNull makes sense for user facing code. We thus use it when we parse request objects. In this case we create this ourselves Objects.requireNonNull should be enough.

przemekwitek · 2019-11-22T10:04:24Z

...n/core/src/test/java/org/elasticsearch/xpack/core/ml/dataframe/info/FieldSelectionTests.java

+            .stream().collect(Collectors.toSet());
+        FieldSelection.FeatureType featureType = randomBoolean() ? null : randomFrom(FieldSelection.FeatureType.values());
+        String reason = randomBoolean() ? null : randomAlphaOfLength(20);
+        return new FieldSelection(randomAlphaOfLength(10),


Suggested change

return new FieldSelection(randomAlphaOfLength(10),

return new FieldSelection(

randomAlphaOfLength(10),

docs/reference/ml/df-analytics/apis/dfanalytics-info.asciidoc

This commit replaces the _estimate_memory_usage API with a new API, the _info API. The API consolidates information that is useful before creating a data frame analytics job. It includes: - memory estimation - field selection explanation Memory estimation is moved here from what was previously calculated in the _estimate_memory_usage API. Field selection is a new feature that explains to the user whether each available field was selected to be included or not in the analysis. In the case it was not included, it also explains the reason why.

…documentation/MlClientDocumentationIT.java Co-Authored-By: Benjamin Trent <ben.w.trent@gmail.com>

…frame/extractor/ExtractedFieldsDetector.java Co-Authored-By: Benjamin Trent <ben.w.trent@gmail.com>

…/dataframe/RestDataFrameAnalyticsInfoAction.java Co-Authored-By: Benjamin Trent <ben.w.trent@gmail.com>

Co-Authored-By: István Zoltán Szabó <istvan.szabo@elastic.co>

dimitris-athanasiou · 2019-11-22T15:37:06Z

run elasticsearch-ci/1

…arch#49455

This commit replaces the _estimate_memory_usage API with a new API, the _explain API. The API consolidates information that is useful before creating a data frame analytics job. It includes: - memory estimation - field selection explanation Memory estimation is moved here from what was previously calculated in the _estimate_memory_usage API. Field selection is a new feature that explains to the user whether each available field was selected to be included or not in the analysis. In the case it was not included, it also explains the reason why. Backport of elastic#49455

This commit replaces the _estimate_memory_usage API with a new API, the _explain API. The API consolidates information that is useful before creating a data frame analytics job. It includes: - memory estimation - field selection explanation Memory estimation is moved here from what was previously calculated in the _estimate_memory_usage API. Field selection is a new feature that explains to the user whether each available field was selected to be included or not in the analysis. In the case it was not included, it also explains the reason why. Backport of #49455

przemekwitek

LGTM

dimitris-athanasiou added >feature :ml Machine learning v8.0.0 v7.6.0 labels Nov 21, 2019

dimitris-athanasiou commented Nov 21, 2019

View reviewed changes

benwtrent reviewed Nov 21, 2019

View reviewed changes

szabosteve approved these changes Nov 22, 2019

View reviewed changes

docs/reference/ml/df-analytics/apis/dfanalytics-info.asciidoc Outdated Show resolved Hide resolved

docs/reference/ml/df-analytics/apis/dfanalytics-info.asciidoc Outdated Show resolved Hide resolved

docs/reference/ml/df-analytics/apis/dfanalytics-info.asciidoc Outdated Show resolved Hide resolved

szabosteve reviewed Nov 22, 2019

View reviewed changes

docs/reference/ml/df-analytics/apis/dfanalytics-info.asciidoc Outdated Show resolved Hide resolved

przemekwitek self-requested a review November 22, 2019 09:24

przemekwitek reviewed Nov 22, 2019

View reviewed changes

benwtrent approved these changes Nov 22, 2019

View reviewed changes

dimitris-athanasiou changed the title ~~[ML] Data frame analytics info API~~ [ML] Explain data frame analytics API Nov 22, 2019

dimitris-athanasiou and others added 9 commits November 22, 2019 17:11

Update client/rest-high-level/src/test/java/org/elasticsearch/client/…

37f2dc5

…documentation/MlClientDocumentationIT.java Co-Authored-By: Benjamin Trent <ben.w.trent@gmail.com>

Update client/rest-high-level/src/test/java/org/elasticsearch/client/…

16301df

…documentation/MlClientDocumentationIT.java Co-Authored-By: Benjamin Trent <ben.w.trent@gmail.com>

Update x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/data…

3e3c312

…frame/extractor/ExtractedFieldsDetector.java Co-Authored-By: Benjamin Trent <ben.w.trent@gmail.com>

Update x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/rest…

4209da1

…/dataframe/RestDataFrameAnalyticsInfoAction.java Co-Authored-By: Benjamin Trent <ben.w.trent@gmail.com>

Update docs/reference/ml/df-analytics/apis/dfanalytics-info.asciidoc

f3c6eca

Co-Authored-By: István Zoltán Szabó <istvan.szabo@elastic.co>

Update docs/reference/ml/df-analytics/apis/dfanalytics-info.asciidoc

36233c5

Co-Authored-By: István Zoltán Szabó <istvan.szabo@elastic.co>

Update docs/reference/ml/df-analytics/apis/dfanalytics-info.asciidoc

8de503d

Co-Authored-By: István Zoltán Szabó <istvan.szabo@elastic.co>

Update docs/reference/ml/df-analytics/apis/dfanalytics-info.asciidoc

7471169

Co-Authored-By: István Zoltán Szabó <istvan.szabo@elastic.co>

dimitris-athanasiou force-pushed the df-analytics-info-api branch from 666e764 to 0096136 Compare November 22, 2019 15:12

dimitris-athanasiou added 4 commits November 22, 2019 17:12

Rename to _explain API

408e1ff

Improve error message using either

daab12f

Fix is_included to is_required in docs

807af2c

Add license check

0096136

delvedor added a commit to elastic/elasticsearch-js that referenced this pull request Nov 22, 2019

Removed ml api from reference.asciidoc to fix CI in elastic/elasticse…

e8aebee

…arch#49455

Fix YAML tests

dd5a4d8

dimitris-athanasiou merged commit 0390ec3 into elastic:master Nov 22, 2019

dimitris-athanasiou deleted the df-analytics-info-api branch November 22, 2019 18:08

dimitris-athanasiou mentioned this pull request Nov 22, 2019

[7.x][ML] Explain data frame analytics API (#49455) #49504

Merged

alvarezmelissa87 mentioned this pull request Nov 22, 2019

[ML] DF analytics: Ensure model memory usage estimate can be fetched elastic/kibana#51501

Closed

przemekwitek reviewed Nov 25, 2019

View reviewed changes

This was referenced Feb 3, 2020

[meta] 7.6 release elastic/elasticsearch-net#4340

Closed

[meta] 7.6 release elastic/elasticsearch-net#4341

Closed

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Explain data frame analytics API #49455

[ML] Explain data frame analytics API #49455

dimitris-athanasiou commented Nov 21, 2019 •

edited

Loading

elasticmachine commented Nov 21, 2019

dimitris-athanasiou Nov 21, 2019

dimitris-athanasiou Nov 22, 2019

dimitris-athanasiou commented Nov 21, 2019

dimitris-athanasiou commented Nov 21, 2019 •

edited

Loading

benwtrent left a comment

szabosteve left a comment

przemekwitek Nov 22, 2019

dimitris-athanasiou Nov 22, 2019

przemekwitek Nov 22, 2019

dimitris-athanasiou Nov 22, 2019

przemekwitek Nov 22, 2019

przemekwitek Nov 22, 2019

dimitris-athanasiou Nov 22, 2019

przemekwitek Nov 22, 2019

dimitris-athanasiou commented Nov 22, 2019

przemekwitek left a comment

	return new FieldSelection(randomAlphaOfLength(10),
	return new FieldSelection(
	randomAlphaOfLength(10),

[ML] Explain data frame analytics API #49455

[ML] Explain data frame analytics API #49455

Conversation

dimitris-athanasiou commented Nov 21, 2019 • edited Loading

elasticmachine commented Nov 21, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dimitris-athanasiou commented Nov 21, 2019

dimitris-athanasiou commented Nov 21, 2019 • edited Loading

benwtrent left a comment

Choose a reason for hiding this comment

szabosteve left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dimitris-athanasiou commented Nov 22, 2019

przemekwitek left a comment

Choose a reason for hiding this comment

dimitris-athanasiou commented Nov 21, 2019 •

edited

Loading

dimitris-athanasiou commented Nov 21, 2019 •

edited

Loading