MSQ: Report the warning directly as an error if none of it is allowed by the user #13198

LakshSingla · 2022-10-10T04:01:09Z

Description

In MSQ, there can be an upper limit to the number of worker warnings. For example, for parseExceptions encountered while parsing the external data, the user can specify an upper limit to the number of parse exceptions that can be allowed before it throws an error of type TooManyWarnings.

This PR makes it so that if the user disallows warnings of a certain type i.e. the limit is 0 (or is executing in strict mode), instead of throwing an error of type TooManyWarnings, we can directly surface the warning as the error, saving the user from the hassle of going throw the warning reports.

Key changed/added classes in this PR

ControllerImpl

This PR has:

been self-reviewed.
- using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
added documentation for new or modified features or behaviors.
added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
added or updated version, license, or notice information in licenses.yaml
added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
added integration tests.
been tested in a test Druid cluster.

cryptoe

Thanks for the PR @LakshSingla. Left some comments.

cryptoe · 2022-10-10T14:06:06Z

extensions-core/multi-stage-query/src/test/java/org/apache/druid/msq/test/MSQTestBase.java

+                msqErrorReport.getFault().getCodeWithMessage()
+            );
+          }
+          if (expectedMSQFaultClass != null) {


Shouldn't we always assert the MSQ fault?
Is there any specific reason to assert on the MSQ class?

With the unparseable exceptions, the exception message contained nonstandard characters (from the erroneous line) which I didn't want to use in the code file.

We change the get message call at line 863 to be a pattern matcher. That way you need not code out the whole string.

Or change the source test file.

This got me thinking, if we have a malformed line, we can have a parseException with x mb's of input.
Multiple such lines would blow up our report.
Should we chomp the input to lets day 300 bytes or something in ParseException#75
cc @jon-wei

Stack traces of innocuous warnings and errors can also cause a similar issue. Should this be taken up as a separate PR? Also, 300 bytes seem less to me, something like 4KB might work better, wdyt? (I think we should be fine with an even larger limit as the number of warnings sent to the controller are limited).

Yup We can do that outside this PR.

extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/ControllerImpl.java

cryptoe · 2022-10-10T14:21:49Z

extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/ControllerImpl.java

+            workerError(errorReport1);
+            break;
+          }
+        }


We should also throw an illegal state exception if we donot find the error code in the workerWarnings.

Makes sense, or else the controller would fail to stop.

I was thinking, let the worker impl directly hit worker error endpoint in case the limit is 0 wdyt ?

Trying out the changes, that seems like a much better alternative.

cryptoe · 2022-10-13T07:10:49Z

extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerImpl.java

+        ImmutableMap.of(CannotParseExternalDataFault.CODE, maxVerboseParseExceptions),
+        disallowedWarningCode,
+        controllerClient,
+        id(),


nit: id, host, controllerClient should be the first arguments

extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerImpl.java

...uery/src/main/java/org/apache/druid/msq/indexing/error/MSQWarningReportLimiterPublisher.java

cryptoe · 2022-10-13T11:51:45Z

...uery/src/main/java/org/apache/druid/msq/indexing/error/MSQWarningReportLimiterPublisher.java

+        try {
+          controllerClient.postWorkerError(workerId, MSQErrorReport.fromException(workerId, host, stageNumber, e));
+        }
+        catch (IOException e2) {


Can we throw a new RE with the message failed to post the worker error xyz to the controller. That way the worker will get terminated with a relevant error.

Nit: rename from e2 to postException?

cryptoe · 2022-10-13T11:54:04Z

...uery/src/main/java/org/apache/druid/msq/indexing/error/MSQWarningReportLimiterPublisher.java

-  final ConcurrentHashMap<String, Long> errorCodeToCurrentCount = new ConcurrentHashMap<>();
+  private final MSQWarningReportPublisher delegate;
+  private final long totalLimit;
+  private final Map<String, Long> errorCodeToVerboseCountLimit;


nit: errorCodeToLimit seems like a better variable to me.
we really do not need verboseCountLimt as that's a concept outside this class.

I think that the errorCodeToLimit was slightly misleading. This is because there can be more than those error codes can be present in the controller. However, I do get your point. Maybe I will add clearer javadocs so that this ambiguity isn't present and revert the change.

cryptoe · 2022-10-13T11:59:48Z

extensions-core/multi-stage-query/src/test/java/org/apache/druid/msq/test/MSQTestBase.java

+                msqErrorReport.getFault().getCodeWithMessage()
+            );
+          }
+          if (expectedMSQFaultClass != null) {


This got me thinking, if we have a malformed line, we can have a parseException with x mb's of input.
Multiple such lines would blow up our report.
Should we chomp the input to lets day 300 bytes or something in ParseException#75
cc @jon-wei

cryptoe

Minor comments.
LGTM (+1 non binding)

cryptoe · 2022-10-17T16:10:51Z

...uery/src/main/java/org/apache/druid/msq/indexing/error/MSQWarningReportLimiterPublisher.java

-
+  /**
+   * Creates a publisher which publishes the warnings to the controller if they have not yet exceeded the allowed limit.
+   * Moreover, if a warning is disallowed, i.e. it's limit is set to 0, then the publisher directly reports the warning


Nit: It would be much better if we could explain each variable. We really do not need to mention the tooManyWarningsFault piece as it's just confusing.

cryptoe · 2022-10-17T16:12:13Z

...uery/src/main/java/org/apache/druid/msq/indexing/error/MSQWarningReportLimiterPublisher.java

+        try {
+          controllerClient.postWorkerError(workerId, MSQErrorReport.fromException(workerId, host, stageNumber, e));
+        }
+        catch (IOException e2) {


Nit: rename from e2 to postException?

LakshSingla added 2 commits October 10, 2022 08:11

Initial commit

96a0cca

Update tests

9fb2870

kfaraz added the Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 label Oct 10, 2022

cryptoe reviewed Oct 10, 2022

View reviewed changes

LakshSingla added 6 commits October 11, 2022 11:44

review changes

90d0784

new approach

f3b15fd

remove stray log line

9620ba1

checkstyle fix

6cbf762

fix cast exception

d707e85

fix cast exception 2

5664453

cryptoe reviewed Oct 13, 2022

View reviewed changes

review comments

98007b6

cryptoe approved these changes Oct 17, 2022

View reviewed changes

docs update

6834285

abhishekagarwal87 approved these changes Oct 20, 2022

View reviewed changes

abhishekagarwal87 merged commit fc262df into apache:master Oct 20, 2022

kfaraz added this to the 25.0 milestone Nov 22, 2022

This was referenced Dec 18, 2022

[Draft] 25.0.0 Release Notes #13592

Closed

Add SegmentAllocationQueue to batch allocation actions #13369

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MSQ: Report the warning directly as an error if none of it is allowed by the user #13198

MSQ: Report the warning directly as an error if none of it is allowed by the user #13198

LakshSingla commented Oct 10, 2022 •

edited

Loading

cryptoe left a comment

cryptoe Oct 10, 2022

LakshSingla Oct 11, 2022 •

edited

Loading

cryptoe Oct 12, 2022

cryptoe Oct 12, 2022

cryptoe Oct 13, 2022

LakshSingla Oct 13, 2022

cryptoe Oct 14, 2022 •

edited

Loading

cryptoe Oct 10, 2022

LakshSingla Oct 11, 2022

cryptoe Oct 12, 2022

LakshSingla Oct 12, 2022

cryptoe Oct 13, 2022

cryptoe Oct 13, 2022

cryptoe Oct 17, 2022

cryptoe Oct 13, 2022

LakshSingla Oct 13, 2022

cryptoe Oct 13, 2022

cryptoe left a comment

cryptoe Oct 17, 2022

cryptoe Oct 17, 2022

MSQ: Report the warning directly as an error if none of it is allowed by the user #13198

MSQ: Report the warning directly as an error if none of it is allowed by the user #13198

Conversation

LakshSingla commented Oct 10, 2022 • edited Loading

Description

Key changed/added classes in this PR

cryptoe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LakshSingla Oct 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cryptoe Oct 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cryptoe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LakshSingla commented Oct 10, 2022 •

edited

Loading

LakshSingla Oct 11, 2022 •

edited

Loading

cryptoe Oct 14, 2022 •

edited

Loading