Skip to content

Use multiple workers in MSQ WF drill test suite#16949

Merged
cryptoe merged 1 commit intoapache:masterfrom
Akshat-Jain:msq-wf-drill-test-use-multiple-workers
Aug 26, 2024
Merged

Use multiple workers in MSQ WF drill test suite#16949
cryptoe merged 1 commit intoapache:masterfrom
Akshat-Jain:msq-wf-drill-test-use-multiple-workers

Conversation

@Akshat-Jain
Copy link
Contributor

@Akshat-Jain Akshat-Jain commented Aug 23, 2024

Description

Currently, MSQ WF drill test suite uses maxNumTasks: 2 (default).

This means we can't detect any issues/regressions via this test, when we use multiple workers.

This PR updates the MSQ WF drill test suite to use maxNumTasks: 5.

However, there are some test cases which can give different correct results across different runs due to MixShuffleSpec being present in the previous stage. Such test cases have been overridden to use a single worker. This has been done for 8 tests.

Some tests have been modified to give deterministic result, while ensuring that the test is still validating what it was originally added for.

I've ran MSQDrillWindowQueryTest 20+ times locally to ensure no flakiness for any other test that's not overridden to use a single worker.


This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@github-actions github-actions bot added Area - Batch Ingestion Area - Querying Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels Aug 23, 2024
@cryptoe cryptoe merged commit 72f8e79 into apache:master Aug 26, 2024
@cryptoe
Copy link
Contributor

cryptoe commented Aug 26, 2024

Failures seems unrelated to this PR.

hevansDev pushed a commit to hevansDev/druid that referenced this pull request Aug 29, 2024
edgar2020 pushed a commit to edgar2020/druid that referenced this pull request Sep 5, 2024
@kfaraz kfaraz added this to the 31.0.0 milestone Oct 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area - Batch Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 Area - Querying

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants