Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add new context parameter for using concurrent locks #15684

Merged
merged 2 commits into from Jan 16, 2024
Merged

Add new context parameter for using concurrent locks #15684

merged 2 commits into from Jan 16, 2024

Conversation

AmatyaAvadhanula
Copy link
Contributor

This PR introduces a new task context flag useConcurrentLocks.
This can be set for an individual task or at a cluster level using druid.indexer.task.default.context.

When set to true any appending task would use an APPEND lock and any other ingestion task would use a REPLACE lock when using time chunk locking.
If not, we fall back on the context flag taskLockType and then useSharedLock.

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@github-actions github-actions bot added Area - Batch Ingestion Area - Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels Jan 15, 2024
kfaraz
kfaraz reviewed Jan 15, 2024
View reviewed changes
Copy link
Contributor

@kfaraz kfaraz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good, left a minor comment.

Also, there seem to be some coverage issues. Maybe add a couple more tests for the new context parameter.

...s-core/multi-stage-query/src/main/java/org/apache/druid/msq/util/MultiStageQueryContext.java Outdated Show resolved Hide resolved
kfaraz
kfaraz approved these changes Jan 16, 2024
View reviewed changes
Copy link
Contributor

@kfaraz kfaraz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kfaraz
Copy link
Contributor

kfaraz commented Jan 16, 2024

Merging as failure is unrelated.

@kfaraz kfaraz merged commit 6b951b9 into apache:master Jan 16, 2024
82 of 83 checks passed
@kfaraz kfaraz deleted the use_concurrent_locks branch January 16, 2024 07:13
@kfaraz kfaraz added this to the Druid 29.0.0 milestone Jan 17, 2024
AmatyaAvadhanula added a commit to AmatyaAvadhanula/druid that referenced this pull request Jan 29, 2024
@AmatyaAvadhanula
Add new context parameter for using concurrent locks (apache#15684)
07326cf 
Changes:
- Add new task context flag useConcurrentLocks.
- This can be set for an individual task or at a cluster level using `druid.indexer.task.default.context`.
- When set to true, any appending task would use an APPEND lock and any other
ingestion task would use a REPLACE lock when using time chunk locking.
- If false (default), we fall back on the context flag taskLockType and then useSharedLock.
@AmatyaAvadhanula AmatyaAvadhanula mentioned this pull request Jan 29, 2024
Merged
LakshSingla pushed a commit that referenced this pull request Jan 30, 2024
@AmatyaAvadhanula
Add new context parameter for using concurrent locks (#15684) (#15775)
d527bc1 
Changes:
- Add new task context flag useConcurrentLocks.
- This can be set for an individual task or at a cluster level using `druid.indexer.task.default.context`.
- When set to true, any appending task would use an APPEND lock and any other
ingestion task would use a REPLACE lock when using time chunk locking.
- If false (default), we fall back on the context flag taskLockType and then useSharedLock.
@LakshSingla LakshSingla mentioned this pull request Feb 13, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kfaraz kfaraz kfaraz approved these changes

Assignees
No one assigned
Labels
Area - Batch Ingestion Area - Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262
Projects
None yet
Milestone
29.0.0
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@AmatyaAvadhanula @kfaraz