Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Register first lookup immediately, regardless of the cache status. #15598

Merged
merged 1 commit into from Jan 9, 2024

Conversation

pranavbhole
Copy link
Contributor

@pranavbhole pranavbhole commented Dec 22, 2023

Description

In case of the first version of the lookup, we can skip waiting for initialization where it is good or bad configuration and just load it as it is. Esp for MSQ workers, it get spin off for each tasks, and issues loadNotice with no previous lookup version.
For first lookup, it get registered immediately regardless of the cache status. For subsequent version of the lookup, we make sure that new lookup come up before we kill the existing good lookup.

In subsequent version of the same lookup, we always make sure that we wait until new lookup config is loaded with cache before we kill the old working lookup.

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@github-actions github-actions bot added Area - Batch Ingestion Area - Querying Area - Web Console Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels Dec 22, 2023
@pranavbhole pranavbhole changed the title First loading lookups1 skip waiting for initialization for first version of lookup config Dec 22, 2023
Copy link
Contributor

@soumyava soumyava left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vogievetsky
Copy link
Contributor

hmm why did the bot add a web console label?

@abhishekrb19
Copy link
Contributor

hmm why did the bot add a web console label?

@vogievetsky, my guess is that a web-console file was touched in one of the earlier commits and removed in a later commit. The auto-labeler bot config has sync-labels turned off by default because we noticed it'd otherwise spuriously remove labels manually added by maintainers. So the sync feature is turned off by default and folks need to remove irrelevant labels manually when the scope of a PR changes.

(It'll be nice if auto-labeler bot syncs only bot-added labels with sync-labels = true, but oh well, it doesn't exist :))

@pranavbhole pranavbhole changed the title skip waiting for initialization for first version of lookup config Register first lookup immediately, regardless of the cache status. Jan 8, 2024
@clintropolis clintropolis merged commit 747d973 into apache:master Jan 9, 2024
83 checks passed
@clintropolis clintropolis deleted the firstLoadingLookups1 branch January 9, 2024 21:18
@LakshSingla LakshSingla added this to the 29.0.0 milestone Jan 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area - Batch Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 Area - Querying
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants