Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Quidem record #16624

Merged
merged 134 commits into from
Aug 5, 2024
Merged

Quidem record #16624

merged 134 commits into from
Aug 5, 2024

Conversation

kgyrtkirk
Copy link
Member

@kgyrtkirk kgyrtkirk commented Jun 18, 2024

  • enables to launch a fake broker based on test resources (druidtest uri)
  • could record queries into new testfiles during usage
  • instead of re-purpose Calcite's Hook migrates to use DruidHook which we can add further keys
  • added a quidem-it module which could be the place for tests which could iteract with modules/etc

* Altered `QueryTestBuilder` to be able to switch to a backing quidem test
* added a small crc to ensure that the shadow testcase does not deviate from the original one
* Packaged all decoupled related things into a a single `DecoupledExtension` to reduce copy-paste
* `DecoupledTestConfig#quidemReason` must describe why its being used
* `DecoupledTestConfig#separateDefaultModeTest` can be used to make multiple case files based on `NullHandling` state
* fixed a cosmetic bug during decoupled join translation
* enhanced `!druidPlan` to report the final logical plan in non-decoupled mode as well
* add check to ensure that only supported params are present in a druidtest uri
* enabled shadow testcases for previously disabled testcases
style fixes

clenaup
This reverts commit 3fbb3cb.
/**
* Closely related to {@link CliBroker#getModules}.
*/
static List<? extends Module> brokerModules()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where do we bind the ClientQuerySegmentWalker here? I haven't gone through the complete PR, however I wanted to know that so that I have a clearer understanding of what's going on.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the role of this module is only to expose the ComponentSupplier as a broker; the QuerySegmentWalker is a more core part which is needed for regular tests as well - its exposed at SqlTestFramework#TestSetupModule#querySegmentWalker

actually this is the PR I was finally able to pull that object into the guice world - doing these from guice will make it possible to get rid of the complicated way the system have launched - and could also make it easier to customize the environment further

@@ -563,7 +563,7 @@ protected PlannerResult planWithDruidConvention() throws ValidationException
.plus(DruidLogicalConvention.instance()),
newRoot
);
Hook.JAVA_PLAN.run(newRoot);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be a bit nicer if this didn't have to rely on statics and could instead rely on an instance object. For this QueryHandler itself, it looks like these end up constructed via the DruidPlanner which is built by the PlannerFactory which is injected via Guice. Could we get the DruidHook (or something equivalent to it) injected into that and pushed down to this object instead of relying on statics?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was also thinking about doing that - but opted to not do that at first
fortunately 90% of the necessary machinery was already in place...so it wasn't really hard to plug it in via guice...but I had to touch a few more files (+20)

Copy link
Member

@asdf2014 asdf2014 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

This failure doesn't seem related to this PR. We need to increase the value of timeout option in this unit test case to stabilize it.

@kgyrtkirk kgyrtkirk merged commit 26e3c44 into apache:master Aug 5, 2024
89 checks passed
@kgyrtkirk
Copy link
Member Author

merged into master
Thank you all for reviewing the changes!

@kfaraz kfaraz added this to the 31.0.0 milestone Oct 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area - Batch Ingestion Area - Dependencies Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 Area - Querying
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants