Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(pat-5034): include order-by expressions when computing the group-by clause #208

Merged
merged 3 commits into from
Dec 19, 2023
Merged

fix(pat-5034): include order-by expressions when computing the group-by clause #208

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c06f6a3
fix(pat-5034): include order-by expressions when computing the group-…
Dec 18, 2023
254709e
test(pat-5034): assert non-aggs in order-by that are selected are not…
Dec 19, 2023
796aeb8
fix(pat-5034): use field name in selection set for testing order-by i…
Dec 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 68 additions & 61 deletions static/nodejs/src/backends/dpm_agent/dpm_agent_client.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,76 +221,83 @@ function makeDpmOrderByExpression(
}

/**
* DpmAgentClient uses a gRPC client to compile and execute queries by using the
* `dpm-agent` which routes the queries to the specific source specified in the
* query's package descriptor.
* Makes a query message from the table expression to send to dpm-agent.
* @param query Table expression
* @returns Query RPC message to send to dpm-agent.
*/
export class DpmAgentClient implements Backend {
private metadata: Metadata;
export function makeDpmAgentQuery(query: Table): DpmAgentQuery {
ajmasci marked this conversation as resolved.
Show resolved Hide resolved
const dpmAgentQuery = new DpmAgentQuery();
const id = new DpmAgentQuery.Id().setPackageid(query.packageId);
dpmAgentQuery.setId(id);

const clientVersion = new ClientVersion()
.setClient(ClientVersion.Client.NODE_JS)
.setDatasetversion(query.datasetVersion)
.setCodeversion(codeVersion);
dpmAgentQuery.setClientversion(clientVersion);
dpmAgentQuery.setSelectfrom(query.name);

const {
filterExpr: filter,
selection,
ordering: orderBy,
limitTo: limit,
} = query;
const dpmSelectExprs = selection?.map(makeDpmSelectExpression);
if (dpmSelectExprs) {
dpmAgentQuery.setSelectList(dpmSelectExprs);
}

/**
* Makes a query message from the table expression to send to dpm-agent.
* @param query Table expression
* @returns Query RPC message to send to dpm-agent.
*/
private async makeDpmAgentQuery(query: Table): Promise<DpmAgentQuery> {
const dpmAgentQuery = new DpmAgentQuery();
const id = new DpmAgentQuery.Id().setPackageid(query.packageId);
dpmAgentQuery.setId(id);

const clientVersion = new ClientVersion()
.setClient(ClientVersion.Client.NODE_JS)
.setDatasetversion(query.datasetVersion)
.setCodeversion(codeVersion);
dpmAgentQuery.setClientversion(clientVersion);
dpmAgentQuery.setSelectfrom(query.name);

const {
filterExpr: filter,
selection,
ordering: orderBy,
limitTo: limit,
} = query;
const selections = selection?.map(makeDpmSelectExpression);
if (selections) {
dpmAgentQuery.setSelectList(selections);
}
// Process filter.
if (filter) {
dpmAgentQuery.setFilter(makeDpmBooleanExpression(filter));
}

// Process filter.
if (filter) {
dpmAgentQuery.setFilter(makeDpmBooleanExpression(filter));
// Process any groupings defined in selection or orderBy.
const selectionMap = new Set<FieldExpr>(selection ?? []);
const expandedSelection = [
...(selection ?? []),
...(orderBy ?? [])
.map((x: Ordering) => x[0])
.filter((x) => !selectionMap.has(x)),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can selectionMap.has(x) ever return true? Given how value equality works in Sets, which is pretty much like ===, it's not clear to me that the FieldExprs in selection could ever === the FieldExprs in the array that the filter is called on here.

Could you add a test that has equivalent, non-aggregateFieldExprs in the select and orderby, and see if they get deduped properly in the GroupbyList?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In fact, the test case with avgPrice as the first order-by expression (see my comment below), results in a true value returned by selectionMap.has, and the group-by is correctly de-duped. Is there any reason that you think a non-aggregate field in the order-by that matches a selection expression will behave differently?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ajith's idea: Two derived fields would be unlikely to match.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good callout here @spencerwilson ! The derived field test did reveal the bug. Fixed it by using the field name in the selectionSet.

];
if (
expandedSelection.findIndex(
(fieldExpr) => fieldExpr instanceof AggregateFieldExpr
) !== -1
) {
const grouping = expandedSelection.filter(
(fieldExpr) => !(fieldExpr instanceof AggregateFieldExpr)
);
if (grouping) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI grouping will always be truthy (though it may have grouping.length === 0)

dpmAgentQuery.setGroupbyList(grouping.map(makeDpmGroupByExpression));
}
}

// Process any groupings defined in selection.
if (
selection?.findIndex(
(fieldExpr) => fieldExpr instanceof AggregateFieldExpr
) !== -1
) {
const grouping = selection?.filter(
(fieldExpr) => !(fieldExpr instanceof AggregateFieldExpr)
);
if (grouping) {
dpmAgentQuery.setGroupbyList(grouping.map(makeDpmGroupByExpression));
}
}
// Process orderBy.
if (orderBy !== undefined && orderBy.length > 0) {
const dpmOrderings = orderBy.map(makeDpmOrderByExpression);
dpmAgentQuery.setOrderbyList(dpmOrderings);
}

// Process orderBy.
if (orderBy !== undefined && orderBy.length > 0) {
const dpmOrderings = orderBy.map(makeDpmOrderByExpression);
dpmAgentQuery.setOrderbyList(dpmOrderings);
}
if (limit > 0) {
dpmAgentQuery.setLimit(limit);
}

if (limit > 0) {
dpmAgentQuery.setLimit(limit);
}
return dpmAgentQuery;
}

return Promise.resolve(dpmAgentQuery);
}
/**
* DpmAgentClient uses a gRPC client to compile and execute queries by using the
* `dpm-agent` which routes the queries to the specific source specified in the
* query's package descriptor.
*/
export class DpmAgentClient implements Backend {
private metadata: Metadata;

constructor(
private client: DpmAgentGrpcClient,
private dpmAuthToken: string,
private dpmAuthToken: string
) {
this.metadata = new Metadata();
this.metadata.set('dpm-auth-token', this.dpmAuthToken);
Expand All @@ -303,7 +310,7 @@ export class DpmAgentClient implements Backend {
* dpm-agent, or rejects on error.
*/
async compile(query: Table): Promise<string> {
const dpmAgentQuery = await this.makeDpmAgentQuery(query);
const dpmAgentQuery = makeDpmAgentQuery(query);
dpmAgentQuery.setDryrun(true);
return new Promise((resolve, reject) => {
this.client.executeQuery(
Expand All @@ -328,7 +335,7 @@ export class DpmAgentClient implements Backend {
* dpm-agent, or rejects on error.
*/
async execute<Row extends object>(query: Table): Promise<Row[]> {
const dpmAgentQuery = await this.makeDpmAgentQuery(query);
const dpmAgentQuery = makeDpmAgentQuery(query);
return new Promise((resolve, reject) => {
this.client.executeQuery(
dpmAgentQuery,
Expand Down
Loading
Loading