Skip to content

HIVE-28655: Implement HMS Related Drop Stats Changes Part2 (param COLUMN_STAT_ACCURATE related changes) #5790

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

HIVE-28655: Implement HMS Related Drop Stats Changes Part2 (param COLUMN_STAT_ACCURATE related changes) #5790

wants to merge 1 commit into from
+203 −34

Conversation

DanielZhu58
Copy link
Contributor

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

dengzhhu653
dengzhhu653 reviewed May 6, 2025
View reviewed changes
...ne-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java Outdated
@@ -7431,6 +7431,13 @@ public boolean delete_column_statistics_req(DeleteColumnStatisticsRequest req) t
if (!isPartitioned || req.isTableLevel()) {
ret = rawStore.deleteTableColumnStatistics(parsedDbName[CAT_NAME], parsedDbName[DB_NAME], tableName, colNames, engine);
if (ret) {
Map<String, String> parameters = table.getParameters();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we just remove the deleted columns from the COLUMN_STATS_ACCURATE? using the StatsSetupConst.removeColumnStatsState

dengzhhu653
dengzhhu653 reviewed May 6, 2025
View reviewed changes
...ne-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java Outdated
}
partParams.put("COLUMN_STATS_ACCURATE", "false");
partition.setParameters(partParams);
rawStore.alterPartition(parsedDbName[CAT_NAME], parsedDbName[DB_NAME], tableName, partVals, partition, null);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alter the partitions in bulk?

dengzhhu653
dengzhhu653 reviewed May 21, 2025
View reviewed changes
...ne-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java Outdated
} else {
// remove the deleted column names in parameter COLUMN_STATS_ACCURATE
StatsSetupConst.removeColumnStatsState(table.getParameters(), colNames);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to persist the changed parameters by rawStore.alterTable()

dengzhhu653
dengzhhu653 reviewed May 21, 2025
View reviewed changes
...ne-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java Outdated
} else {
// remove the deleted column names in parameter COLUMN_STATS_ACCURATE
StatsSetupConst.removeColumnStatsState(partParams, colNames);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need to persist the partition parameters after resetting the column stats

dengzhhu653
dengzhhu653 reviewed May 22, 2025
View reviewed changes
...e-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java Outdated
@@ -10363,6 +10365,8 @@ private boolean deleteTableColumnStatisticsViaJdo(String catName, String dbName,
}
mStatsObjColl = (List<MTableColumnStatistics>) query.executeWithArray(params.toArray());
pm.retrieveAll(mStatsObjColl);
Long tableID = directSql.getTableId(dbName, tableName);
Copy link
Member

@dengzhhu653 dengzhhu653 May 22, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there any reason to mix the direct sql with the jdo together? this deleteTableColumnStatisticsViaJdo is for jdo query I guess. And we need to take care of the same in getSqlResult path.

@sonarqubecloud SonarQubeCloud
Copy link

Quality Gate Passed Quality Gate passed

Issues
6 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dengzhhu653 dengzhhu653

@wecharyu wecharyu

Assignees
No one assigned
Labels
tests failed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@DanielZhu58 @dengzhhu653 @wecharyu @asf-ci-hive