Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[HUDI-3879] Suppress exceptions that are not fatal in HoodieMetadataTableValidator #5344

Merged
merged 4 commits into from
Sep 13, 2022
Merged

[HUDI-3879] Suppress exceptions that are not fatal in HoodieMetadataTableValidator #5344

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e742cc9
Suppress exceptions that are not fatal in HoodieMetadataTableValidator
Apr 18, 2022
8a31c74
fix check style
Apr 18, 2022
b409296
code review
Apr 19, 2022
e4facd5
Fix typos
yihua Sep 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 24 additions & 4 deletions hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@
import org.apache.hudi.exception.HoodieException;
import org.apache.hudi.exception.HoodieIOException;
import org.apache.hudi.exception.HoodieValidationException;
import org.apache.hudi.exception.TableNotFoundException;
import org.apache.hudi.io.storage.HoodieFileReader;
import org.apache.hudi.io.storage.HoodieFileReaderFactory;
import org.apache.hudi.metadata.HoodieTableMetadata;
Expand Down Expand Up @@ -399,7 +400,9 @@ public void doMetadataTableValidation() {
Set<String> baseFilesForCleaning = Collections.emptySet();

// check metadata table is available to read.
checkMetadataTableIsAvailable();
if (!checkMetadataTableIsAvailable()) {
return;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

even if Metadata table does not have any commits, do we need to add an else block here and check that there are no completed commits in data table?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, changed. add another else block in checkMetadataTableIsAvailable()
If there is no completed commits neither in data table nor MDT, HoodieMetadataTableValidator will skip current loop and log an info message.


if (cfg.skipDataFilesForCleaning) {
HoodieTimeline inflightCleaningTimeline = metaClient.getActiveTimeline().getCleanerTimeline().filterInflights();
Expand Down Expand Up @@ -428,6 +431,12 @@ public void doMetadataTableValidation() {

HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
List<String> allPartitions = validatePartitions(engineContext, basePath);

if (allPartitions.isEmpty()) {
LOG.warn("The result of getting all partitions is null or empty, skip current validation.");
return;
}

HoodieMetadataValidationContext metadataTableBasedContext =
new HoodieMetadataValidationContext(engineContext, cfg, metaClient, true);
HoodieMetadataValidationContext fsBasedContext =
Expand Down Expand Up @@ -465,18 +474,29 @@ public void doMetadataTableValidation() {
* Check metadata is initialized and available to ready.
* If not we will log.warn and skip current validation.
*/
private void checkMetadataTableIsAvailable() {
private boolean checkMetadataTableIsAvailable() {
try {
HoodieTableMetaClient mdtMetaClient = HoodieTableMetaClient.builder()
.setConf(jsc.hadoopConfiguration()).setBasePath(new Path(cfg.basePath, HoodieTableMetaClient.METADATA_TABLE_FOLDER_PATH).toString())
.setLoadActiveTimelineOnLoad(true)
.build();
int finishedInstants = mdtMetaClient.getActiveTimeline().filterCompletedInstants().countInstants();
if (finishedInstants == 0) {
throw new HoodieValidationException("There is no completed instant for metadata table.");
if (metaClient.getActiveTimeline().filterCompletedInstants().countInstants() == 0) {
LOG.info("There is no completed instant both in metadata table and corresponding data table.");
return false;
} else {
throw new HoodieValidationException("There is no completed instant for metadata table.");
}
}
return true;
} catch (TableNotFoundException tbe) {
// Suppress the TableNotFound exception if Metadata table is not available to read for now
LOG.warn("Metadata table is not found. Skip current validation.");
return false;
} catch (Exception ex) {
LOG.warn("Metadata table is not available to ready for now, ", ex);
LOG.warn("Metadata table is not available to read for now, ", ex);
return false;
}
}

Expand Down