Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[HUDI-5336] Fixing log file pattern match to ignore extraneous files #7612

Merged
merged 5 commits into from
Jan 20, 2023
Merged

[HUDI-5336] Fixing log file pattern match to ignore extraneous files #7612

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
82e6b63
Fixing log file pattern match to ignore extraneous files
nsivabalan Jan 5, 2023
66370e1
minor fixes
nsivabalan Jan 6, 2023
1dc0a07
addressing comments
nsivabalan Jan 18, 2023
18c64ac
addressing more comments
nsivabalan Jan 19, 2023
3ea3fec
Fix nit
yihua Jan 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ public class FSUtils {
// Log files are of this pattern - .b5068208-e1a4-11e6-bf01-fe55135034f3_20170101134598.log.1_1-0-1
// Archive log files are of this pattern - .commits_.archive.1_1-0-1
public static final Pattern LOG_FILE_PATTERN =
Pattern.compile("\\.(.+)_(.*)\\.(.+)\\.(\\d+)(_((\\d+)-(\\d+)-(\\d+))(.cdc)?)?");
Pattern.compile("^\\.(.+)_(.*)\\.(log|archive)\\.(\\d+)(_((\\d+)-(\\d+)-(\\d+))(.cdc)?)?");
private static final int MAX_ATTEMPTS_RECOVER_LEASE = 10;
private static final long MIN_CLEAN_TO_KEEP = 10;
private static final long MIN_ROLLBACK_TO_KEEP = 10;
Expand Down Expand Up @@ -358,7 +358,7 @@ public static String createNewFileId(String idPfx, int id) {
* Get the file extension from the log file.
*/
public static String getFileExtensionFromLog(Path logPath) {
Matcher matcher = LOG_FILE_PATTERN.matcher(logPath.getName());
Matcher matcher = LOG_FILE_PATTERN.matcher(logPath.getName());
nsivabalan marked this conversation as resolved.
Show resolved Hide resolved
if (!matcher.find()) {
throw new InvalidHoodiePathException(logPath, "LogFile");
}
Expand All @@ -370,7 +370,7 @@ public static String getFileExtensionFromLog(Path logPath) {
* the file name.
*/
public static String getFileIdFromLogPath(Path path) {
Matcher matcher = LOG_FILE_PATTERN.matcher(path.getName());
Matcher matcher = LOG_FILE_PATTERN.matcher(path.getName());
if (!matcher.find()) {
throw new InvalidHoodiePathException(path, "LogFile");
}
Expand Down
2 changes: 1 addition & 1 deletion hudi-common/src/test/java/org/apache/hudi/common/functional/TestHoodieLogFormat.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -343,7 +343,7 @@ public void testAppendNotSupported(@TempDir java.nio.file.Path tempDir) throws I

for (int i = 0; i < 2; i++) {
Writer writer = HoodieLogFormat.newWriterBuilder().onParentPath(testPath)
.withFileExtension(HoodieArchivedLogFile.ARCHIVE_EXTENSION).withFileId("commits.archive").overBaseCommit("")
.withFileExtension(HoodieArchivedLogFile.ARCHIVE_EXTENSION).withFileId("commits").overBaseCommit("")
.withFs(localFs).build();
writer.appendBlock(dataBlock);
writer.close();
Expand Down
4 changes: 2 additions & 2 deletions ...mon/src/test/java/org/apache/hudi/common/functional/TestHoodieLogFormatAppendFailure.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ public void testFailedToGetAppendStreamFromHDFSNameNode()
HoodieAvroDataBlock dataBlock = new HoodieAvroDataBlock(records, header, HoodieRecord.RECORD_KEY_METADATA_FIELD);

Writer writer = HoodieLogFormat.newWriterBuilder().onParentPath(testPath)
.withFileExtension(HoodieArchivedLogFile.ARCHIVE_EXTENSION).withFileId("commits.archive")
.withFileExtension(HoodieArchivedLogFile.ARCHIVE_EXTENSION).withFileId("commits")
.overBaseCommit("").withFs(fs).build();

writer.appendBlock(dataBlock);
Expand Down Expand Up @@ -134,7 +134,7 @@ public void testFailedToGetAppendStreamFromHDFSNameNode()
// Opening a new Writer right now will throw IOException. The code should handle this, rollover the logfile and
// return a new writer with a bumped up logVersion
writer = HoodieLogFormat.newWriterBuilder().onParentPath(testPath)
.withFileExtension(HoodieArchivedLogFile.ARCHIVE_EXTENSION).withFileId("commits.archive")
.withFileExtension(HoodieArchivedLogFile.ARCHIVE_EXTENSION).withFileId("commits")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this mean the original tests do not properly construct the archived log file name?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes.

.overBaseCommit("").withFs(fs).build();
header = new HashMap<>();
header.put(HoodieLogBlock.HeaderMetadataType.COMMAND_BLOCK_TYPE,
Expand Down
2 changes: 1 addition & 1 deletion ...common/src/test/java/org/apache/hudi/common/table/view/TestHoodieTableFileSystemView.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -306,7 +306,7 @@ protected void testInvalidLogFiles() throws Exception {
String fileName2 =
FSUtils.makeLogFileName(fileId, HoodieLogFile.DELTA_EXTENSION, instantTime1, 1, TEST_WRITE_TOKEN);
// create a dummy log file mimicing cloud stores marker files
String fileName3 = "_DUMMY_" + fileName1.substring(1, fileName1.length());
String fileName3 = "_GCS_SYNCABLE_TEMPFILE_" + fileName1;
nsivabalan marked this conversation as resolved.
Show resolved Hide resolved
// this file should not be deduced as a log file.

Paths.get(basePath, partitionPath, fileName1).toFile().createNewFile();
Expand Down