-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[HUDI-995] Migrate HoodieTestUtils APIs to HoodieTestTable #2143
Conversation
6fb5d84
to
3be9ec7
Compare
Remove APIs in `HoodieTestUtils` - listAllDataFilesAndLogFilesInPath - listAllLogFilesInPath - listAllDataFilesInPath - writeRecordsToLogFiles - createCleanFiles - createPendingCleanFiles Migrate the callers to use `HoodieTestTable` and `HoodieWriteableTestTable` with new APIs added - listAllBaseAndLogFiles - listAllLogFiles - listAllBaseFiles - withLogAppends - addClean - addInflightClean Also added related APIs in `FileCreateUtils` - createCleanFile - createRequestedCleanFile - createInflightCleanFile
3be9ec7
to
1d9dac8
Compare
@yanghua this should be the 2nd last PR of this migration. Thanks. |
private void appendRecordsToLogFile(List<HoodieRecord> groupedRecords) throws Exception { | ||
String partitionPath = groupedRecords.get(0).getPartitionPath(); | ||
HoodieRecordLocation location = groupedRecords.get(0).getCurrentLocation(); | ||
try (HoodieLogFormat.Writer logWriter = HoodieLogFormat.newWriterBuilder().onParentPath(new Path(basePath, partitionPath)) | ||
.withFileExtension(HoodieLogFile.DELTA_EXTENSION).withFileId(location.getFileId()) | ||
.overBaseCommit(location.getInstantTime()).withFs(fs).build()) { | ||
Map<HoodieLogBlock.HeaderMetadataType, String> header = new HashMap<>(); | ||
header.put(HoodieLogBlock.HeaderMetadataType.INSTANT_TIME, location.getInstantTime()); | ||
header.put(HoodieLogBlock.HeaderMetadataType.SCHEMA, schema.toString()); | ||
logWriter.appendBlock(new HoodieAvroDataBlock(groupedRecords.stream().map(r -> { | ||
try { | ||
GenericRecord val = (GenericRecord) r.getData().getInsertValue(schema).get(); | ||
HoodieAvroUtils.addHoodieKeyToRecord(val, r.getRecordKey(), r.getPartitionPath(), ""); | ||
return (IndexedRecord) val; | ||
} catch (IOException e) { | ||
return null; | ||
} | ||
}).collect(Collectors.toList()), header)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
refer to hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestUtils.java#writeRecordsToLogFiles
public List<FileStatus> listAllFilesInTempFolder() throws IOException { | ||
return FileSystemTestUtils.listRecursive(fs, new Path(Paths.get(basePath, HoodieTableMetaClient.TEMPFOLDER_NAME).toString())); | ||
public FileStatus[] listAllFilesInTempFolder() throws IOException { | ||
return FileSystemTestUtils.listRecursive(fs, new Path(Paths.get(basePath, HoodieTableMetaClient.TEMPFOLDER_NAME).toString())).toArray(new FileStatus[0]); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
change to return array to work with HoodieTableFileSystemView
APIs which take in array
@xushiyan Sorry for the late reply, the past week was during the National Day holiday, you know. Will review tomorrow. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@xushiyan left two comments.
...ient/hudi-spark-client/src/test/java/org/apache/hudi/testutils/HoodieWriteableTestTable.java
Outdated
Show resolved
Hide resolved
...ient/hudi-spark-client/src/test/java/org/apache/hudi/testutils/HoodieWriteableTestTable.java
Show resolved
Hide resolved
Codecov Report
@@ Coverage Diff @@
## master #2143 +/- ##
============================================
+ Coverage 47.38% 53.61% +6.23%
- Complexity 2567 2845 +278
============================================
Files 358 359 +1
Lines 16484 16535 +51
Branches 1767 1777 +10
============================================
+ Hits 7811 8866 +1055
+ Misses 8032 6912 -1120
- Partials 641 757 +116
Flags with carried forward coverage won't be shown. Click here to find out more. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. @xushiyan Thanks for your contribution.
* [HUDI-995] Migrate HoodieTestUtils APIs to HoodieTestTable Remove APIs in `HoodieTestUtils` - listAllDataFilesAndLogFilesInPath - listAllLogFilesInPath - listAllDataFilesInPath - writeRecordsToLogFiles - createCleanFiles - createPendingCleanFiles Migrate the callers to use `HoodieTestTable` and `HoodieWriteableTestTable` with new APIs added - listAllBaseAndLogFiles - listAllLogFiles - listAllBaseFiles - withLogAppends - addClean - addInflightClean Also added related APIs in `FileCreateUtils` - createCleanFile - createRequestedCleanFile - createInflightCleanFile
Remove APIs in
HoodieTestUtils
Migrate the callers to use
HoodieTestTable
andHoodieWriteableTestTable
with new APIs addedAlso added related APIs in
FileCreateUtils
Committer checklist
Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.