ACCUMULO-4813 New bulk import process and API #436
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Provide initial API * Provide some initial client code
* New Bulk Import Fate REPO * GSON serialize & implement Bulk class * Deprecated old BulkImporter * Created BulkLoadIT for new technique * TODO add more tests to BulkLoadIT
* In load RPC aow sending multiple file per tablet instead of one * Now saving mapping file to HDFS on client side * Simplified JSON serialization * Refactored new and old bulk import code into two packages * Now only iterate over mapping in master, no longer load into memory
* Fix bug for single tablet ingest * Fix BulkImportMove * Created 2 new tests in BulkLoadIT * Catch ExecutionException in Move
keith-turner
changed the title
keith-turner
commented
Apr 20, 2018
| * Load files from this directory | ||
| * @return ImportSourceOptions | ||
| */ | ||
| ImportSourceOptions from(String directory); |
There was a problem hiding this comment.
I think this should return ImportExecutorOptions
|
|
||
| /** | ||
| * Read Json array of Bulk.Mapping objects and return SortedMap of the bulk load mapping | ||
| */ |
There was a problem hiding this comment.
Need to update this javadoc
| OTHER(7), | ||
| NAMESPACE_EXISTS(8), | ||
| NAMESPACE_NOTFOUND(9), | ||
| INVALID_NAME(10); |
There was a problem hiding this comment.
Need to put the new type at the END of the list so the existing numbers don't change.
| TABLE_CANCEL_COMPACT(12), | ||
| NAMESPACE_CREATE(13), | ||
| NAMESPACE_DELETE(14), | ||
| NAMESPACE_RENAME(15); |
There was a problem hiding this comment.
Need to put the new type at the END of the list so the existing numbers don't change.
| * Normal operations, like bulk imports, will grab the read lock and prevent merges (writes) while | ||
| * they run. Merge operations will lock out some operations while they run. | ||
| * Normal operations, like bulkDir imports, will grab the read lock and prevent merges (writes) | ||
| * while they run. Merge operations will lock out some operations while they run. |
There was a problem hiding this comment.
Looks like find/replace caught some unintentional javadoc. We didn't modify this file
ctubbsii
reviewed
Apr 20, 2018
| // TODO need to handle case of file existing | ||
| BulkSerialize.writeLoadMapping(mappings, srcPath.toString(), p -> fs.create(p)); | ||
|
|
||
| List<ByteBuffer> args = Arrays.asList(ByteBuffer.wrap(tableName.getBytes(UTF_8)), |
There was a problem hiding this comment.
This should probably send tableId instead of name.
| } | ||
|
|
||
| int nThreads; | ||
| if (threads.toUpperCase().endsWith("C")) { |
There was a problem hiding this comment.
Should probably move this configuration type parsing into the ConfigurationTypeHelper.
* Update ordering of new thrift objects * Move Property parsing to ConfigTypeHelper * Fix Javadoc
* Updated documentation * Fixed bug in API * Unit tested merge detection code (and found bugs in the proccess) * Updated existing bulk IT to use new and old APIs * Addressed many TODOs
This was referenced May 7, 2018
|
Since this PR was merged, should the ACCUMULO-4813 JIRA issue be closed now? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a big change to how bulk import works. This new API does the following for bulk import.
The new API and functionality is complete, however the following still needs to be done.
After this PR is finished with review, will open issues for these and anything else that comes up.