-
Notifications
You must be signed in to change notification settings - Fork 40
Add data chunk and task result enums and dtos #2442
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
bf31a01
Data chunk and task result enums and dtos
inv-jishnu 57cd330
Spotless applied
inv-jishnu 7a39564
Changes
inv-jishnu be4583c
Added transaction batch dtos
inv-jishnu ca71bff
Merge branch 'master' into feat/data-loader/import-utils-dtos
ypeckstadt d52743e
Merge branch 'master' into feat/data-loader/import-utils-dtos
inv-jishnu e584ace
Corrected comment
inv-jishnu ffd3edc
Merge branch 'master' into feat/data-loader/import-utils-dtos
inv-jishnu c1b036e
Merge branch 'master' into feat/data-loader/import-utils-dtos
ypeckstadt File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
14 changes: 14 additions & 0 deletions
14
...ore/src/main/java/com/scalar/db/dataloader/core/dataimport/datachunk/ImportDataChunk.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| package com.scalar.db.dataloader.core.dataimport.datachunk; | ||
|
|
||
| import java.util.List; | ||
| import lombok.Builder; | ||
| import lombok.Data; | ||
|
|
||
| /** * Import data chunk data */ | ||
| @Data | ||
| @Builder | ||
| public class ImportDataChunk { | ||
|
|
||
| int dataChunkId; | ||
| List<ImportRow> sourceData; | ||
| } |
41 changes: 41 additions & 0 deletions
41
...c/main/java/com/scalar/db/dataloader/core/dataimport/datachunk/ImportDataChunkStatus.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| package com.scalar.db.dataloader.core.dataimport.datachunk; | ||
|
|
||
| import com.fasterxml.jackson.annotation.JsonProperty; | ||
| import com.fasterxml.jackson.databind.annotation.JsonDeserialize; | ||
| import java.time.Instant; | ||
| import lombok.Builder; | ||
| import lombok.Data; | ||
|
|
||
| /** * A DTO to store import data chunk details */ | ||
| @Data | ||
| @Builder | ||
| @JsonDeserialize(builder = ImportDataChunkStatus.ImportDataChunkStatusBuilder.class) | ||
| public class ImportDataChunkStatus { | ||
|
|
||
| @JsonProperty("dataChunkId") | ||
| private final int dataChunkId; | ||
|
|
||
| @JsonProperty("startTime") | ||
| private final Instant startTime; | ||
|
|
||
| @JsonProperty("endTime") | ||
| private final Instant endTime; | ||
|
|
||
| @JsonProperty("totalRecords") | ||
| private final int totalRecords; | ||
|
|
||
| @JsonProperty("successCount") | ||
| private final int successCount; | ||
|
|
||
| @JsonProperty("failureCount") | ||
| private final int failureCount; | ||
|
|
||
| @JsonProperty("batchCount") | ||
| private final int batchCount; | ||
|
|
||
| @JsonProperty("totalDurationInMilliSeconds") | ||
| private final int totalDurationInMilliSeconds; | ||
|
|
||
| @JsonProperty("status") | ||
| private final ImportDataChunkStatusState status; | ||
| } |
8 changes: 8 additions & 0 deletions
8
...n/java/com/scalar/db/dataloader/core/dataimport/datachunk/ImportDataChunkStatusState.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| package com.scalar.db.dataloader.core.dataimport.datachunk; | ||
|
|
||
| /** * Status of the import data chunk during the import process */ | ||
| public enum ImportDataChunkStatusState { | ||
| START, | ||
| IN_PROGRESS, | ||
| COMPLETE | ||
| } |
11 changes: 11 additions & 0 deletions
11
...ader/core/src/main/java/com/scalar/db/dataloader/core/dataimport/datachunk/ImportRow.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| package com.scalar.db.dataloader.core.dataimport.datachunk; | ||
|
|
||
| import com.fasterxml.jackson.databind.JsonNode; | ||
| import lombok.Value; | ||
|
|
||
| /** Stores data related to a single row on import file */ | ||
| @Value | ||
| public class ImportRow { | ||
| int rowNumber; | ||
| JsonNode sourceData; | ||
| } |
7 changes: 7 additions & 0 deletions
7
...er/core/src/main/java/com/scalar/db/dataloader/core/dataimport/task/ImportTaskAction.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| package com.scalar.db.dataloader.core.dataimport.task; | ||
|
|
||
| /** * Import task actions */ | ||
| public enum ImportTaskAction { | ||
| INSERT, | ||
| UPDATE, | ||
| } |
12 changes: 12 additions & 0 deletions
12
...rc/main/java/com/scalar/db/dataloader/core/dataimport/task/result/ImportResultStatus.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| package com.scalar.db.dataloader.core.dataimport.task.result; | ||
|
|
||
| public enum ImportResultStatus { | ||
| SUCCESS, | ||
| PARTIAL_SUCCESS, | ||
| FAILURE, | ||
| VALIDATION_FAILED, | ||
| RETRIEVAL_FAILED, | ||
| MAPPING_FAILED, | ||
| TIMEOUT, | ||
| CANCELLED | ||
| } |
19 changes: 19 additions & 0 deletions
19
...rc/main/java/com/scalar/db/dataloader/core/dataimport/task/result/ImportTargetResult.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| package com.scalar.db.dataloader.core.dataimport.task.result; | ||
|
|
||
| import com.fasterxml.jackson.databind.JsonNode; | ||
| import com.scalar.db.dataloader.core.dataimport.task.ImportTaskAction; | ||
| import java.util.List; | ||
| import lombok.Builder; | ||
| import lombok.Value; | ||
|
|
||
| @Builder | ||
| @Value | ||
| public class ImportTargetResult { | ||
| String namespace; | ||
| String tableName; | ||
| ImportTaskAction importAction; | ||
| List<String> errors; | ||
| boolean dataMapped; | ||
| JsonNode importedRecord; | ||
| ImportTargetResultStatus status; | ||
| } |
12 changes: 12 additions & 0 deletions
12
...n/java/com/scalar/db/dataloader/core/dataimport/task/result/ImportTargetResultStatus.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| package com.scalar.db.dataloader.core.dataimport.task.result; | ||
|
|
||
| public enum ImportTargetResultStatus { | ||
| VALIDATION_FAILED, | ||
| RETRIEVAL_FAILED, | ||
| MISSING_COLUMNS, | ||
| DATA_ALREADY_EXISTS, | ||
| DATA_NOT_FOUND, | ||
| SAVE_FAILED, | ||
| SAVED, | ||
| ABORTED | ||
| } |
25 changes: 25 additions & 0 deletions
25
.../src/main/java/com/scalar/db/dataloader/core/dataimport/task/result/ImportTaskResult.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| package com.scalar.db.dataloader.core.dataimport.task.result; | ||
|
|
||
| import com.fasterxml.jackson.annotation.JsonProperty; | ||
| import com.fasterxml.jackson.databind.JsonNode; | ||
| import com.fasterxml.jackson.databind.annotation.JsonDeserialize; | ||
| import java.util.List; | ||
| import lombok.Builder; | ||
| import lombok.Value; | ||
|
|
||
| @Builder | ||
| @Value | ||
| @JsonDeserialize(builder = ImportTaskResult.ImportTaskResultBuilder.class) | ||
| public class ImportTaskResult { | ||
| @JsonProperty("rowNumber") | ||
| int rowNumber; | ||
|
|
||
| @JsonProperty("targets") | ||
| List<ImportTargetResult> targets; | ||
|
|
||
| @JsonProperty("rawRecord") | ||
| JsonNode rawRecord; | ||
|
|
||
| @JsonProperty("dataChunkId") | ||
| int dataChunkId; | ||
| } | ||
14 changes: 14 additions & 0 deletions
14
...ava/com/scalar/db/dataloader/core/dataimport/transactionbatch/ImportTransactionBatch.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| package com.scalar.db.dataloader.core.dataimport.transactionbatch; | ||
|
|
||
| import com.scalar.db.dataloader.core.dataimport.datachunk.ImportRow; | ||
| import java.util.List; | ||
| import lombok.Builder; | ||
| import lombok.Value; | ||
|
|
||
| /** Transaction batch details */ | ||
| @Builder | ||
| @Value | ||
| public class ImportTransactionBatch { | ||
| int transactionBatchId; | ||
| List<ImportRow> sourceData; | ||
| } |
32 changes: 32 additions & 0 deletions
32
...m/scalar/db/dataloader/core/dataimport/transactionbatch/ImportTransactionBatchResult.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| package com.scalar.db.dataloader.core.dataimport.transactionbatch; | ||
|
|
||
| import com.fasterxml.jackson.annotation.JsonProperty; | ||
| import com.fasterxml.jackson.databind.annotation.JsonDeserialize; | ||
| import com.scalar.db.dataloader.core.dataimport.task.result.ImportTaskResult; | ||
| import java.util.List; | ||
| import lombok.Builder; | ||
| import lombok.Value; | ||
|
|
||
| /** Transaction batch result */ | ||
| @Builder | ||
| @Value | ||
| @JsonDeserialize(builder = ImportTransactionBatchResult.ImportTransactionBatchResultBuilder.class) | ||
| public class ImportTransactionBatchResult { | ||
| @JsonProperty("dataChunkId") | ||
| int dataChunkId; | ||
|
|
||
| @JsonProperty("transactionBatchId") | ||
| int transactionBatchId; | ||
|
|
||
| @JsonProperty("transactionId") | ||
| String transactionId; | ||
|
|
||
| @JsonProperty("records") | ||
| List<ImportTaskResult> records; | ||
|
|
||
| @JsonProperty("errors") | ||
| List<String> errors; | ||
|
|
||
| @JsonProperty("success") | ||
| boolean success; | ||
| } |
18 changes: 18 additions & 0 deletions
18
...m/scalar/db/dataloader/core/dataimport/transactionbatch/ImportTransactionBatchStatus.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| package com.scalar.db.dataloader.core.dataimport.transactionbatch; | ||
|
|
||
| import com.scalar.db.dataloader.core.dataimport.task.result.ImportTaskResult; | ||
| import java.util.List; | ||
| import lombok.Builder; | ||
| import lombok.Value; | ||
|
|
||
| /** Batch status details */ | ||
| @Builder | ||
| @Value | ||
| public class ImportTransactionBatchStatus { | ||
| int dataChunkId; | ||
| int transactionBatchId; | ||
| String transactionId; | ||
| List<ImportTaskResult> records; | ||
| List<String> errors; | ||
| boolean success; | ||
| } |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Honestly speaking, it's hard to me to get the relationship of these classes and how to use them from this PR. It would be great if there is a simple diagram or something to show the relationship.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, my previous comment might be unclear. I just read the design document again. The definition of the following terms are still unclear to me, so adding comments about the definitions of xxxxx would be enough:
batch(I guessImportTransactionBatchis related)task(I guessImportTaskActionandImportTaskResultare related)There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@komamitsu san,
Sorry for the late update, I will provide the general idea for batch amd task.
The import file is split into smaller data chunks for import initially. Each data chunk when the import is in
transaction modeis further split into transaction batches. This is indicated as abatch.In each transaction batch, each individual row of data is imported by a transaction. The process of importing each individual row via a transaction is termed as a
task.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@inv-jishnu Thanks for the explanation! I think the information will be very helpful for future code reviewers and maintainers. Could you add the description to the design doc or as source code comments?
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@komamitsu san,
I will check with @ypeckstadt and add these in the design doc.
Thank you.