Skip to content

Conversation

@inv-jishnu
Copy link
Contributor

@inv-jishnu inv-jishnu commented Dec 19, 2024

Description

I have added the export options validator and a related exception

Related issues and/or PRs

NA

Changes made

I have added the export options validator and a related exception

Checklist

The following is a best-effort checklist. If any items in this checklist are not applicable to this PR or are dependent on other, unmerged PRs, please still mark the checkboxes after you have read and understood each item.

  • I have commented my code, particularly in hard-to-understand areas.
  • I have updated the documentation to reflect the changes.
  • Any remaining open issues linked to this PR are documented and up-to-date (Jira, GitHub, etc.).
  • Tests (unit, integration, etc.) have been added for the changes.
  • My changes generate no new warnings.
  • Any dependent changes in other PRs have been merged and published.

Additional notes (optional)

Road map to merge remaining data loader core files. Current status

  • General

  • Export

  • Import

    • Dto classes and utilities: 1 PR incoming
    • Import processor and task code: 2-3 PRs incoming
    • Code for Import transaction batch and data chunk import: 1 PR Incoming
    • ControlFile related Dtos: 1 PR incoming
    • Import logger: 1 PR incoming
    • Import Manager: 1 PR incoming

Release notes

NA

@inv-jishnu inv-jishnu added the enhancement New feature or request label Dec 19, 2024
@inv-jishnu inv-jishnu self-assigned this Dec 19, 2024
@inv-jishnu inv-jishnu marked this pull request as draft December 19, 2024 12:12
@ypeckstadt ypeckstadt marked this pull request as ready for review December 19, 2024 23:49
Copy link
Contributor

@komamitsu komamitsu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 👍

if (clusteringKeyNames == null) {
return;
}
String columnName = key.getColumnName(0);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, why are you only checking the first column?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jishnu is off for a bit so I took over this task and updated the code. I have fixed the validation and completely rewrote the unit tests to cover all scenarios.

  • For the partition key(s) making sure all keys are provided and in order
  • For clustering keys, make sure the order or partial order is correct. (full key A,B,C -->. so you can call with (A), (A,B), (A,B,C))

This fix/problem exposes a limitation in the current data loader where you can only call the data loader like this

./data-loader export -pk <KEY=VALUE>. This will be fixed so you can do <KEY=VALUE>,<KEY=VALUE>. This code is coming with the next PR so it will already be fixed. But just FYI.

@inv-jishnu As mentioned, make sure to update the CLI arguments for export in the next PR to protected List<ColumnKeyValue> partitionKeyValues;

Copy link
Collaborator

@brfrn169 brfrn169 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you!

Copy link
Contributor

@Torch3333 Torch3333 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you!

@ypeckstadt ypeckstadt merged commit a2a68f9 into master Jan 7, 2025
48 checks passed
@ypeckstadt ypeckstadt deleted the feat/data-loader/export-options-validations branch January 7, 2025 05:19
feeblefakie pushed a commit that referenced this pull request Jan 7, 2025
Co-authored-by: Peckstadt Yves <peckstadt.yves@gmail.com>
inv-jishnu added a commit that referenced this pull request Jan 7, 2025
Co-authored-by: Peckstadt Yves <peckstadt.yves@gmail.com>
inv-jishnu added a commit that referenced this pull request Jan 7, 2025
Co-authored-by: Peckstadt Yves <peckstadt.yves@gmail.com>
inv-jishnu added a commit that referenced this pull request Jan 7, 2025
Co-authored-by: Peckstadt Yves <peckstadt.yves@gmail.com>
inv-jishnu added a commit that referenced this pull request Jan 7, 2025
Co-authored-by: Peckstadt Yves <peckstadt.yves@gmail.com>
inv-jishnu added a commit that referenced this pull request Jan 7, 2025
Co-authored-by: Peckstadt Yves <peckstadt.yves@gmail.com>
@inv-jishnu inv-jishnu mentioned this pull request Apr 11, 2025
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants