Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOP-7957] Rename mode -> if_exists, and make modes more descriptive #108

Merged
merged 1 commit into from
Aug 11, 2023

Conversation

dolfinus
Copy link
Member

@dolfinus dolfinus commented Aug 8, 2023

Change Summary

  • Renamed everywhere mode -> if_exists. New name should be more descriptive than old one, even if it is different from Spark naming.
  • For DB connections renamed overwrite and overwrite_table to replace_entire_table. For MongoDB it is replace_entire_collection.
  • For Hive renamed overwrite_partitions -> replace_overlapping_partitions (to match FileWriter.Options.if_exists)
  • For FileDownloader, FileUploader and FileMover renamed mode values:
    • overwrite -> replace_file (should be more descriptive)
    • delete_all -> replace_entire_directory (for consistent naming with other classes, also more desciptive)
  • Old names are automatically converted to new ones for backward compatibility, but with warnings emitted. Added related unit tests.
  • Updated integration tests. For Hive added more test cases using parametrize, e.g. writing to partitioned table without passing partitionBy, or vice versa.
  • Updated examples and documentation. For Hive updated WriteOptions attributes documentation to match FileWriter.Options (for Spark they its just the same options).

Related issue number

Checklist

  • Commit message and PR title is comprehensive
  • Keep the change as small as possible
  • Unit and integration tests for the changes exist
  • Tests pass on CI and coverage does not decrease
  • Documentation reflects the changes where applicable
  • docs/changelog/next_release/<pull request or issue id>.<change type>.rst file added describing change
    (see CONTRIBUTING.rst for details.)
  • My PR is ready to review.

@dolfinus dolfinus self-assigned this Aug 8, 2023
@dolfinus dolfinus temporarily deployed to test-pypi August 8, 2023 14:31 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi August 8, 2023 14:35 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi August 8, 2023 15:14 — with GitHub Actions Inactive
@codecov
Copy link

codecov bot commented Aug 8, 2023

Codecov Report

Merging #108 (63eedf0) into develop (c97e907) will increase coverage by 0.15%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           develop     #108      +/-   ##
===========================================
+ Coverage    93.72%   93.87%   +0.15%     
===========================================
  Files          169      169              
  Lines         7285     7352      +67     
  Branches      1404     1428      +24     
===========================================
+ Hits          6828     6902      +74     
+ Misses         332      328       -4     
+ Partials       125      122       -3     
Files Changed Coverage Δ
onetl/db/db_writer/db_writer.py 85.41% <ø> (ø)
onetl/connection/db_connection/greenplum.py 75.19% <100.00%> (+0.91%) ⬆️
onetl/connection/db_connection/hive.py 93.58% <100.00%> (+1.14%) ⬆️
onetl/connection/db_connection/jdbc_connection.py 96.05% <100.00%> (+3.38%) ⬆️
onetl/connection/db_connection/jdbc_mixin.py 95.58% <100.00%> (-0.03%) ⬇️
onetl/connection/db_connection/mongodb.py 95.34% <100.00%> (+0.18%) ⬆️
onetl/file/file_downloader/file_downloader.py 95.01% <100.00%> (+0.08%) ⬆️
onetl/file/file_mover/file_mover.py 97.28% <100.00%> (+0.09%) ⬆️
onetl/file/file_uploader/file_uploader.py 94.95% <100.00%> (+0.14%) ⬆️
onetl/file/file_writer/options.py 93.44% <100.00%> (-3.28%) ⬇️
... and 2 more

... and 2 files with indirect coverage changes

@dolfinus dolfinus temporarily deployed to test-pypi August 8, 2023 15:38 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi August 8, 2023 15:39 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi August 8, 2023 15:42 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi August 8, 2023 15:45 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi August 8, 2023 15:57 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi August 9, 2023 09:37 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi August 9, 2023 09:39 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi August 9, 2023 09:55 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi August 9, 2023 09:58 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi August 9, 2023 09:59 — with GitHub Actions Inactive
@dolfinus dolfinus force-pushed the feature/DOP-7957 branch 2 times, most recently from 0473293 to e4205ce Compare August 9, 2023 10:36
@dolfinus dolfinus temporarily deployed to test-pypi August 9, 2023 10:36 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi August 9, 2023 10:42 — with GitHub Actions Inactive
@dolfinus dolfinus marked this pull request as ready for review August 9, 2023 10:58
@dolfinus dolfinus temporarily deployed to test-pypi August 9, 2023 13:05 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi August 9, 2023 13:09 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi August 9, 2023 13:15 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi August 10, 2023 08:15 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi August 10, 2023 08:23 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi August 11, 2023 08:19 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi August 11, 2023 09:29 — with GitHub Actions Inactive
@dolfinus dolfinus merged commit abf0590 into develop Aug 11, 2023
49 checks passed
@dolfinus dolfinus deleted the feature/DOP-7957 branch August 11, 2023 12:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants