Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOP-6710] Add ORC file format support #86

Merged
merged 1 commit into from
Jul 26, 2023
Merged

Conversation

dolfinus
Copy link
Member

@dolfinus dolfinus commented Jul 26, 2023

Change Summary

  • Added ORC class wrapping options for specific file format.
  • Added unit & integration tests.
  • Added functions for generating .orc files to generate_files.py, using PyArrow.
  • Small documentation updates related to passing custom options to classes.

Related issue number

Checklist

  • Commit message and PR title is comprehensive
  • Keep the change as small as possible
  • Unit and integration tests for the changes exist
  • Tests pass on CI and coverage does not decrease
  • Documentation reflects the changes where applicable
  • docs/changelog/next_release/<pull request or issue id>.<change type>.rst file added describing change
    (see CONTRIBUTING.rst for details.)
  • My PR is ready to review.

@dolfinus dolfinus self-assigned this Jul 26, 2023
@dolfinus dolfinus added the ci:skip-changelog Add this label to skip changelog file check label Jul 26, 2023
@codecov
Copy link

codecov bot commented Jul 26, 2023

Codecov Report

Merging #86 (5d03025) into develop (abdcdf9) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           develop      #86      +/-   ##
===========================================
+ Coverage    93.04%   93.06%   +0.01%     
===========================================
  Files          155      156       +1     
  Lines         6575     6592      +17     
  Branches      1209     1213       +4     
===========================================
+ Hits          6118     6135      +17     
  Misses         340      340              
  Partials       117      117              
Files Changed Coverage Δ
onetl/connection/db_connection/greenplum.py 69.33% <ø> (ø)
onetl/connection/db_connection/hive.py 92.37% <ø> (ø)
onetl/connection/db_connection/jdbc_connection.py 92.67% <ø> (ø)
onetl/connection/db_connection/kafka/options.py 100.00% <ø> (ø)
onetl/connection/db_connection/mongodb.py 93.54% <ø> (ø)
onetl/file/file_reader/options.py 100.00% <ø> (ø)
onetl/file/file_writer/options.py 96.77% <ø> (ø)
onetl/file/format/csv.py 100.00% <ø> (ø)
onetl/file/format/json.py 100.00% <ø> (ø)
onetl/file/format/jsonline.py 100.00% <ø> (ø)
... and 2 more

@dolfinus dolfinus temporarily deployed to test-pypi July 26, 2023 08:13 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi July 26, 2023 08:17 — with GitHub Actions Inactive
@dolfinus dolfinus marked this pull request as ready for review July 26, 2023 08:18
@dolfinus dolfinus temporarily deployed to test-pypi July 26, 2023 08:22 — with GitHub Actions Inactive
@dolfinus dolfinus merged commit d318108 into develop Jul 26, 2023
36 checks passed
@dolfinus dolfinus deleted the feature/DOP-6710 branch July 26, 2023 11:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci:skip-changelog Add this label to skip changelog file check
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants