Skip to content

[DOP-6713] Add Avro file format support#92

Merged
dolfinus merged 1 commit intodevelopfrom
feature/DOP-6713
Jul 31, 2023
Merged

[DOP-6713] Add Avro file format support#92
dolfinus merged 1 commit intodevelopfrom
feature/DOP-6713

Conversation

@dolfinus
Copy link
Member

@dolfinus dolfinus commented Jul 28, 2023

Change Summary

  • Added Avro class wrapping options for specific file format.
  • Added unit & integration tests.
  • Added functions for generating .avro files to generate_files.py, using avro Python package.
  • Updated check if Java class is available to use new Connection.get_packages(...) syntax instead of Connection.package.
  • Added prohibited options for Avro, ORC and Parquet - they should be passed in Spark session config, not format options. Also wrapped options with frozenset for consistency.

Related issue number

Checklist

  • Commit message and PR title is comprehensive
  • Keep the change as small as possible
  • Unit and integration tests for the changes exist
  • Tests pass on CI and coverage does not decrease
  • Documentation reflects the changes where applicable
  • docs/changelog/next_release/<pull request or issue id>.<change type>.rst file added describing change
    (see CONTRIBUTING.rst for details.)
  • My PR is ready to review.

@dolfinus dolfinus self-assigned this Jul 28, 2023
@dolfinus dolfinus added the ci:skip-changelog Add this label to skip changelog file check label Jul 28, 2023
@dolfinus dolfinus temporarily deployed to test-pypi July 28, 2023 12:06 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi July 28, 2023 12:17 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi July 28, 2023 13:05 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi July 28, 2023 13:07 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi July 28, 2023 13:12 — with GitHub Actions Inactive
@codecov
Copy link

codecov bot commented Jul 28, 2023

Codecov Report

Merging #92 (c72f95d) into develop (997955c) will decrease coverage by 0.03%.
The diff coverage is 86.73%.

@@             Coverage Diff             @@
##           develop      #92      +/-   ##
===========================================
- Coverage    93.31%   93.29%   -0.03%     
===========================================
  Files          160      161       +1     
  Lines         6774     6843      +69     
  Branches      1273     1288      +15     
===========================================
+ Hits          6321     6384      +63     
- Misses         337      343       +6     
  Partials       116      116              
Files Changed Coverage Δ
onetl/connection/db_connection/jdbc_mixin.py 94.50% <0.00%> (-0.53%) ⬇️
onetl/exception.py 100.00% <ø> (ø)
onetl/connection/db_connection/mongodb.py 93.92% <50.00%> (-0.39%) ⬇️
onetl/connection/db_connection/greenplum.py 72.72% <66.66%> (-0.31%) ⬇️
onetl/file/format/avro.py 91.80% <91.80%> (ø)
onetl/connection/db_connection/mssql.py 100.00% <100.00%> (ø)
onetl/connection/db_connection/oracle.py 86.09% <100.00%> (ø)
onetl/file/format/__init__.py 100.00% <100.00%> (ø)
onetl/file/format/csv.py 100.00% <100.00%> (ø)
onetl/file/format/file_format.py 100.00% <100.00%> (ø)
... and 4 more

... and 1 file with indirect coverage changes

@dolfinus dolfinus temporarily deployed to test-pypi July 28, 2023 13:35 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi July 28, 2023 13:36 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi July 28, 2023 13:42 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi July 28, 2023 13:47 — with GitHub Actions Inactive
@dolfinus dolfinus marked this pull request as ready for review July 28, 2023 13:59
@dolfinus dolfinus merged commit 923010c into develop Jul 31, 2023
@dolfinus dolfinus deleted the feature/DOP-6713 branch July 31, 2023 11:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:skip-changelog Add this label to skip changelog file check

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants