Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GOBBLIN-1749] Add dependency for handling xz-compressed Avro file #3609

Merged
merged 1 commit into from Feb 14, 2023

Conversation

sekikn
Copy link
Contributor

@sekikn sekikn commented Nov 26, 2022

  • Add dependency on xz for handling xz-compressed Avro files

  • Fix unit test to ensure all codecs are correctly supported

  • Update AvroHdfsDataWriter's document for covering all compression codecs

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

Description

  • Here are some details about my PR, including screenshots (if applicable):

After upgrading Avro to 1.9.2, reading and writing xz-compressed Avro file fails by default. This PR fixes it.

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:

I updated AvroHdfsDataWriterTest to ensure that all codecs are supported

Commits

  • My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

* Add dependency on xz for handling xz-compressed Avro files

* Fix unit test to ensure all codecs are correctly supported

* Update AvroHdfsDataWriter's document for covering all compression codecs
@codecov-commenter
Copy link

codecov-commenter commented Nov 26, 2022

Codecov Report

Merging #3609 (b4568ba) into master (c6d6c1b) will increase coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #3609   +/-   ##
=========================================
  Coverage     46.87%   46.88%           
- Complexity    10687    10688    +1     
=========================================
  Files          2125     2125           
  Lines         83157    83157           
  Branches       9266     9266           
=========================================
+ Hits          38983    38985    +2     
+ Misses        40598    40593    -5     
- Partials       3576     3579    +3     
Impacted Files Coverage Δ
...e/gobblin/service/monitoring/GitConfigMonitor.java 95.23% <0.00%> (-4.77%) ⬇️
...a/org/apache/gobblin/cluster/GobblinHelixTask.java 60.21% <0.00%> (-4.31%) ⬇️
.../apache/gobblin/runtime/api/JobExecutionState.java 79.43% <0.00%> (-0.94%) ⬇️
...anagement/copy/replication/ConfigBasedDataset.java 68.87% <0.00%> (ø)
...main/java/org/apache/gobblin/yarn/YarnService.java 15.68% <0.00%> (+0.65%) ⬆️
...in/java/org/apache/gobblin/cluster/HelixUtils.java 44.88% <0.00%> (+1.70%) ⬆️
...he/gobblin/source/PartitionAwareFileRetriever.java 55.55% <0.00%> (+7.40%) ⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@Will-Lo
Copy link
Contributor

Will-Lo commented Jan 10, 2023

Thank you @sekikn for the contribution!

@Will-Lo Will-Lo merged commit 26fd244 into apache:master Feb 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants