Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add Parquet as a batch encoding option #2044

Merged
merged 51 commits into from
Nov 28, 2023

Conversation

jamielxcarter
Copy link
Contributor

@jamielxcarter jamielxcarter commented Nov 14, 2023

Closes #1148


📚 Documentation preview 📚: https://meltano-sdk--2044.org.readthedocs.build/en/2044/

jamielxcarter and others added 30 commits November 7, 2022 16:43
Don't currently have a way to support different numeric types like
Decimal.decimal. Reverting to using pyarrow's
schema inference.
Copy link

codspeed-hq bot commented Nov 14, 2023

CodSpeed Performance Report

Merging #2044 will not alter performance

Comparing jamielxcarter:parquet-encoding (3003777) with main (2289173)

Summary

✅ 4 untouched benchmarks

Copy link

codecov bot commented Nov 14, 2023

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (2289173) 87.50% compared to head (3003777) 87.65%.

Files Patch % Lines
singer_sdk/sinks/core.py 90.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2044      +/-   ##
==========================================
+ Coverage   87.50%   87.65%   +0.14%     
==========================================
  Files          58       60       +2     
  Lines        4882     4940      +58     
  Branches      996     1006      +10     
==========================================
+ Hits         4272     4330      +58     
  Misses        429      429              
  Partials      181      181              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@edgarrmondragon
Copy link
Collaborator

FYI: I plan to give this a proper review next Tuesday

@edgarrmondragon edgarrmondragon self-assigned this Nov 17, 2023
Copy link
Collaborator

@edgarrmondragon edgarrmondragon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jamielxcarter, great work!

@edgarrmondragon edgarrmondragon added this pull request to the merge queue Nov 28, 2023
Merged via the queue into meltano:main with commit 24127d0 Nov 28, 2023
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature proposal: Add Parquet as a Batch encoding option
2 participants