Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Allow SQL tap developers to leverage post_process #1681

Conversation

BuzzCutNorman
Copy link
Contributor

@BuzzCutNorman BuzzCutNorman commented May 8, 2023

Gives SQL Streams the ability to utilize the Core Stream post_process method out of the box just like REST Streams.

Closes #1674


📚 Documentation preview 📚: https://meltano-sdk--1681.org.readthedocs.build/en/1681/

@kgpayne
Copy link
Contributor

kgpayne commented May 9, 2023

Thanks @BuzzCutNorman 🙌 Out of interest, do you have a link to a Tap where you are using or want to use this functionality? What kinds of transforms are you doing?

@codecov
Copy link

codecov bot commented May 9, 2023

Codecov Report

Merging #1681 (0e689c4) into main (1583937) will decrease coverage by 0.04%.
The diff coverage is 50.00%.

❗ Current head 0e689c4 differs from pull request most recent head ab319af. Consider uploading reports for the commit ab319af to get more accurate results

@@            Coverage Diff             @@
##             main    #1681      +/-   ##
==========================================
- Coverage   85.11%   85.08%   -0.04%     
==========================================
  Files          59       59              
  Lines        4865     4868       +3     
  Branches      806      807       +1     
==========================================
+ Hits         4141     4142       +1     
- Misses        526      527       +1     
- Partials      198      199       +1     
Impacted Files Coverage Δ
singer_sdk/streams/sql.py 86.36% <50.00%> (-2.53%) ⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@BuzzCutNorman
Copy link
Contributor Author

@kgpayne I am using this in my tap-mssql variant. Here is the link to the post_process method in tap-mssql.
https://github.com/BuzzCutNorman/tap-mssql/blob/e1a877df2c30f5fb7bbdec37873459f3af8f56e9/tap_mssql/client.py#L408. I am doing a date fix and encoding binary strings.

@edgarrmondragon edgarrmondragon changed the title feat: Add post_process to SQLStream get_records feat: Allow SQL tap developers to leverage post_process May 9, 2023
Copy link
Collaborator

@edgarrmondragon edgarrmondragon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @BuzzCutNorman!

@edgarrmondragon edgarrmondragon merged commit 7430f77 into meltano:main May 9, 2023
22 checks passed
@BuzzCutNorman BuzzCutNorman deleted the 1674-add-postprocess-to-sqlstream-getrecords branch December 8, 2023 18:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feat: Add post_process to SQLStream.get_records to match RESTStream functionality
3 participants