Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add a results_format field to run_raw_sql #1584

Merged
merged 13 commits into from
Jan 23, 2023
Merged

Conversation

jlaneve
Copy link
Collaborator

@jlaneve jlaneve commented Jan 13, 2023

Description

related: #1583

This PR extends the run_raw_sql to make it easier to get results. Previously, you'd have to write a handler yourself. The idea behind this change is to abstract some of the common handlers (get the results as a list, dataframe).

Specifically, it:

  • introduces two new fields to the run_raw_sql operator:
    • results_format: let the user specify a common format they want their results (list, dataframe)
    • fail_on_empty: let the user decide what the behavior should be if there are no results

Does this introduce a breaking change?

No, everything is incremental

Checklist

  • Created tests which fail without the change (if possible)
  • Extended the README / documentation, if necessary

@codecov
Copy link

codecov bot commented Jan 13, 2023

Codecov Report

Base: 97.72% // Head: 92.95% // Decreases project coverage by -4.78% ⚠️

Coverage data is based on head (2c4b414) compared to base (2f8c8d0).
Patch coverage: 100.00% of modified lines in pull request are covered.

❗ Current head 2c4b414 differs from pull request most recent head 640f97e. Consider uploading reports for the commit 640f97e to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1584      +/-   ##
==========================================
- Coverage   97.72%   92.95%   -4.78%     
==========================================
  Files          21       89      +68     
  Lines         835     4570    +3735     
  Branches        0      450     +450     
==========================================
+ Hits          816     4248    +3432     
- Misses         19      230     +211     
- Partials        0       92      +92     
Impacted Files Coverage Δ
python-sdk/src/astro/constants.py 94.00% <100.00%> (ø)
python-sdk/src/astro/databases/base.py 90.30% <100.00%> (ø)
python-sdk/src/astro/databases/databricks/delta.py 83.68% <100.00%> (ø)
python-sdk/src/astro/sql/operators/raw_sql.py 93.25% <100.00%> (ø)
python-sdk/src/astro/sql/operators/cleanup.py 96.15% <0.00%> (ø)
python-sdk/src/astro/utils/load.py 92.30% <0.00%> (ø)
python-sdk/src/astro/settings.py 100.00% <0.00%> (ø)
python-sdk/src/astro/files/base.py 100.00% <0.00%> (ø)
python-sdk/src/astro/databases/google/bigquery.py 88.83% <0.00%> (ø)
python-sdk/src/astro/files/locations/base.py 77.88% <0.00%> (ø)
... and 62 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Copy link
Collaborator

@kaxil kaxil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address my comments around logs, docs, example DAGs and then feel free to merge

@kaxil kaxil linked an issue Jan 19, 2023 that may be closed by this pull request
9 tasks
@utkarsharma2 utkarsharma2 merged commit 5cb32f0 into main Jan 23, 2023
@utkarsharma2 utkarsharma2 deleted the raw-sql-results-format branch January 23, 2023 22:19
@utkarsharma2 utkarsharma2 mentioned this pull request Jan 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add interface to get results in standard formats from run_raw_sql
3 participants