Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation around LoadOptions with example DAGs #1567

Merged
merged 3 commits into from
Jan 11, 2023

Conversation

sunank200
Copy link
Contributor

Description

What is the current behavior?

In Astro SDK 1.4 we introduce a series of LoadOptions classes for classes such as databricks, pandas, and snowflake. Each of these classes is unique and we should create documentation that best showcases how users can use these classes to create the loading settings they want.

closes: #1556

What is the new behavior?

  • Add example DAG using PandasLoadOptions, SnowflakeLoadOptions and DeltaLoadOptions
  • Add the documentation for PandasLoadOptions, SnowflakeLoadOptions and DeltaLoadOptions. Add detailed documentation of how a user can use these classes.
  • Add examples for PandasOptions, DeltaOptions, and Snowflake Options.

Does this introduce a breaking change?

No

Checklist

  • Created tests which fail without the change (if possible)
  • Extended the README/documentation, if necessary

@sunank200
Copy link
Contributor Author

sunank200 commented Jan 11, 2023

Screenshots of how docs render:
Screenshot 2023-01-11 at 6 29 20 PM
Screenshot 2023-01-11 at 6 29 27 PM
Screenshot 2023-01-11 at 6 30 38 PM

Screenshot 2023-01-11 at 6 30 11 PM

Copy link
Contributor

@pankajastro pankajastro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@codecov
Copy link

codecov bot commented Jan 11, 2023

Codecov Report

Base: 97.71% // Head: 94.24% // Decreases project coverage by -3.46% ⚠️

Coverage data is based on head (621b8a3) compared to base (763f64e).
Patch has no changes to coverable lines.

❗ Current head 621b8a3 differs from pull request most recent head da148c4. Consider uploading reports for the commit da148c4 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1567      +/-   ##
==========================================
- Coverage   97.71%   94.24%   -3.47%     
==========================================
  Files          22       89      +67     
  Lines         787     4448    +3661     
  Branches        0      444     +444     
==========================================
+ Hits          769     4192    +3423     
- Misses         18      173     +155     
- Partials        0       83      +83     
Impacted Files Coverage Δ
...sdk/src/astro/databases/databricks/load_options.py 96.42% <ø> (ø)
python-sdk/src/astro/dataframes/load_options.py 100.00% <ø> (ø)
python-sdk/src/astro/options.py 96.55% <ø> (ø)
...hon-sdk/src/astro/files/locations/google/gdrive.py 97.64% <0.00%> (ø)
python-sdk/src/astro/custom_backend/serializer.py 54.32% <0.00%> (ø)
python-sdk/src/astro/files/types/ndjson.py 100.00% <0.00%> (ø)
python-sdk/src/astro/databases/postgres.py 94.50% <0.00%> (ø)
python-sdk/src/astro/files/locations/sftp.py 100.00% <0.00%> (ø)
python-sdk/src/astro/sql/table.py 100.00% <0.00%> (ø)
python-sdk/src/astro/databases/databricks/delta.py 91.42% <0.00%> (ø)
... and 60 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@utkarsharma2
Copy link
Collaborator

@sunank200 Just a minor comment, we need to add these options in specific sections

  1. Parameters to use when loading a file to a database table
  2. Parameters to use when loading a file to a Pandas dataframe
  3. Parameters for native transfer

It may be tough to list relevant options as per specific sections, maybe adding a generic section and referencing it makes more sense. WDYT?

@sunank200
Copy link
Contributor Author

sunank200 commented Jan 11, 2023

@sunank200 Just a minor comment, we need to add these options in specific sections

  1. Parameters to use when loading a file to a database table
  2. Parameters to use when loading a file to a Pandas dataframe
  3. Parameters for native transfer

It may be tough to list relevant options as per specific sections, maybe adding a generic section and referencing it makes more sense. WDYT?

@utkarsharma2 referenced different locations as mentioned. da148c4

Also addressed comments from this PR: #1566

@sunank200
Copy link
Contributor Author

Added a screenshot for the same.

Screenshot 2023-01-11 at 7 50 28 PM
Screenshot 2023-01-11 at 7 50 55 PM

@sunank200 sunank200 merged commit 7fd61f7 into main Jan 11, 2023
@sunank200 sunank200 deleted the load-options-documentation branch January 11, 2023 14:14
utkarsharma2 pushed a commit that referenced this pull request Jan 17, 2023
# Description
## What is the current behavior?
<!-- Please describe the current behavior that you are modifying. -->
In Astro SDK 1.4 we introduce a series of LoadOptions classes for
classes such as databricks, pandas, and snowflake. Each of these classes
is unique and we should create documentation that best showcases how
users can use these classes to create the loading settings they want.

<!--
Issues are required for both bug fixes and features.
Reference it using one of the following:

closes: #ISSUE
related: #ISSUE
-->
closes: #1556


## What is the new behavior?
<!-- Please describe the behavior or changes that are being added by
this PR. -->

- Add example DAG using `PandasLoadOptions`, `SnowflakeLoadOptions` and
`DeltaLoadOptions`
- Add the documentation for `PandasLoadOptions`, `SnowflakeLoadOptions`
and `DeltaLoadOptions`. Add detailed documentation of how a user can use
these classes.
- Add examples for PandasOptions, DeltaOptions, and Snowflake Options.

## Does this introduce a breaking change?
No

### Checklist
- [x] Created tests which fail without the change (if possible)
- [x] Extended the README/documentation, if necessary
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create documentation for LoadOptions classes
3 participants