New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add dynamic fields to Snowflake connection #14724
Conversation
The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*. |
@kaxil thank you! |
The run associated with this PR appears to have failed because of secuity issue? |
Just re-started the test, let's see |
Just re-started the test, let's see. GH is experiencing issues at the moment: https://www.githubstatus.com/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@sunkickr I'm not sure if having AWS stuff in Snowflake connection is good idea. In future there can be any XtoSnowflake operator, will we be extending the list?
@turbaszek I agree that the AWS stuff shouldn't be in the connection, but the S3ToSnowflakeOperator is currently written in a way that requires this(I think). I thought rewriting the S3ToSnowflakeOperator was beyond the scope of this PR. Other transfers could have users set up a connection for each service so we wouldn't need to extend the list. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good! As discussed it may be worth to decouple aws from sf but that's definitely out of scope here 👌
@sunkickr can you please rebase onto current master to rerun all builds? |
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
Thank you for the review! I can do that |
771a7e8
to
57b913d
Compare
add dynamic fields to snowflake connection add dynamic fields to snowflake connection add dynamic fields to snowflake connection
a102fd5
to
3b76592
Compare
Awesome work, congrats on your first merged pull request! |
Adds form fields and custom form behavior for the Snowflake connection so it is more obvious to new users what fields need to be filled out. Also the doc_strings for the hook are updated to reflect the params along with helpful information using sphinx directives. I have categorized the fields below to explain the change.
Fields/inputs Used by the average user
Fields/inputs not used by user
Fields/inputs not used by user but reflected in code(not used by average user)
Proposal
aws_accces_key_id
andaws_secret_access_key
because aws secret key is a secretex. Login: "your snowflake username"
screenshot:
Ideally
aws_accces_key_id
andaws_secret_access_key
would be stored in Extra because they are not used by the average user. A feature in the UI that allows users to obfuscate the Extra field is needed for secretes to be stored there.This PR is related to #14631 (comment) where something went wrong during rebase and the PR closed.