Add hook and operator for google cloud life sciences #8481
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kaxil
reviewed
Apr 22, 2020
kaxil
reviewed
Apr 22, 2020
kaxil
reviewed
Apr 22, 2020
Member
kaxil
left a comment
kaxil
left a comment
There was a problem hiding this comment.
Good work, however can you please rebase on the latest master to fix the failing CI.
mik-laj
reviewed
Apr 22, 2020
airflow/providers/google/cloud/example_dags/example_life_sciences.py
Outdated
Show resolved
Hide resolved
mik-laj
reviewed
Apr 22, 2020
mik-laj
reviewed
Apr 22, 2020
Member
|
I have run the system test and I have the error message below. I suspect the file you are referring to does not exist in my bucket. Can you create it in the |
Contributor
Author
|
Hi @mik-laj , I didn't run the system tests. I have not been able to set it up. However I did use example dag to test it in real scenario and it worked. Can you point me to how I can be able to create and upload file to storage so that the system can be able to run the test. I am currently trying hard to do the system test myself. Thanks |
Member
|
The system tests only work if you have run breeze with a MySQL or Postgres database. To run system tests for GCP you need two things:
The key can be any key that has sufficient permissions. I often use the key with the "Project owner" role. The key should be saved in a These variables are important because they provide test isolation or are required by authorization. When we have everything configured, you can run the following command to run the test. During development, to make sure everything works, I often run tests using the following command. That way I can be sure that everything works. The pytest displays helpful error messages. However, you can get lost in them if you run tests for the entire project. The crucial is the "short test summary info" section This message indicates that the key is missing and you must create it. More information about creating a service account is available: https://cloud.google.com/iam/docs/creating-managing-service-account-keys#iam-service-accounts-upload-gcloud Details |
mik-laj
reviewed
Apr 25, 2020
Comment on lines
21
to
25
Member
There was a problem hiding this comment.
Suggested change
| This DAG relies on the following OS environment variables: | |
| * GCP_PROJECT_ID - Google Cloud Project to use for the Cloud Function. | |
| * GCP_GCS_BUCKET - Google Cloud Storage Bucket to use | |
| * GCP_LIFE_SCIENCES_LOCATION - The Location of the Google Cloud Project |
We do not need variable lists because they are very difficult to maintain in good condition. In this case, one variable is missing.
Contributor
Author
|
Thanks so much @mik-laj , you make things so easy! Will like to be like you when I grow up on this!! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses #8272
Make sure to mark the boxes below before creating PR: [x]
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.