Re-haul this example with more realistic example #2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
author Jacob Ferriero <jferriero@google.com> 1567538559 -0400 committer Jacob Ferriero <jferriero@google.com> 1580182917 -0800 Improve to use airflow and testing best practices improve python readability add module docstring to xcom_utils_plugin make docstrings consistent change directory references in build-pipeline fix yaml typos fix test fix dir in build deploy test yaml add dag validation test add dag validation test set AIRFLOW_HOME in run_tests.sh moving airflow homee trying to get local airflow working fix typo typoe fix initdb ordering in run_tests.sh improve testing step name plugin imports for testing stage variables in gcs data dir remove unused Dockerfile update solution url specify airflow docker image version tag change on docker hub Improve to use airflow and testing best practices improve python readability add module docstring to xcom_utils_plugin make docstrings consistent change directory references in build-pipeline fix yaml typos fix test fix dir in build deploy test yaml add dag validation test add dag validation test set AIRFLOW_HOME in run_tests.sh moving airflow homee trying to get local airflow working fix typo typoe fix initdb ordering in run_tests.sh improve testing step name plugin imports for testing stage variables in gcs data dir remove unused Dockerfile fix region typo removing double gs:// Add rendering of enviornment variables add perms change add dir trailing slash add ./ add debugging simplify simplify composer variables fix plugins modify jar ref after importing variables. fix typo in BUILD and add print statements render Variables.json add variables renderer remove envsubst hard code Variables.json add script to render Variables.json change to rsync dags folder fix typo lowercase input.txt add create_composer_environment script parent 0e7b039 author Jacob Ferriero <jferriero@google.com> 1567542258 -0400 committer Jacob Ferriero <jferriero@google.com> 1567721062 -0400 improve python readability add module docstring to xcom_utils_plugin make docstrings consistent change directory references in build-pipeline fix yaml typos fix test fix dir in build deploy test yaml add dag validation test add dag validation test set AIRFLOW_HOME in run_tests.sh moving airflow homee trying to get local airflow working fix typo typoe fix initdb ordering in run_tests.sh improve testing step name plugin imports for testing stage variables in gcs data dir remove unused Dockerfile update solution url specify airflow docker image version tag change on docker hub Improve to use airflow and testing best practices improve python readability add module docstring to xcom_utils_plugin make docstrings consistent change directory references in build-pipeline fix yaml typos fix test fix dir in build deploy test yaml add dag validation test add dag validation test set AIRFLOW_HOME in run_tests.sh moving airflow homee trying to get local airflow working fix typo typoe fix initdb ordering in run_tests.sh improve testing step name plugin imports for testing stage variables in gcs data dir remove unused Dockerfile fix region typo removing double gs:// Add rendering of enviornment variables add perms change add dir trailing slash add ./ add debugging simplify simplify composer variables fix plugins modify jar ref after importing variables. fix typo in BUILD and add print statements render Variables.json add variables renderer remove envsubst hard code Variables.json add script to render Variables.json change to rsync dags folder fix typo lowercase input.txt deploy_dag.sh latest add airflow monitoring to running_dags test improve delete handling of null add deploy_dags with robust delete handling test both dags remove tree fix typo remove commented out code PROJECT_ID shell updates fail on failed tests remove airflow install from run_tests.sh remove prod_word_count form running_dags.txt updates for sql dir build iam env vars fix dag test fix strings in prod only test dags in running_dags merge yaml move build running_dag.txt empty update build updating prod variable remove test daG build iam env vars fix dag test fix strings in prod only test dags in running_dags merge yaml move build running_dag.txt empty update build updating prod variable remove test daG improve step id for deploy dag add sql to prod_word_count no dags prod_word_count airflow config fix bql sql update set_env update word could bq pin airflow version deploy sql fix local directory ref for sql/ update sql path getter don't deploy dags in background update airflow image no dags fix sql prefix getter update gitignore add word count read sql file fix sql reader fix sql webserver fix test sql path fix cloudbuild sql dir update README docs clean up polishing README and test_dag_validation Replacing Solution proposed ci-cd-for-data-processing-workflow 2.0 remove old setup scripts scrub project ref
added 28 commits
March 25, 2020 16:15
fix repo reference to github
add more comments to cloudbuild.yaml
Feature/linters
terraform and bash linting
* Revert "Revert "restructure for nested cloudbuild"" This reverts commit f36fa02. * fix cloudbuild missing comma * remove unused refs * terragrunt multi-project * remove merge dep in precommit * init git repo * revise pre-commit strategy * checkout pr * add empty commit * add pr number env var * add pr num to precommit * patch post commit refs * fix post commit refs and remove terragrunt cache * missing comma * fix init repo * try agian * walk dagsRoot * patch composer ops * revent dags deployer changes * debug print * add func * more debug * func * where does deploy dags run? * fixed dags deployer * run precommits * fix redundant dir in dataflow cloudbuild.yaml * dataflow builds * fix makefile * fix cloud build finder * make extra args optional * add separate ids * use unittest cuz pytest not in airflow image * package the whole repo for precommits * yapf * remove jinja templating * fix test_sql.sh * fix root cloudbuild typo * fix root cloudbuild * fix reference to helper script * fix passing of additional build args * make additional args optional * use default machine type in precommit * fix wordcount cloudbuild * fix wordcount cloudbuild * improve wordcount cloudbuild variable overriding Co-authored-by: ia-tests <ia-tests@presubmit.example.com>
|
We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. ℹ️ Googlers: Go here for more info. |
[skip ci] update readmes
Demoprep
remove example dags
remove all running_dags.txt
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a proposed change which would require some refactoring of the solutions page.
The current solutions page is not inline with what PSO is seeing in the field for a good example for composer deployments.
The following gaps would be addressed by this PR:
see more details in my fork's closed issues