Creates dbt based GCP workflows.
Use the package manager pip to install dbt-workflows-factory for [dp (data-pipelines-cli)]:
pip install dbt-workflows-factory
To call from cli, you can
python -m dbt_workflows_factory.cli convert \
--image-uri my-image-uri \
--gcs-key-volume-remote-path google-cloud-storage/path/ \
--gcs-key-volume-mount-path /etc/gcs-key/ \
--gcs-key-volume-container-mount-path /etc/gcs-key/:/etc/gcs-key/:ro \
--container-gcp-key-path /etc/gcs-key/path.json \
--container-gcp-project-id some-project-id \
--pretty \
tests/unit/dbt_workflows_factory/test_data/manifest.json > workflow_source.json
- .devcontainer - This directory contains required files for creating a Codespace.
- .github
- workflows - Contains GitHub Actions used for building, testing and publishing.
- publish.yml - Publish wheels to https://pypi.org/
- pull-request.yml - Build and Test pull requests before commiting to main.
- template-sync.yml - Update GitHub Repo with enhancments to base template
- workflows - Contains GitHub Actions used for building, testing and publishing.
- docs - collect documents (default format .md)
- src - place new source code here
- python_package - sample package (this can be deleted when creating a new repository)
- tests - contains Python based test cases to validation src code
- .pre-commit-config.yaml - Contains various pre-check fixes for Python
- pyproject.toml - Python Project Declaration
In order to publish to PyPi, a repostirory secret must be created, "PYPI_PASSWORD". In order to publish to the Test PyPi, a second secret must be added, "TEST_PYPI_PASSWORD".