This is a demo of how the gusty package works with Airflow to assist in the organization, construction, and management of DAGs, tasks, dependencies, and operators. It requires that you have Docker and Docker Compose installed on your machine.
Running the demo
Note: this demo takes some time to build, and a little effort up front. If you are in a hurry, please check out gusty-demo-lite.
cryptography package, generate Fernet keys using the following line. (You will need two)
from cryptography.fernet import Fernet Fernet.generate_key().decode()
These keys will be used to encrypt passwords in our connections, and power the Airflow webserver.
Create a .env
Save the generated keys and a default user/password in a
.env file in the same directory as your
.env file should look like this:
DEFAULT_USER=your_username DEFAULT_PASSWORD=your_password FERNET_KEY='a_fernet_key' SECRET_KEY='another_fernet_key'
Build and run
Build with the following (this may take some time):
Then run with the following:
Airflow should be available for you at
localhost:8080. You can log in with the
DEFAULT_PASSWORD from your
After you turn on the example DAGs and let them run, you can connect to the the containerized Postgres database with a URI akin to
postgresql://DEFAULT_USER:DEFAULT_PASSWORD@localhost:5678/datawarehouse. A reminder all tables are housed under the schema
Please feel free to create issues, or fork and use to start your own gusty pipeline. Your feedback is very important. Hope you consider using gusty in your data pipeline projects.