Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Push Data to GCS and BigQuery #299

Merged
merged 4 commits into from
Jul 2, 2024
Merged

Push Data to GCS and BigQuery #299

merged 4 commits into from
Jul 2, 2024

Conversation

lprzychodzien
Copy link
Collaborator

@lprzychodzien lprzychodzien commented Jul 1, 2024 •
edited
Loading

Explanation

  • Added functionality to upload to Google Cloud Storage (GCS) bucket
  • Added functionality to create datasets on Google Big Query (BQ)
  • Created a new postgres table to track available data materialized as tables (such as DBT) to then be able to reference for upload
  • Ran DBT runs in dbt container, using docker network
  • Created function to upload CSVs (sources) created by DAG to GCS

Tests

  1. What testing did you do?
  • Ran several DAGs
  • Uploaded data to GCS and then linked to datasets on GCP though new DAG

image

image

@lprzychodzien
gcp workflow
c419240
@lprzychodzien
enit
7d51707 
init2

finalize logic to push and create datasets

finalize logic to push and create datasets

rm gcp

spacing fix
@lprzychodzien
fix avail
19679f5
@lprzychodzien
gcs common functions
f83d8c0
@lprzychodzien lprzychodzien changed the title Gcp push Push Data to GCS and BigQuery Jul 1, 2024
jrlegrand
jrlegrand approved these changes Jul 1, 2024
View reviewed changes
Copy link
Member

@jrlegrand jrlegrand left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the airflow-dbt network doesn't work, it's a matter of just removing the docker exec dbt part of the subprocess - as long as we leave dbt as being installed in the airflow-webserver container.

@lprzychodzien lprzychodzien merged commit 9441d0d into main Jul 2, 2024
@lprzychodzien lprzychodzien deleted the gcp_push branch July 2, 2024 10:44
@jrlegrand jrlegrand mentioned this pull request Jul 8, 2024
Open
@jrlegrand
Copy link
Member

Fixes #295

@jrlegrand
Copy link
Member

Fixes #287

@jrlegrand jrlegrand mentioned this pull request Jul 8, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jrlegrand jrlegrand jrlegrand approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@lprzychodzien @jrlegrand