New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automate ingest GCS to BQ data load #26

Closed

nyghtowl opened this issue Sep 21, 2020 · 1 comment

Assignees

Labels

core

Contributor

nyghtowl commented Sep 21, 2020

Add automation ingesting data from GCS to BQ

Proposal is use Cloud Functions with triggers on GCS buckets

nyghtowl mentioned this issue

Automate Pipermail and Mailman data ingestion #20

Closed

nyghtowl assigned emikuszewski

Contributor Author

nyghtowl commented Oct 19, 2020 •

edited

Setup Cloud Functions / Data Flow? triggered by GCS bucket updates to clean up and and load BQ
Evaluate if need to include with Run/Build

nyghtowl added the core label

nyghtowl self-assigned this

nyghtowl mentioned this issue

Adding Cloud Function Storage to BQ #64

Merged

nyghtowl closed this as completed

nyghtowl mentioned this issue

First half of pipeline automation to monthly pull mailing list data #66

Merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment