Extract gcp bigquery data to aws s3 by providing sql query in gcp pub/sub.
- external system publish sql query and aws s3 path with delimiter "|" to gcp pub/sub
- Pub/sub triggers cloud fn
- cloud fn gets authenticated and call data-flow rest api
- cloud fn extract data from big query
- compress and write data to csv files based on underline shard design in aws S3
- create a service account and provides required access to push data to gcp pub sub
- Create a service account to deploy dataflow as a service in gcp
- mvn compile exec:java
-Dexec.mainClass=com.org.bigquery.s3.dataflow.pipeline.BigQueryToCsv
-Dexec.args="--project=
--bucketUrl=s3://
--awsRegion=eu-west-1
--bqTableName=
--awsAccessKey=
--awsSecretKey=
--runner=DataflowRunner
--region=europe-west1
--stagingLocation=gs:///stage/
--tempLocation=gs:///temp/"