Import files (data) from Intercom, FTP(S), SFTP, MySQL, etc. servers into BigQuery.
This documentation is also available in RU.
BigQuery-integrations is a set of Python scripts that let you automate data import to Google BigQuery using Google Cloud Functions.
The current version of BigQuery-integrations features scripts for importing from:
An HTTP POST request invokes a Cloud function that gets the file from the server and uploads it to a BigQuery table. If the table already exists in the selected dataset, it will be rewritten.
To launch any of these scripts, you need:
- A Google Cloud Platform project with an activated billing account;
- Read access to the data source;
- The WRITER access to the dataset and Job User roles for the Cloud functions service account in the BigQuery project to which you are going to upload the table;
- An HTTP client for POST requests invoking the Cloud function.
To launch the scripts, follow the steps described in the readme files in the function folder. You don’t need to edit the functions’ code. The detailed documentation for each function is by the links below:
- amoCRM (RU);
- FTP;
- FTPS;
- HTTP(s);
- Intercom;
- ExpertSender (RU);
- MySQL;
- SFTP.