Skip to content
Import files (data) from Intercom, FTP(S), SFTP, MySQL, etc. servers into BigQuery.
Branch: master
Clone or download
Latest commit 4240ca5 Apr 15, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
amocrm Update README_RU.md Dec 16, 2018
expertsender Update README_RU.md Dec 12, 2018
ftp Update README_RU.md Dec 12, 2018
ftps Update main.py Apr 15, 2019
https Update README_RU.md Dec 12, 2018
intercom Update README.md Dec 12, 2018
mysql Update README_RU.md Dec 12, 2018
sftp Update README_RU.md Dec 12, 2018
LICENSE.md license.md added Aug 30, 2018
README.md Update README.md Dec 9, 2018
README_RU.md Update README_RU.md Dec 9, 2018

README.md

BigQuery-integrations

Import files (data) from Intercom, FTP(S), SFTP, MySQL, etc. servers into BigQuery.

This documentation is also available in RU.

What is BigQuery-integrations

BigQuery-integrations is a set of Python scripts that let you automate data import to Google BigQuery using Google Cloud Functions.

The current version of BigQuery-integrations features scripts for importing from:

How it works

An HTTP POST request invokes a Cloud function that gets the file from the server and uploads it to a BigQuery table. If the table already exists in the selected dataset, it will be rewritten.

Requirements

To launch any of these scripts, you need:

  • A Google Cloud Platform project with an activated billing account;
  • Read access to the data source;
  • The WRITER access to the dataset and Job User roles for the Cloud functions service account in the BigQuery project to which you are going to upload the table;
  • An HTTP client for POST requests invoking the Cloud function.

Setup and usage

To launch the scripts, follow the steps described in the readme files in the function folder. You don’t need to edit the functions’ code. The detailed documentation for each function is by the links below:

Questions

If you have any questions on the BigQuery-integrations scripts, write us at analytics-dev@owox.com.

You can’t perform that action at this time.