Ingestly is a simple tool for ingesting beacons to Google BigQuery. Digital Marketers and Front-end Developers often want to measure user's activities on their service without limitations and/or sampling, in real-time, having ownership of data, within reasonable cost. There are huge variety of web analytics tools in the market but those tools are expensive, large footprint, less flexibility, fixed UI, and you will be forced to use SDKs including legacy technologies like
Ingestly is focusing on Data Ingestion from the front-end to Google BigQuery by leveraging Fastly's features. Also, Ingestly can be implemented seamlessly into your existing web site with in the same Fastly service, so you can own your analytics solution and ITP does not matter.
- Completely server-less. Fastly and Google manages all of your infrastructure for Ingestly. No maintenance resource required.
- Near real-time data in Google BigQuery. You can get the latest data in less than seconds just after user's activity.
- Fastest response time for beacons. The endpoint is Fastly's global edge nodes, no backend, response is HTTP 204 and SDK uses ASYNC request.
- Direct ingestion into Google BigQuery. You don't need to configure any complicated integrations, no need to export/import by batches.
- Easy to start. You can start using Ingestly within 2 minutes for free if you already have a trial account on Fastly and GCP.
- WebKit's ITP friendly. The endpoint issues 1st party cookie with secure flags.
- A Google Cloud Platform account, and a project used for Ingestly.
- A Fastly account, and a service used for Ingestly.
- This endpoint issues a cookie named
ingestlyIdunder your specified domain name.
Note that a GCP project and a Fastly service can be created for Ingestly or you can use your existing one.
Google Cloud Platform
Create a service account for Fastly
- Go to the GCP console, then open
IAM & admin>
- Create a service account like
ingestlyand grant a
BigQuery Data Ownerpermission.
- Create a key and download it as JSON format.
- Open the JSON you just downloaded and note
Create a table for the log data on BigQuery
- Go to the GCP console, then open
- Create a dataset like
Ingestlyif you haven't have.
- Create a table with your preferred table name like
logs, then enable
Edit as textin Schema section. (note your table name)
table_schemafile in this repository, copy the content and paste it to the schema text box of table creation modal.
- In the
Partition and cluster settingssection, Select
timestampcolumn for partitioning.
- Finish creating the table.
ingestly.vclfile in this repository and change a value for
cookie_domainto make a cookie compatible with your domain, then save the file.
Custom VCLin CONFIGURE page under your service.
Upload a VCL filebutton, then set preferred name like
ingestly.vcland upload the file.
Integrate with Google BigQuery
Loggingin CONFIGURE page.
CREATE ENDPOINTbutton and select
attach a condition.link near highlighted
CONDITION, and select
CREATE A NEW RESPONSE CONDITION.
- Enter a name like
Data Ingestionand set
(resp.status == 204 && req.url ~ "^/ingestly-ingest/(.*?)/\?.*" || resp.status == 200 && req.url ~ "^/ingestly-sync/(.*?)/\?.*")into
- Fill information into fields:
Name: anything you want.
Log format: copy and paste the content of
log_formatfile in this repository.
client_emailfield of GCP credential JSON file.
Secret key: a value from
private_keyfield of GCP credential JSON file.
Project ID: your project ID of GCP.
Dataset: a dataset name you created for Ingestly. (eg.
Table: a table name you created for Ingestly. (eg.
Template: this field can be empty but you can configure time-sliced tables if you enter like
CREATEto finish the setup process.
- Now you are ready to receive beacons. You can install Ingestly Client JS to your website.