Skip to content

[Logs] BigQuery info for logpush #1285

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jul 7, 2021
Merged

[Logs] BigQuery info for logpush #1285

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
821b2a8
adding draft bigquery content
crwaters16 May 6, 2021
e9adb7d
Merge branch 'production' into claire/enable-bigquery
crwaters16 May 20, 2021
5d87012
kody edits
crwaters16 May 20, 2021
a3073fc
Merge branch 'production' into claire/enable-bigquery
crwaters16 May 21, 2021
34060bd
Merge branch 'production' into claire/enable-bigquery
crwaters16 Jun 28, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions products/logs/src/content/get-started/enable-destinations/bigquery/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
order: 61
pcx-content-type: concept
---

# Enable BigQuery

Configure Logpush to send batches of Cloudflare logs to BigQuery.

BigQuery supports loading up to 1,500 jobs per table per day (including failures) with up to 10 million files in each load. That means you can load into BigQuery once per minute and include up to 10 million files in a load. See BigQuery's quotas for load jobs for more information.

Logpush delivers batches of logs as soon as possible, which means you could receive more than one batch of files per minute. Ensure your BigQuery job is configured to ingest files on a given time interval, like every minute, as opposed to when files are received. Ingesting files into BigQuery as each Logpush file is received could exhaust your BigQuery quota quickly.

For an example of how to set up a schedule job load with BigQuery, see the [Cloudflare + Google Cloud | Integrations repository](https://github.com/cloudflare/cloudflare-gcp/tree/master/logpush-to-bigquery).