Stream NSQ messages to Google's BigQuery using the streaming API
Go Makefile
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.gitignore
.travis.yml
Godeps
LICENSE
Makefile
README.md
nsq_to_bigquery.go

README.md

nsq to bigquery logo

Build Status

nsq-to-bigquery

Stream NSQ messages to Google's BigQuery using the streaming API.

Written by Eran Sandler (@erans)

This is a very early version that does work but was not (yet) tested in real hard production environment.

Assumptions / Limitations:

  • Message body is a valid JSON that is a flat dictionary (only key and simple values).
  • Each message must contain data that will constitute a single row that will go into BigQuery.
  • BigQuery table MUST exists and the schema MUST match the JSON being sent.
  • This version sends each message in its own request and does not use any batch API (yet).