Automatic Anomaly Decetor
-
Updated
Jan 11, 2022 - HCL
Google BigQuery enables companies to handle large amounts of data without having to manage infrastructure. Google’s documentation describes it as a « serverless architecture (that) lets you use SQL queries to answer your organization's biggest questions with zero infrastructure management. BigQuery's scalable, distributed analysis engine lets you query terabytes in seconds and petabytes in minutes. » Its client libraries allow the use of widely known languages such as Python, Java, JavaScript, and Go. Federated queries are also supported, making it flexible to read data from external sources.
📖 A highly rated canonical book on it is « Google BigQuery: The Definitive Guide », a comprehensive reference.
Another enriching read on the subject is the inside story told in the article by the founding product manager of BigQuery celebrating its 10th anniversary.
Automatic Anomaly Decetor
A IaC script to ingest and process messages containing data of trips taken by vehicles.
Terraform module for BigQuery sink connector on Aiven KafkaConnect cluster
Simple HTTP endpoint for telemetry data type events in GCP.
Here we are have automated the entire process of Creation of Google Cloud Storage buckets to store that that we are transferring via various transfer services, as of now Transfer from other Google cloud Storage buckets and public http/https links are supported, finally all this data is being processed into BigQuery for observations and Analysis.
Creating datasets and tables in Google BigQuery via Terraform
This module allows you to execute sql queries in big queries, simply by specifying the list of sql files (in the source bucket) to be executed, the result of which is then stored in the result bucket.
Cloud function subscribes itself to given topic and inserts each message to BigQuery table.
Data sync via CDC from GCP Cloud SQL to Big Query using Datastream
Use Bigquery Machine Learning (BQML) to apply advertising techniques
Terraform module for managing Google BigQuery datasets
Simple data quality on top of BigQuery by using just scheduled queries and terraform
Terraform module of Google BigQuery S3 Data Transfer Config
Final project for DataTalks.Club Data Engineering bootcamp
...an automated data pipeline that retrieves cryptocurrency data from the CoinCap API, processes and transforms it for analysis, and presents key metrics on a near-real-time dashboard
Released May 19, 2010