DNAQuery is a command line utility to take LogDNA archives and load them into BigQuery. This allows long-term, queryable storage of logs in BigQuery (which is considerably more cost effective). In our use case, we want real-time access to many different logs and long-term storage of a subset of those logs. This approach has helped us find a balance between access and cost.
go get -u github.com/lifechurch/dnaquery
DNAQuery has been tested on Go v1.11
cp example.toml dnaquery.toml
All of these settings are currently required.
[storage] LogDirectory = 'logs' # temp local directory to download logs to and store results, this directory will be created if it doesn't exist
[[apps]] Name = "production-app" # name of app Regex = '^([\d.]+) - \[([^\]]*)\] - - \[([^\]]*)\] "([^"]*)" (\d+) (\d+) "([^"]*)" "([^"]*)" (\d+) ([\d.]+) ([\d.:]+) (\d+) ([\d.]+) (\d+)$' # regex used to pull parts of logs out, currently we don't use any named capture groups TimeGroup = 3 # the number of the capture group (1-based) in above regex that holds the time component of the log TimeFormat = "2/Jan/2006:15:04:05 -0700" # the format of the time field using https://golang.org/pkg/time/#Parse [[apps.excludes]] # an array of tables for exclusions Group = 4 # the number of the capture group in above regex to be used in the exclusion check Contains = "ping" # excludes logs if the value here is contained in the string in the above capture group
[gcp] # Google Cloud Project settings ProjectID = "gcpproj" # name of GCP project CredentialsFile = "gcp_credentials.json" # relative or absolute path to the credentials file downloaded from GCP Bucket = "logdna_to_bq" # name of bucket in Google Cloud Storage to save results for ingestion into BigQuery, bucket will need to be created before first run Dataset = "logdna" # BigQuery dataset TemplateTable = "logdna" # currently DNAQuery uses a template table. More details below. LogBucket = "logs" # name of bucket set in LogDNA archive setup above LogPrefix = "a7112abc9d" # each archive file starts with a prefix specific to your LogDNA account
Currently the schema for the BigQuery table is specified by creating a table with the schema that matches the regex. This is likely to change in the future as we may need a different schema for each app above.
dnaquery --date 2017-11-20
This project uses dep for dependency management
go get -u github.com/golang/dep/cmd/dep dep ensure -update
go test -v ./...
Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests.
- Chris Vaughn - Initial work - chrisvaughn
See also the list of contributors who participated in this project.
This project is licensed under the MIT License - see the LICENSE.md file for details