a command line utility to take logdna exports and load them into BigQuery
Switch branches/tags
Nothing to show
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
test_data
.gitignore
.goreleaser.yml
.travis.yml switch to go modules Nov 6, 2018
CHANGELOG.md
CONTRIBUTING.md
LICENSE.md
README.md
config.go
config_test.go
example.toml
go.mod switch to go modules Nov 6, 2018
go.sum
main.go still working towards build 0.3.1 Apr 23, 2018
main_test.go
utils.go

README.md

DNAQuery

Go Report Card Build Status Coverage Status

DNAQuery is a command line utility to take LogDNA archives and load them into BigQuery. This allows long-term, queryable storage of logs in BigQuery (which is considerably more cost effective). In our use case, we want real-time access to many different logs and long-term storage of a subset of those logs. This approach has helped us find a balance between access and cost.

Getting Started

go get -u github.com/lifechurch/dnaquery

DNAQuery has been tested on Go v1.11

Prerequisites

Configuring

cp example.toml dnaquery.toml

Edit dnaquery.toml

All of these settings are currently required.

Storage

[storage]
LogDirectory = 'logs' # temp local directory to download logs to and store results, this directory will be created if it doesn't exist

Apps

[[apps]]
    Name = "production-app"  # name of app
    Regex = '^([\d.]+) - \[([^\]]*)\] - - \[([^\]]*)\] "([^"]*)" (\d+) (\d+) "([^"]*)" "([^"]*)" (\d+) ([\d.]+)  ([\d.:]+) (\d+) ([\d.]+) (\d+)$'  # regex used to pull parts of logs out, currently we don't use any named capture groups
    TimeGroup = 3 # the number of the capture group (1-based) in above regex that holds the time component of the log
    TimeFormat = "2/Jan/2006:15:04:05 -0700" # the format of the time field using https://golang.org/pkg/time/#Parse
    [[apps.excludes]]  # an array of tables for exclusions
        Group = 4  # the number of the capture group in above regex to be used in the exclusion check
        Contains = "ping" # excludes logs if the value here is contained in the string in the above capture group

GCP

[gcp]  # Google Cloud Project settings
ProjectID = "gcpproj" # name of GCP project
CredentialsFile = "gcp_credentials.json" # relative or absolute path to the credentials file downloaded from GCP
Bucket = "logdna_to_bq" # name of bucket in Google Cloud Storage to save results for ingestion into BigQuery, bucket will need to be created before first run
Dataset = "logdna" # BigQuery dataset
TemplateTable = "logdna" # currently DNAQuery uses a template table. More details below.
LogBucket = "logs" # name of bucket set in LogDNA archive setup above
LogPrefix = "a7112abc9d" # each archive file starts with a prefix specific to your LogDNA account

TemplateTable

Currently the schema for the BigQuery table is specified by creating a table with the schema that matches the regex. This is likely to change in the future as we may need a different schema for each app above.

Usage

dnaquery --date 2017-11-20

Dependencies

This project uses dep for dependency management

go get -u github.com/golang/dep/cmd/dep
dep ensure -update

testing

go test -v ./...

Changelog

CHANGELOG.md

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests.

Versioning

We use SemVer for versioning. For the versions available, see the tags on this repository.

Authors

See also the list of contributors who participated in this project.

License

This project is licensed under the MIT License - see the LICENSE.md file for details