[BETA] The plugin-driven CLI for backing up & restoring data.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci ci: Build on go 1.10 Apr 16, 2018
agent
cmd notification: Add Body Dec 15, 2018
context context: Fix CreateTempFile Apr 20, 2018
log Initial commit Feb 14, 2018
pkg Initial commit Feb 14, 2018
plugin notification/slack: Include notification body Dec 15, 2018
.gitignore Initial commit Feb 14, 2018
.goreleaser.yml
Dockerfile docker: Remove update-ca-certificates Jun 21, 2018
Dockerfile.consul docker: Add kargo-consul Jun 21, 2018
Dockerfile.foundationdb docker: Build FoundationDB 6.0.15 Dec 12, 2018
Dockerfile.influxdb docker: Add kargo-influxdb Jun 21, 2018
Gopkg.lock Update dependencies Apr 20, 2018
Gopkg.toml Initial commit Feb 14, 2018
LICENSE Initial commit Feb 14, 2018
Makefile Initial commit Feb 14, 2018
README.md readme: Add FoundationDB Jun 21, 2018
main.go Add version command Feb 20, 2018

README.md

🚢 Kargo - Backup & Restore CircleCI Go Report Card FOSSA Status

Kargo is a CLI written in Go for transporting everything everywhere, which makes it the ideal companion to backup & restore data.

The design is unapologetically taken from Telegraf ❤️ with its plugin system. That allows developers to create adapters for any kind of databases, storages, or notifiers.

For example, you might want to backup a Couchbase database, compress and encrypt the backup, and store it on Amazon S3. You also may want to send a Pagerduty alert when a backup failed or timed out. Kargo can be seen as a database-agnostic backup tool.

Kargo is plugin-driven and has the concept of 4 distinct plugins:

1. Source Plugins backup and restore a source, such as a database, a folder, or anything else.
2. Processor Plugins encode/decode backup data (compress, cipher, ...).
3. Store Plugins persist backups and can be queried.
4. Notification Plugins send notifications on success/failure.

New plugins are designed to be easy to contribute, we'll eagerly accept pull requests and will manage the set of plugins that Kargo supports.

Installation

You can download the binaries from the release section.

From Homebrew (macOS):

  1. Run brew tap stairlin/homebrew-tap
  2. Run brew install stairlin/tap/kargo

From Source:

Kargo is currently tested on go 1.9.

  1. Install Go
  2. Setup your GOPATH
  3. Run go get -d github.com/stairlin/kargo
  4. Run cd $GOPATH/src/github.com/stairlin/kargo
  5. Run go get -d ./...
  6. Run go run main.go

Usage

Backup & restore data:

kargo backup
kargo restore my_backup_key

List backups:

kargo list
kargo list --from 2018-02-14
kargo list --prefix foo --to 2017-12-31
kargo list --pattern ^[a-z]{3}.*
kargo list --limit 50

Backup with a custom key name

kargo backup --key dat_snapshot

Pull a backup to local disk & restore data:

kargo pull my_backup_key
kargo restore --local my_backup_key

Help:

kargo help

Configuration

Environment variables

Environment variables can be used anywhere in the config file, simply prepend them with $. The variable must be within quotes (ie, "$STR_VAR"). Numbers, booleans, and other data types are not supported.

Configuration file locations

The location of the configuration file can be set via the --config command line flag.

Order on which Kargo looks for a configuration file:

1. --config flag
1. kargo.toml in the working directory
2. $KARGO_CONFIG environment variable
4. default location `/etc/kargo/kargo.toml`

Show the current configuration:

kargo config

Example of a configuration file.

[agent]
  debug = true

[source.dir]
  path = "/oh/my/dir/"

[processors.gzip]

[processors.cipher]
  default = 0
  keys = ["my_base64_key", "another_key"]

[storage.s3]
  id = "foo"
  secret = "bar"
  token = ""
  bucket = "my_bucket"
  region = "eu-central-1"

[[notifiers.pagerduty]]
  key = "api_key"

[[notifiers.slack]]
  username = "My Project"
  url = "https://hooks.slack.com/services/foo"

Plugins

Sources

  1. Consul
  2. Couchbase
  3. Directory
  4. FoundationDB
  5. InfluxDB
  6. PostgreSQL

Storages

  1. Filesystem
  2. Amazon S3

Processors

  1. Cipher
  2. GZip

Notifiers

  1. Pagerduty
  2. Slack

Dependencies

Most dependencies are packed into the binary, such as AWS S3, gzip, Pagerduty, etc. However, Source plugins mainly rely on shell commands to work, so these dependencies must be installed separately and set to the $PATH.

Internal design

Kargo piggyback on the powerful Go I/O library to keep the memory and disk footprint minimal. In most cases, data is being streamed from the source to the storage with no or minimal internal buffering. There are plugins, such as cipher that must work data by chunks for obvious reasons, so it will use a small buffer.

It is worth noting that most Source plugins cannot stream data right away. Indeed, they have to create a temporary file that contains the backup before. postgres is currently the only plugin that can stream data directly, thanks to pg_dump.

Contributing

This project is still unstable and not production ready. However, new plugins and bug fixes are welcome.

There is currently no guidelines, no red tape to contribute. Enjoy while it last. :)

License

FOSSA Status