Skip to content
A tool used to collect and merge tidb's binlog for real-time data backup and synchronization.
Go Shell Makefile
Branch: master
Clone or download
lichunzhu fix bug when some error occurs in loader arbiter's syncBinlog may blo…
…ck (#714)

* fix bug when some error occurs syncBinlog may block

* add test for arbiter server.Run to test whether arbiter will get blocked
Latest commit a4fa9d2 Aug 20, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github add issue templates for bugs, features, questions (#700) Aug 6, 2019
arbiter fix bug when some error occurs in loader arbiter's syncBinlog may blo… Aug 20, 2019
binlogctl binlogctl: Use registry.Node to get a single node directly (#695) Jul 26, 2019
cmd add node-id cmd to specify drainer's node-id (#684) Jul 23, 2019
docs remove outdate docs/ and tools/, duplicate /reparo.toml (#482) Mar 11, 2019
drainer drainer: Remove unused waitGroup (#710) Aug 13, 2019
hack migrate to Go 1.11 module instead of glide (#367) Oct 17, 2018
pkg loader: stop retry execute sqls when closed (#691) Aug 1, 2019
proto Add Apache License v2.0 headers to Go sources (#555) Apr 23, 2019
pump Avoid repeated called of NewIterator which might be slow (#718) Aug 20, 2019
reparo Return error when loading some undefined configs from local (#687) Jul 24, 2019
tests Use mustExec to simplify SQL execution in test cases (#704) Aug 9, 2019
tools/check *: Simplify checkpoint (#615) May 31, 2019
.gitignore *: Fix some missing error checks as reported by golangci-lint (#600) May 18, 2019
CHANGELOG.md update changelog.md and tools dep to 2.1.5 (#477) Feb 28, 2019
CONTRIBUTING.md Update README and Add CONTRIBUTING (#566) Apr 26, 2019
LICENSE Initial commit Sep 5, 2016
Makefile *: add binlogctl (#567) Apr 28, 2019
README.md doc: Add more badges on README (#616) May 23, 2019
generate-binlog.sh drainer: refactor executor & add self-description pb output (#121) Feb 28, 2017
gitcookie.sh pump: implement gRPC server and make code more solid. (#9) Sep 26, 2016
go.mod update sarama and mysql driver (#689) Jul 25, 2019
go.sum update sarama and mysql driver (#689) Jul 25, 2019

README.md

TiDB-Binlog

Build Status Coverage Status Go Report Card

TiDB-Binlog introduction

TiDB-Binlog is a tool used to collect TiDB's binary logs with the following features:

  • Data replication

    Synchronize data from the TiDB cluster to heterogeneous databases.

  • Real-time backup and recovery

    Backup the TiDB cluster into the Dump file and it can be used for recovery.

  • Multiple output format

    Support MySQL, Dump file, etc.

  • History replay

    Replay from any history point.

Documentation

Architecture

architecture

Service list

Pump

Pump is a daemon that receives real-time binlogs from tidb-server and writes in sequential disk files synchronously.

Drainer

Drainer collects binlogs from each Pump in the cluster, transforms binlogs to various dialects of SQL, and applies to the downstream database or filesystem.

How to build

To check the code style and build binaries, you can simply run:

make build   # build all components

If you only want to build binaries, you can run:

make pump  # build pump

make drainer  # build drainer

When TiDB-Binlog is built successfully, you can find the binary in the bin directory.

Run Test

Run all tests, including unit test and integration test

make test

See tests for how to execute and add integration tests.

Deployment

The recommended startup sequence: PD -> TiKV -> Pump -> TiDB -> Drainer

The best way to install TiDB-Binlog is via TiDB-Binlog-Ansible

Tutorial

Here's a tutorial to experiment with TiDB-Binlog (not for production use).

Config File

Contributing

Contributions are welcomed and greatly appreciated. See CONTRIBUTING.md for details on submitting patches and the contribution workflow.

License

TiDB-Binlog is under the Apache 2.0 license. See the LICENSE file for details.

You can’t perform that action at this time.