Skip to content

segmentio/ctlstore

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

Not sure if this will work with the segmentio team since that seems private.

Testing not required because this is a change to the GitHub settings.
42c823c

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
February 27, 2023 12:28
July 16, 2019 11:55
July 16, 2019 11:55
July 16, 2019 11:55
July 16, 2019 11:55
November 18, 2021 17:20
July 16, 2019 11:55
July 16, 2019 11:55
July 16, 2019 11:55

ctlstore

ctlstore is a distributed data store that provides very low latency, always-available, "infinitely" scalable reads. The underlying mechanism for this is a SQLite database that runs on every host called the LDB. A daemon called the Reflector plays logged writes from a central database into the LDB. As this involves replicating the full data store on every host, it is only practical for situations where the write rate (<100/s total) and data volumes (<10GB total) are low.

Recommended reading:

Security

Note that because ctlstore replicates the central database to an LDB on each host with a reflector, that LDB contains all of the control data. In its current state that means that any application which has access to the LDB can access all of the data within it.

The implications of this are that you should not store data in ctlstore that should only be accessed by a subset of the applications that can read the LDB. Things like secrets, passwords, and so on, are an example of this.

The ctlstore system is meant to store non-sensitive configuration data.

Development

A MySQL database is needed to run the tests, which can be started using Docker Compose:

$ docker-compose up -d

Run the tests using make:

$ make test
# For more verbosity (`Q=` trick applies to all targets)
$ make test Q=

A single ctlstore binary is used for all functionality. Build it with make:

$ make build

Sync non-stdlib dependencies and pull them into ./vendor

$ make deps

Ctlstore uses Go modules. To build a docker image, the dependencies must be vendored first:

$ make vendor

Many of ctlstore's unit tests use mocks. To regenerate the mocks using counterfeiter:

$ make generate

Tying the Pieces Together

This project includes a docker-compose file docker-compose-example.yml. This initializes and runs

  • mysql (ctlstore SoR)
  • executive service (guards the ctlstore SoR)
  • reflector (builds the LDB)
  • heartbeat (mutates a ctlstore table periodically)
  • sidecar (provides HTTP API access to ctlstore reader API)
  • supervisor (periodically snapshots LDB)

To start it, run:

$ make deps
$ make vendor
$ docker-compose -f docker-compose-example.yml up -d