A key/value store for serving static batch data
Go Makefile Shell
Switch branches/tags
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
backend ensure that target version has success file too Feb 24, 2018
blocks Allow ingestion of partitioned Sparkey files Oct 20, 2017
doc Use a feature flag to control fetching versions (#159) Feb 28, 2018
sharding add max_replication to config to allow for a partition replication limit Nov 16, 2017
test_databases Allow ingestion of partitioned Sparkey files Oct 20, 2017
vendor Open downloaded files for write with `O_DIRECT`. May 23, 2018
workqueue Parallelize by individual file, rather than dataset, when downloading. Sep 13, 2017
zk Serialize cleaning up znodes for old db versions and avoid a race con… Mar 2, 2018
.gitignore Ignore intellij projects and sequins configs. Aug 16, 2017
.travis.yml Add new S3 credentials Feb 16, 2018
Dockerfile bump golang version to 1.8.3 Aug 4, 2017
LICENSE.txt initial open-source release Nov 20, 2014
Makefile Run tests verbosely Feb 16, 2018
README.md Add chat to the landing page and readme Oct 26, 2016
align.go Make write buffer size configurable. May 23, 2018
align_test.go Make write buffer size configurable. May 23, 2018
build.go Make write buffer size configurable. May 23, 2018
cluster_test.go Use a feature flag to control fetching versions (#159) Feb 28, 2018
config.go Make write buffer size configurable. May 23, 2018
config_test.go Add a bunch of path validations into validateConfig Aug 17, 2016
db.go Use a feature flag to control fetching versions (#159) Feb 28, 2018
debug.go Add support to log requests to stdout instead of a different file. Apr 19, 2018
debug_test.go fix requestlog test Apr 19, 2018
hdfs_sequins_test.go refactor test source directories Oct 3, 2017
healthcheck_test.go include more info in healthcheck, return 200 if any versions available Oct 9, 2017
jenkins_build.sh Run `go vet` as part of CI, and fix all violations. Aug 3, 2017
main.go Add more zk logging (#162) Mar 1, 2018
proxy.go Add more zk logging (#162) Mar 1, 2018
proxy_test.go Increase sleep times in proxy tests Nov 15, 2016
ratelimit_test.go Ratelimit file downloads (#154) Feb 17, 2018
s3_sequins_test.go allow s3_backend to handle _SYMLINK Nov 28, 2017
sequins.conf.example Add request logging via feature flag Mar 7, 2018
sequins.go Merge pull request #165 from stripe/vasi-goforit-request-log Mar 8, 2018
sequins_test.go Update goforit in sequins Mar 7, 2018
serve.go Move error handling up in the stack Oct 13, 2017
status.go update replication status for versionStatus struct on initialization Nov 14, 2017
status.tmpl increase css width some more Nov 14, 2017
utils.go Filter out DBs and versions with invalid characters Jun 28, 2017
version.go Only close/delete a version once (#152) Feb 15, 2018
version_mux.go Add some comments Nov 2, 2016

README.md

Sequins is a key/value database for serving static batch data.

It's well-suited to making data generated by Hadoop available to processes that need low-latency, reliable access.

It's designed to be:

  • Horizontally scalable: replicate a hot dataset over a few nodes, or partition a large one over many.
  • Reliable: serve your data without an online dependency on Hadoop or HDFS. Sequins is built to be resilient to multi-node failures.
  • Interoperable: load data from HDFS or S3 in Hadoop's SequenceFile format. Tools like Spark or Impala can also be used to generate data.
  • Accessible: fetch values with HTTP GET; no client library required.

See the manual to get started.

build Gitter