Skip to content
master
Go to file
Code

Latest commit

… metrics (#289)

* adding metrics to the hash module

* Squashed 'deps/ccommon/' changes from f5efe29..5e64de2

5e64de2 WIP: add fluxcapacitor support for linux build (#242)
55769cc restore cmake for rust libraries (#241)
6b1d8d5 Improve the Rust build story (#240)
5955d6b move nodelay setting to the server socket, and accepted connections will inherit that (#238)
aafd20f formatting (#237)
3b5c069 Update ccommon_rs to use bytes v5.0 (#235)
df74087 Address clippy lints in rust code (#234)
7c5bbd1 Make metrics and options Send + Sync (#233)
c2e617e Ensure that manually implemented C options use the correct name (#232)
6e76741 Add some docs
b5b4c5a Change check_pipe to use nanosleep instead of usleep (#231)
eb0a389 Use name of field instead of description for C metrics (#230)
16ddc76 Fix broken buf impls (#229)
eefcdcb Avoid redundant rebuilds of rust packages in CI (#228)
33f62a8 Update bindgen to also generate bindings recursively (#227)
27ffc7c Implement bytes::Buf and Bytes::ButMut on Buf and OwnedBuf (#226)
f873930 Various small bugfixes and usability improvements for rust code (#225)
37a1ecd Port option parsing module to Rust (#224)
38f7556 Fix failed test detection (#215)
0ab1604 Conditionally use std::any::type_name if it's supported (#223)
98176d3 Backport changes from #265 (#222)
ba54096 Remove test for removed rust logging functionality (#221)
519118d Rewrite cmake cargo build wrapper (#220)
5d23b3a Fix some small typos found in #263 (#219)
475dda7 Clean up logging shim (#218)
1d28dd2 Expand rust bindings to add options, metrics, and ccbox (#217)
88b5400 Upstream changes from #261 (#216)
4e99e63 fix a bug and change how check is found (#214)
54067ef slightly simplify accept error-handling logic (#210)
e9fe980 Fix synchronize ccommon with pelikan deps/ccommon (#212)
7eb6424 Cleanup libcheck related code (#211)
683bc1a cc_bstring simplify and fix (#207)
8737d99 continue on server socket on non-blocking errors (#209)
2a62281 add atoi64 to bstring (#206)
f71c657 cc_option simplify _allowed_in_name (#205)
24e3131 Add ITT instrumentation option (#204)
236c98d Fix docs (#200)
e58f6a8 cc_array and cc_ring_array NULL fixes (#201)
1c8df42 Add basic support of build type (#199)
7107988 Fix now_ns() (#198)
da240e5 cc: extend cc_util module (#196)
4846b15 Fix TAILQ_REINIT (#195)
4f5dbb0 Update Cmake version to 2.8 (#197)
2e6f78a cc_mm use OS_DARWIN macro to detect OS (#194)
57acaf6 cc: extend queue module (#193)
a64ada2 cc: extend duration module (#192)
b117632 reverting CMake file changes (#191)
dea5bee backport changes made to ccommon in pelikan (#190)
a4c0334 add linebreak to stats_log() (#188)
05eb03e fix inconsistent naming and bump version (#187)
4acc53a Stats to file (#186)
2168fec minimize osx build config (#185)
42b24de Simplify rust options, specify fewer output targets (#183)
c9fa905 update CMakeRust used to latest version, tweaks to make build work (#184)
2ef0163 Reorder dependency includes in cmake, don't parallel build (#182)
a6a54d9 remove endian-specific logic from str*cmp (#177)
4c0668b epoll_create* ignores size hint in newer kernels, switch to new API (#179)
c9c5ee5 improve cc_bstring string literal and cstring names (#176)
0184d73 Add unit tests for buffer, fix buf/dbuf bugs and refactor (#174)
d7dab43 create a .cargo/config so intellij uses the same target dir as cmake (#173)
e710712 use accept4 for tcp_accept when available (#171)
21ba10e Remove cargo lock for shared lib, closes #169 (#172)
24660f1 update style guide (#170)
17baf1e Per thread logging (#168)

git-subtree-dir: deps/ccommon
git-subtree-split: 5e64de297dbf1d49b59746431448f5f4dfb5d640

* clean up cmake file

* rename findcheck cmake module

* adding new metrics to rust wrapper

* add ci/cargo.sh to pelikan for consistent invocation

* Squashed 'deps/ccommon/' changes from 5e64de2..0db032a

0db032a tuning cmake / ci further (#243)
49dcaff improve cargo build script (#244)

git-subtree-dir: deps/ccommon
git-subtree-split: 0db032afcac0a75887281c3876ed9f7bb10391ce

* update ci based on changes first tested in ccommon
f55e736

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
ci
 
 
 
 
 
 
 
 
 
 
 
 
 
 
src
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

Pelikan is Twitter's unified cache server.

Content

Build Status

Overview

After years of using and working on various cache services, we built a common framework that reveals the inherent architectural similarity among them.

By creating well-defined modules, most of the low-level functionalities are reused as we create different binaries. The implementation learns from our operational experiences to improve performance and reliability, and leads to software designed for large-scale deployment.

The framework approach allows us to develop new features and protocols quickly.

Products

Currently Pelikan yields three main products, all of which are backends/servers.

  • pelikan_twemcache: a Twemcache replacement
  • pelikan_slimcache: a Memcached-like server with ultra-low memory overhead- compared to Memcached/Redis, the per-key overhead is reduced by up to 90%
  • pelikan_pingserver: an over-engineered, production-ready ping server useful as a tutorial and for measuring baseline RPC performance

Features

  • runtime separation of control and data plane
  • predictably low latencies via lockless data structures, worker never blocks
  • per-module config options and metrics that can be composed easily
  • multiple storage and protocol implementations, easy to further extend
  • low-overhead command logger for hotkey and other important data analysis

Building Pelikan

Requirement

  • platform: Mac OS X or Linux
  • build tools: cmake (>=2.8)
  • compiler: gcc (>=4.8) or clang (>=3.1)
  • (optional) unit testing framework: check (>=0.10.0). See below.

Build

git clone https://github.com/twitter/pelikan.git
mkdir _build && cd _build
cmake ..
make -j

The executables can be found under _bin/ (under build directory)

To run all the tests, including those on ccommon, run:

make test

To skip building tests, replace the cmake step with the following:

cmake -DCHECK_WORKING=off ..

Install check

To compile and run tests, you will have to install check. Please follow instructions in the project.

Note: we highly recommend installing the latest version of check from source, as there are, unfortunately, a linker bug in packages installed by the current versions of brew (OS X), CentOS and Ubuntu LTS. The bug does not affect building executables.

Usage

Using pelikan_twemcache as an example, other executables are highly similar.

To get info of the service, including usage format and options, run:

_bin/pelikan_twemcache -h

To launch the service with default settings, simply run:

_bin/pelikan_twemcache

To launch the service with the sample config file, run:

_bin/pelikan_twemcache config/twemcache.conf

You should be able to try out the server using an existing memcached client, or simply with telnet.

$ telnet localhost 12321
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
set foo 0 0 3
bar
STORED

Attention: use admin port for all non-data commands.

$ telnet localhost 9999
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
version
VERSION 0.1.0
stats
STAT pid 54937
STAT time 1459634909
STAT uptime 22
STAT version 100
STAT ru_stime 0.019172
...

Configuration

Pelikan is file-first when it comes to configurations, and currently is config-file only. You can create a new config file following the examples included under the config directory.

Tip: to get a list of config options for each executable, use -c option:

_bin/pelikan_twemcache -c

Community

Stay in touch

Contributing

Please take a look at our community manifesto and coding style guide.

To get a sense of where things are going next, please visit our Roadmap wiki.

If you want to submit a patch, please follow these steps:

  1. create a new issue
  2. fork on github & clone your fork
  3. create a feature branch on your fork
  4. push your feature branch
  5. create a pull request linked to the issue

Documentation

We have made progress and are actively working on documentation, and will put it on our website. Meanwhile, check out the current material under docs/

We also maintain a list of topics for upcoming blog posts. Please let us know what (else) interests you.

License

This software is licensed under the Apache 2.0 license, see LICENSE for details.

You can’t perform that action at this time.