Skip to content
C++ framework for rapid server development
C++ CMake C Shell Yacc JavaScript Other
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.vscode MR3: Added some documentation Jun 15, 2019
abseil-cpp @ aa84489 Update abseil dependency to 2019 lts, allow subdir with mr shards Aug 21, 2019
base . Jul 5, 2019
cmake
doc Fix doc issues Aug 21, 2019
docker . Oct 6, 2018
examples . Aug 23, 2019
file Improve status check macro Aug 21, 2019
mr MR:. Aug 23, 2019
scripts . Apr 11, 2019
strings Add movies_join Jun 10, 2019
util . Aug 23, 2019
.clang-format . Apr 3, 2019
.clang_complete . Sep 4, 2018
.dockerignore Docker script for building the 'prod' app Sep 3, 2018
.gitignore Initial skeleton for c++ build system with refreshed libraries Dec 15, 2017
.gitmodules Update abseil url Apr 10, 2018
.travis.yml Fix build with doxygen (#11) Aug 2, 2019
CMakeLists.txt MR: propagate line number to mappers via raw context Aug 3, 2019
CPPLINT.cfg Support ECC Aug 18, 2018
Dockerfile Work on Rpc Server Aug 12, 2018
LICENSE . Oct 24, 2018
README.md Update README.md Aug 8, 2019
blaze.sh
install-dependencies.sh . Aug 23, 2019

README.md

Gaia - rapid backend development framework in C++

Build Status

Gaia is a set of libraries and c++ environment that allows you efficient and rapid development in c++14 on linux systems. The focus is mostly for backend development, data processing etc.

  1. Dependency on abseil-cpp
  2. Dependency on Boost 1.69
  3. Uses ninja-build on top of cmake
  4. Build artifacts are docker-friendly.
  5. Generic RPC implementation.
  6. HTTP server implementation.
  7. Many other features.

I will gradually add explanations for most crucial blocks in this library.

Building, Setting Up

  1. abseil is integrated as submodule. To fetch abseil run:

    git submodule update --init --recursive
    
  2. Dependencies setup:

    sudo ./install-dependencies.sh
    
  3. Build setup:

    ./blaze.sh -ninja -release
    

    third_party folder is checked out under build directories.

  4. To build, just run ninja:

    cd build-opt && ninja -j4 asio_fibers
    
  5. To check raw ASIO & fibers in action, run from 2 tabs:

    server> ./asio_fibers --logtostderr
    client> ./asio_fibers --connect=localhost --count 100000 --num_connections=8
    

To use abseil code use #include "absl/...". Third_party packages have TRDP:: prefix in CMakeLists.txt. absl libraries have prefix absl_....

Single node Mapreduce

GAIA library provides a very efficient multi-threaded mapreduce framework for batch processing. It supports out of the box json parsing, compressed formats (gzip, zstd), local disk and GCS (Google Cloud Storage) IO. Using GAIA MR it's possible to map, re-shard (partition), join and group multiple sources of data very efficiently. Fibers in GAIA allowed maximizing pipeline execution and balance IO with CPU workloads in parallel. The example below shows how to process text files and re-shard them based on an imaginary "year" column for each CSV row. Please check out this tutorial to learn more about GAIA MR.

#include "absl/strings/str_cat.h"
#include "mr/local_runner.h"
#include "mr/mr_main.h"
#include "strings/split.h"  // For SplitCSVLineWithDelimiter.

using namespace std;

DEFINE_string(dest_dir, "~/mr_output", "Working dir where the pipeline writes its by products");

int main() {
  // sets up IO threads and optional http console interace via port 8080 by default.
  PipelineMain pm(&argc, &argv);
  vector<string> inputs;
  for (int i = 1; i < argc; ++i) {
    inputs.push_back(argv[i]);  // could be a local file or "gs://...." url.
  }
  CHECK(!inputs.empty()) << "Must provide some inputs to run!";

  Pipeline* pipeline = pm.pipeline();

  // Assuming that the first line of each file is csv header.
  StringTable ss = pipeline->ReadText("read", inputs).set_skip_header(1);
  auto reshard = [](string str) {
    vector<char*> cols;
    SplitCSVLineWithDelimiter(&str.front(), ',', &cols);
    return absl::StrCat("year-", cols_[0]);
  };

  // Simplest example: read and repartition by year.
  ss.Write("write_input", pb::WireFormat::TXT)
      .WithCustomSharding(reshard).AndCompress(pb::Output::ZSTD);

  // Environment is abstracted away through mr3::Runner class. LocalRunner is an implementation
  // that comes out of the box.
  LocalRunner* runner = pm.StartLocalRunner(FLAGS_dest_dir);
  pipeline->Run(runner);

  LOG(INFO) << "Pipeline finished";

  return 0;
}

RPC

In addition to great performance, this RPC supports server streaming API, fully asynchronous processing, low-latency service. GAIA RPC framework employs Boost.ASIO and Boost.Fibers as its core libraries for asynchronous processing.

  1. IoContextPool is used for managing a thread-per-core asynchronous engine based on ASIO. For periodic tasks, look at asio/period_task.h.

  2. The listening server (AcceptServer) is protocol agnostic and serves both HTTP and RPC.

  3. RPC-service methods run inside a fiber. That fiber belongs to a thread that probably serves many other fiber-based connections in the server. Using regular locking mechanisms (std::mutex, pthread_mutex) or calling 3rd party libraries (libmysqlcpp) will block the whole thread and all its connections will be stalled. We need to be mindful of this, and as a policy prohibit thread blocking in fiber-based server code.

  4. Nevertheless, RPC service methods might need to issue RPC calls by themselves or block for some other reason. To do it correctly, we must use fiber-friendly synchronization routines. But even in this case, we will still block the calling fiber (not thread). All other connections will continue processing but this one will stall. By default, there is one dedicated fiber per RPC connection that reads rpc requests and delegates them to the RPC application code. We need to remember that if higher level server-code stalls its fiber during its request processing, it effectively limits total QPS per that socket connection. For spinlock use-cases (i.e. RAM access locking with rw-spinlocks with low contention) having single fiber per rpc-connection is usually good enough to sustain high throughput. For more complicated cases, it's advised to implement fiber-pool (currently not exposed in GAIA).

  5. Server-side streaming is needed for responses that can be very large. Such responses can easily be represented by a stream of smaller responses with an identical schema. Think of SQL response for example. It may consist of many rows returned by SELECT. Instead, of returning all of them as one blob, server-side streaming can send back multiple responses in the context of a single request on a wire. Each small response is propagated to RPC client via a callback based interface. As a result, both systems (client and server) are not required to hold the whole response in RAM at the same time.

While GAIA provides very efficient RPC core library, it does not provide higher level RPC bindings. It's possible though to build a layer that uses protobuf-based declaration language this RPC library. For raw RPC demo see asio_fibers above.

HTTP

HTTP handler is implemented using Boost.Beast library. It's integrated with the IoContextPool similarly to RPC service. Please see http_main.cc, for example. HTTP also provides support for backend monitoring (Varz status page) and for extensible debugging interface. With monitoring C++ backend returns json object that is formatted inside status page in the browser. To check how it looks, please go to localhost:8080 while asio_fibers are running.

Self-profiling

Every http-powered backend has integrated CPU profiling capabilities using gperf-tools and pprof Profiling can be trigerred in prod using magic-url commands. Enabled profiling usually has very minimal impact on cpu performance of the running backend.

Logging

Logging is based on Google's glog library. The library is very reliable, performant and solid. It has many features that allow resilient backend development. Unfortunately, Google's version has some bugs, which I fixed (waiting for review...), so I use my own fork. Glog library gives me the ability to control logging levels of a backend at run-time without restarting it.

Tests

GAIA uses googletest+gmock unit-test environment.

You can’t perform that action at this time.