Skip to content
Service that mirrors documents in RethinkDB to Elasticsearch
Crystal Dockerfile
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
spec
src
.gitignore
.travis.yml
Dockerfile
Dockerfile.alpine
LICENSE
README.md
shard.lock
shard.yml

README.md

rubber-soul

A small (one might even say 'micro') service that hooks into rethinkdb-orm models and generates elasticsearch indicies.
rubber-soul exposes a REST API to reindex/backfill specific models.

TODO

  • refactor to use the bulk api

Usage

  • Set the tables to be mirrored in ES through setting RubberSoul::MANAGED_TABLES with an array of (T < RethinkORM::Base).class
  • Configure Elastic client through ES_HOST and ES_PORT env vars, or through switches on the command line
  • Configure RethinkDB connection RETHINKDB_HOST and RETHINKDB_PORT env vars

POST /api/v1/reindex[?backfill=true]

Deletes indexes and recreates index mappings. Backfills the indices by default (toggle with backfill boolean).

POST /api/v1/backfill

Backfills all indexes with data from RethinkDB.

GET /api/v1/healthz

Healthcheck.

Index Schema

  • Each RethinkDB table receives an ES index, with a mapping generated from the attributes of a RethinkORM model.
  • RethinkORM attributes can accept a tag es_type to specify the correct field datatype for the index schema.
  • belongs_to associations are modelled with ES join datatypes, associated documents are replicated in their parent's index. This is necessary for has_parent and has_child queries.

RethinkDB Mirroring

RubberSoul::TableManager hooks into the changefeed of a table, resolves associations of the model and creates/updates documents in the appropriate ES indices.

Development

Tested against...

  • rethinkdb 2.3.6
  • elasticsearch 7.0.0

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create a new Pull Request

Contributors

You can’t perform that action at this time.