Skip to content

Design

Jump to bottom
Vadim TSesko edited this page Feb 2, 2016 · 1 revision

Design

Using YoctoDB database consists of the following distinct phases:

  1. Add documents to a database in append-only mode
  2. Build and serialize database
  3. Open serialized database (or composite database) in read-only mode

Let's consider general YoctoDB notions.

Document

YoctoDB document is an opaque user provided sequence of bytes augmented with indexing fields used for filtering and/or sorting on the last phase.

We use Google Protocol Buffers to serialize business entities to sequence of bytes, but it is not obligatory.

Index field

Index field value is a named comparable byte sequence attached to the document before adding the document to a database being constructed.

There are several types of index fields:

  • FILTERABLE -- the field can be used to filter documents
  • SORTABLE -- the field can be used to sort documents
  • FULL -- both

Different filterable field values with the same name can be attached to the same document to allow filtering based on multivalued fields.

Database serialization

Database serialization involves building efficient read-only search structures based on all the accumulated information (primarily index field values).

Opening a serialized database in read-only mode costs nothing except possible disk cache warm up and checksum test if necessary.

Database search

YoctoDB query language supports the following features:

  • Filtering documents using several fields (eq, gt, gte, lt, lte, in, range queries)
  • Condition combining using logical operators (and, or and not)
  • Sorting documents using several fields (asc, desc)
  • Skip and limit
  • Count
  • Filtered sorted document payload processing using user provided callback

Query execution

YoctoDB query execution engine is based on BitSet manipulation. Current implementation uses thread local cached BitSet pools for each database.

Memory consumption can be estimated using the following formula:

thread_count * max(3, query_depth) * document_count / 64

There is no query optimizer so a query is executed as is.

Composite database

YoctoDB composite database provides the same database search features considered above for a set of basic databases.

Optimization hints

If you decide to build a partitioned database you should base your partitioning criteria on future request patterns so that query execution engine could skip entire partitions.

See Getting Started Guide to get code examples.

Clone this wiki locally