Repository navigation
Distributed Storage System
ArgyleDB utilizes Apache Accumulo as its core storage engine. Accumulo is a highly scalable structured store based on Google's Bigtable. Accumulo is written in Java and operates over the Hadoop Distributed File System (HDFS), which is part of the popular Apache Hadoop project. Accumulo supports efficient storage and retrieval of structured data, including queries for ranges, and provides support for using Accumulo tables as input and output for MapReduce jobs. Much of the work Accumulo does involves maintaining certain properties of the data, such as organization, availability, and integrity, across many commodity-class machines.
Accumulo features automatic load-balancing and partitioning, data compression and fine-grained security labels. An instance of Accumulo includes many TabletServers, one Garbage Collector process, one Master server and many Clients.
The TabletServer manages some subset of all the tablets (partitions of tables). This includes receiving writes from clients, persisting writes to a write-ahead log, sorting new key-value pairs in memory, periodically flushing sorted key-value pairs to new files in HDFS, and responding to reads from clients, forming a merge-sorted view of all keys and values from all the files it has created and the sorted in-memory store. Data are written to Accumulo by creating Mutation objects that represent all the changes to the columns of a single row. The changes are made atomically in the TabletServer.
TabletServers also perform recovery of a tablet that was previously on a server that failed, reapplying any writes found in the write-ahead log to the tablet.
Accumulo provides a richer data model than simple key-value stores, but is not a fully relational database. Data is represented as key-value pairs, where the key and value are comprised of the following elements:
Key |
Value |
||||
Row ID |
Column |
Timestamp |
|||
Family |
Qualifier |
Visibility |
|||
All elements of the Key and the Value are represented as byte arrays except for Timestamp, which is a Long. Accumulo sorts keys by element and lexicographically in ascending order. Timestamps are sorted in descending order so that later versions of the same Key appear first in a sequential scan. Tables consist of a set of sorted key-value pairs.
Accumulo is written in Java and provides a Java client library. Clients in other languages can communicate with Accumulo via the provided Thrift Proxy. All clients use three basic classes to communicate with Accumulo.
##BatchWriter
All new inserts, updates, and deletes are packaged up into Mutations and given to a BatchWriter. A Mutation is a set of changes to a single row. Each BatchWriter sends Mutations to a single table. The BatchWriter knows how the table is split into tablets and which servers the tablets are assigned to. Using this information, the BatchWriter efficiently groups Mutations into batches to increase write throughput. The BatchWriter is multi-threaded and the tradeoff between latency and throughput can be tuned.
Key-Value pairs are read out of a table using a Scanner. A Scanner may start at the beginning of a table or at a particular key, and may stop at the end of the table or a given key. After seeking to the initial key, Scanners proceed to read out key-value pairs sequentially in key-order until reaching the end of the table or the specified ending key. Scanners can be configured to read only certain columns. Additional configuration for a Scanner can be made to apply specific Iterators, or specific options to Iterators, to alter the set of key-value pairs returned from a particular Scanner.
When multiple ranges of keys are to be read from a table, a BatchScanner can be used to read the key-value pairs for the ranges using multiple threads. The ranges are grouped by Tablet Server to maximize the efficiency of communication between threads and Tablet Servers. This can be useful for applications whose design requires many individual scans to answer a single question. In particular, tables designed for working with time series, secondary indices, and complex text search can all benefit from using BatchScanners.
Accumulo processes each write to a set of log files. By default these are found under $ACCUMULO/logs/. In the event of TabletServer failure or error on shutting Accumulo down, some mutations may not have been minor compacted to HDFS properly. In this case, Accumulo will automatically reapply such mutations from the write-ahead log either when the tablets from the failed server are reassigned by the Master (in the case of a single TabletServer failure) or the next time Accumulo starts (in the event of failure during shutdown).
Recovery is performed by asking a tablet server to sort the logs so that tablets can easily find their missing updates. The sort status of each file is displayed on Accumulo monitor status page. Once the recovery is complete any tablets involved should return to an “online” state. Until then those tablets will be unavailable to clients.
The Accumulo client library is configured to retry failed mutations and in many cases clients will be able to continue processing after the recovery process without throwing an exception.
Obtaining the full benefits of Accumulo requires using novel schemas. The Dynamic Distributed Dimensional Data Model D4M provides a uniform mathematical framework based on associative arrays that encompasses both traditional (i.e., SQL) and non-traditional "NoSQL" databases. For non-traditional databases D4M naturally leads to a general purpose schema that can be used to fully index and rapidly query every unique string in a dataset.
D4M stores associations between multidimensional entities (tuples) in data structures called associative arrays. D4M associative arrays provide a one-to-one mapping onto the tables in a tuple store that makes complex manipulations simple to code. All rows and columns can be searched efficiently without the need to build specialized indexes.
D4M significantly improves search, retrieval, and analysis of massive amounts of data. The D4M Schema is simple, requires minimal parsing, and achieves the highest published Accumulo ingest rates.
Accumulo extends the BigTable data model to implement a security mechanism known as cell-level security. Every key-value pair has its own security label, stored under the column visibility element of the key, which is used to determine whether a given user meets the security requirements to read the value. This enables data of various security levels to be stored within the same row, and users of varying degrees of access to query the same table, while preserving data confidentiality.