Repository navigation
Distributed Storage System
ArgyleDB utilizes Apache Accumulo as its core storage engine. Accumulo is a highly scalable structured store based on Google's Bigtable. Accumulo is written in Java and operates over the Hadoop Distributed File System (HDFS), which is part of the popular Apache Hadoop project. Accumulo supports efficient storage and retrieval of structured data, including queries for ranges, and provides support for using Accumulo tables as input and output for MapReduce jobs. Much of the work Accumulo does involves maintaining certain properties of the data, such as organization, availability, and integrity, across many commodity-class machines.
Accumulo features automatic load-balancing and partitioning, data compression and fine-grained security labels. An instance of Accumulo includes many TabletServers, one Garbage Collector process, one Master server and many Clients.
The TabletServer manages some subset of all the tablets (partitions of tables). This includes receiving writes from clients, persisting writes to a write-ahead log, sorting new key-value pairs in memory, periodically flushing sorted key-value pairs to new files in HDFS, and responding to reads from clients, forming a merge-sorted view of all keys and values from all the files it has created and the sorted in-memory store. Data are written to Accumulo by creating Mutation objects that represent all the changes to the columns of a single row. The changes are made atomically in the TabletServer.
TabletServers also perform recovery of a tablet that was previously on a server that failed, reapplying any writes found in the write-ahead log to the tablet.
Accumulo provides a richer data model than simple key-value stores, but is not a fully relational database. Data is represented as key-value pairs, where the key and value are comprised of the following elements:
Key |
Value |
||||
Row ID |
Column |
Timestamp |
|||
Family |
Qualifier |
Visibility |
|||
All elements of the Key and the Value are represented as byte arrays except for Timestamp, which is a Long. Accumulo sorts keys by element and lexicographically in ascending order. Timestamps are sorted in descending order so that later versions of the same Key appear first in a sequential scan. Tables consist of a set of sorted key-value pairs.
Accumulo processes each write to a set of log files. By default these are found under $ACCUMULO/logs/. In the event of TabletServer failure or error on shutting Accumulo down, some mutations may not have been minor compacted to HDFS properly. In this case, Accumulo will automatically reapply such mutations from the write-ahead log either when the tablets from the failed server are reassigned by the Master (in the case of a single TabletServer failure) or the next time Accumulo starts (in the event of failure during shutdown).
Recovery is performed by asking a tablet server to sort the logs so that tablets can easily find their missing updates. The sort status of each file is displayed on Accumulo monitor status page. Once the recovery is complete any tablets involved should return to an “online” state. Until then those tablets will be unavailable to clients.
The Accumulo client library is configured to retry failed mutations and in many cases clients will be able to continue processing after the recovery process without throwing an exception.
Obtaining the full benefits of Accumulo requires using novel schemas. The Dynamic Distributed Dimensional Data Model D4M provides a uniform mathematical framework based on associative arrays that encompasses both traditional (i.e., SQL) and non-traditional "NoSQL" databases. For non-traditional databases D4M naturally leads to a general purpose schema that can be used to fully index and rapidly query every unique string in a dataset.
D4M stores associations between multidimensional entities (tuples) in data structures called associative arrays. D4M associative arrays provide a one-to-one mapping onto the tables in a tuple store that makes complex manipulations simple to code. All rows and columns can be searched efficiently without the need to build specialized indexes.
D4M significantly improves search, retrieval, and analysis of massive amounts of data. The D4M Schema is simple, requires minimal parsing, and achieves the highest published Accumulo ingest rates.
Accumulo extends the BigTable data model to implement a security mechanism known as cell-level security. Every key-value pair has its own security label, stored under the column visibility element of the key, which is used to determine whether a given user meets the security requirements to read the value. This enables data of various security levels to be stored within the same row, and users of varying degrees of access to query the same table, while preserving data confidentiality.