Skip to content
A flat-file database optimized to hold time-series data.
Branch: master
Clone or download
Latest commit a1d3d3a Feb 23, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
maintenance fix for returning last line of file Dec 1, 2017
src store JSON with unescaped slashes Dec 1, 2017
tests
.gitignore fix for returning last line of file Dec 1, 2017
CONTRIBUTING.md add contributing.md Sep 23, 2015
README.md fix for returning last line of file Dec 1, 2017
TODO.md
composer.json fix license identifier Feb 23, 2018
phpunit.xml add tests for helper class Aug 15, 2015

README.md

QuartzDB

A flat-file database optimized to hold time-series data.

When should I use this?

Some great uses for QuartzDB:

  • Storing GPS logs (just store your records as GeoJSON!)
  • Storing chat logs from IRC, Slack, SMS or iMessage, etc
  • Storing sensor logs from IoT devices, such as in-house environment sensors

You should use this if...

  • ... you believe text files are easier to back up and keep multiple copies of
  • ... you believe text files are more likely than binary database files to last long-term

API

$db = new Quartz\DB('/path/to/folder', 'w');
$db = new Quartz\DB('/path/to/folder', 'r');

Adding Records

$db->add($date, $data);

This will add the entry always to the end of the appropriate file, since it is expected that data will be inserted in order. If you need to insert out of order, you can run the maintenance task to re-sort the date's file when needed.

Querying

Range

$results = $db->queryRange($fromDate, $toDate);
// Loop through the result set.
// The JSON is parsed when the row is accessed.
foreach($results as $id=>$record) {
  echo $id . ": " . $record->property . "\n";
}

Last n records

$results = $db->queryLast(10);
foreach($results as $id=>$record) {
  echo $id . ": " . $record->property . "\n";
}

Single Record

$db->getByDate($date);
$db->getByID("YYYYMMDDline");

Maintenance

// Re-sorts all the data in the shard by date.
// This requires loading the entire file into memory.
$shard = $db->getShard("YYYY","MM","DD");
$shard->sort();

Filesystem

One file per day.

The base path of the database contains a "data" folder and "index" folder.

/data/
     /2015/08/05.txt
     /2015/08/05.meta
     /2015/08/06.txt
     /2015/08/06.meta
     /2015/08/07.txt
     /2015/08/07.meta

Files contain one record per line, separated by newlines. The first 26 characters are the date with microsecond precision, followed by a space, followed by the JSON record.

2015-08-03 09:00:00.000000 {"foo":"bar","value":300}
2015-08-03 09:00:01.000000 {"foo":"bar","value":300}
2015-08-03 09:00:02.000000 {"foo":"bar","value":300}

The ".meta" files are for storing information about the data file. Currently the only thing in the meta file is line 1 contains the number of lines in the data file.

There is also a file in the root of the data folder containing the date of the last date seen. This is for quickly retrieving the last n records in the database without having to search the filesystem for all files.

License

Copyright 2015-2017 by Aaron Parecki

QuartzDB is licensed under the Apache 2.0 license

You can’t perform that action at this time.