Skip to content

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

gem skjvs

Very simple yet efficient "string key -> json value" storage, i.e. keys are automatically converted to MD5, values are serialized to JSON.

Persistent file is concurrently accessible from any number of threads or processes. Consistency is gauranteed by file locking. Faster than YAML::Store by magnitudes. You only pay with theoretical MD5 collisions and growing file size on value overwrite.

Installation

$ gem install skjvs

Usage

Only two methods are supported for now: [] and []=.

require "skjvs/one_file"
store = SKJVS::OneFile.new
store[123] = "2\n3"
assert_equal "2\n3", store[123]

Store file is meant to be accessible from multiple processes at the same time. Though the store file is thread-safe, but store object isn't, so for each thread do own SKJVS::OneFile.new.

Benchmark

Compared to YAML::Store, PStore, gem sequel (sqlite).

image

Loses to Sequel after skjvs_store.txt reaches 30 MB and corresponding sqlite.db reaches 100 KB.

image

Not sure why file size difference is so big -- maybe because benchmark is too aritifical and values are duplicating, and sqlite spots it and compacts, idk, didn't debug, didn't profile.
rake benchmark could be done better, could be split into first-fetch/fetch/write, etc. Later.

TODO: compare to https://github.com/ruby/sdbm, https://github.com/ruby/dbm, https://github.com/propublica/daybreak, etc.

Notes

The SKJVS::OneFile is optimized for speed. Later the SKJVS::TwoFiles may be implemented for memory optimization.

By current design the store file isn't editing, only growing, so value overwriting creates extra line. You may want to compact the file by yourself (TODO: a tool), but only in between of its usage (for example, restart processes).

To better understand the design and trade-offs of test and benchmark, see automated TLDR:

image

Development:

$ bundle install
$ bundle exec ruby test.rb
$ bundle exec rake benchmark
$ rake -rbundler/gem_tasks release

About

gem skjvs -- string key json value storage

Topics

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Contributors

Languages