No description, website, or topics provided.
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.

wterl is an Erlang interface to the WiredTiger database, and is written to support a Riak storage backend that uses WiredTiger.

Remaining work includes:


  • Find/fix any code marked "TODO:"
    • Why do we see {error, {eperm, _}} result on wterl:cursor_close/1 during fold_objects/4?
    • Why do we see {error, {eperm, _}} result on wterl:cursor_close/1?
    • Why do we see {error, {eperm, _}} result on wterl:cursor_next/1 during is_empty/1?
    • Why do we see {error, {eperm, _}} result on wterl:cursor_next_value/1 during status/1?
    • Why do we see {error, {ebusy, _}} result on wterl:drop/2?
    • Determine a better way to estimate the number of sessions we should configure WT for at startup in riak_kv_wterl_backend:max_sessions/1.
  • Make sure Erlang is optimizing for selective receive in async_nif_enqueue/3 because in the eLevelDB driver there is a comment: "This cannot be a separate function. Code must be inline to trigger Erlang compiler's use of optimized selective receive."
  • Provide a way to configure the cursor options, right now they are always "raw,overwrite".
  • Add support for Riak/KV 2i indexes using the same design pattern as eLevelDB (in a future version consider alternate schema)
  • If an operation using a shared cursor results in a non-normal error then it should be closed/discarded from the recycled pool
  • Cache cursors based on hash(table/config) rather than just table.
  • Finish NIF unload/reload functions and test.
  • Test an upgrade, include a format/schema/WT change.
  • When WT_PANIC is returned first try to unload/reload then driver and reset all state, if that fails then exit gracefully.
  • Currently the riak_kv_wterl_backend module is stored in this repository, but it really belongs in the riak_kv repository.
  • wterl:truncate/5 can segv, and its tests are commented out
  • Add async_nif and wterl NIF stats to the results provided by the stats API
  • Longer term ideas/changes to consider:
    • More testing, especially pulse/qc
    • Riak/KV integration
      • Store 2i indexes in separate tables
      • Store buckets, in separate tables and keep a <<bucket/key>> index to ensure that folds across a vnode are easy
      • Provide a drop bucket API call
      • Support key expirey
      • An ets API (like the LevelDB's lets project)
      • Use mime-type to inform WT's schema for key value encoding
  • Other use cases within Riak
    • An AAE driver using WT
    • An ability to store the ring file via WT


You can deploy wterl into a Riak devrel cluster using the enable-wterl script. Clone the riak repo, change your working directory to it, and then execute the enable-wterl script. It adds wterl as a dependency, runs make all devrel, and then modifies the configuration settings of the resulting dev nodes to use the WiredTiger storage backend.