Skip to content

Raynos/charwise-compact

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

52 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

charwise-compact

like charwise, except numbers are compact.

codec for js values (including arrays) that preserves lexiographic sort order when encoded. (the order is compatible with bytewise and thus indexeddb and couchdb, but the encoded format is different)

The api provided follows the level codec standard so this encoding can easily be used with level and flumedb

motivation

for building indexes on top of leveldb, bytewise is great! it lets you structure you keys and reason about how they will be ordered in a very simple and reliable way.

But bytewise is too slow! it's slow enough to have quite visible effects on a bulk load on a reasonable sized database with a couple of indexes. (i.e. 100k secure-scuttlebutt messages with indexes, measured by bench-ssb)

stability: experimental

Expect breaking changes to encoded format. We are still making breaking changes if necessary to improve performance.

(although, codec api is fully stable and will not change)

simple benchmark

run a simple benchmark for one second, encoding & decoding ops in one second.

# name, ops, multiplier
bytewise encode 35661
charwise encode 131366 x3.6
bytewise decode 107571
charwise decode 144557 x1.3

It was easy to make charwise faster than bytewise when it was only a partial implementation, but once correct escaping and nested arrays where added it got slow.

But then @PaulBlanche had the genious idea of encoding items in an array with their depth inside the array. This supports deeply nested arrays or shallowly nested arrays with only one pass escaping the items. This made encoding much faster again!

Examples

const level = require('level')
const charwise = require('charwise')

const db = level('./db8', {
  keyEncoding: charwise
})

await db.put(['users', 2], 'example')
await db.put(['users', 10], 'example2')

const userStream = db.createStream({
  gte: ['users', charwise.LO],
  lte: ['users', charwise.HI]
})

// This will print ['users', 2], ['users', 10]
// If you dont use charwise its sorted numerically and would
// print ['users', 10] , ['users', 2]
userStream.on('data', console.log)

License

MIT

About

charwise-compact

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • JavaScript 100.0%