-
-
Notifications
You must be signed in to change notification settings - Fork 266
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Custom encodings #149
Custom encodings #149
Conversation
speaking on behalf of the web browser lobby I would like to ask if this API would let me ensure that levelup/leveldown never try to create a |
Yes! so this is passed straight to leveldown, and leveldown returns whatever it wants https://github.com/rvagg/node-leveldown#leveldowngetkey-options-callback that should be flexible - we could use bops in the leveldown tests, so that a Bufferish object is acceptable. |
@maxogden the reason for the option is to avoid a memcopy in leveldown, depending on whether the custom encoding wants to retrive a buffer or a string (example: JSON needs string, msgpack needs buffers) |
@chrisdickinson have you benchmarked bops at all? there is a question of if it is any slower than plain Buffers when running in node on top of Buffers |
@maxogden I haven't, but I would expect to see a (very) small slowdown as the node version of bops is just directly delegating to the appropriate |
@@ -389,6 +390,38 @@ buster.testCase('ReadStream', { | |||
}.bind(this)) | |||
} | |||
|
|||
, 'test injectable encoding': function (done) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is this testing other than the ability to pass in the encoding object? Can we do a utf8 readStream and collect the data and compare it to what it should look like in msgpack format?
Yeah, I'm cool with this I think. It gives us the flexibility to still pass in alternative encodings to individual operations so you could have crazy stuff like msgpack and bytewise encoded values/keys in the same store. |
Cool, agree on need to refactor. |
In (if you can figure out how to deal with that without some LevelUp support, I'll be very happy!) |
@dominictarr just enlightened me to the fact that I could use this for validations by just throwing if the key didn't match certain parameters. +1 |
Where are we at with this @dominictarr? would be a good one to get out soon. Do you need someone else to help get this finalised? |
I want to put some time into this soon, and also, getting lt, gt, lte, gte into leveldown - probably next week? |
excellent, le tme know if you need any help cleaning up the leveldown work, looking forward to that! |
okay! I've reorganised this to use some utility functions, Also, https://github.com/eugeneware/byteup will break with this change, so @eugeneware should update it. Also, added documentation to readme. |
okay, this is now passing the tests on master & good to merge! |
LATM ("Looks Awesome To Me") If there's no objections by the time you're around next, go ahead and merge and I'll push it in a release, this is fantastic. |
@dominictarr no probs. I'll update it work again. Can't wait for this sucker to land! :-) |
Can't find any easy way to update byteup. @dominictarr any ideas? But that's fine. The new method is better. Byteup will be deprecated once this land in any case. It was an interim hack. Though it would be nice if you could register global types. There is a But it's not a big deal. Let's push this out :-) |
merged. @eugeneware to be honest, I'm a little confused about how byteup is used, I can see it's called by subindex. |
@dominictarr presently you use byteup like so: var bytewise = byteup();
// adds the type "bytewise" to the encodings list with the bytewise.encode / bytewise.decode fns
var db = levelup(dbPath, { keyEncoding: 'bytewise', valueEncoding: 'json' });
// the string 'bytewise' will map to the codec entry added. Once again, not a big deal, but checking if there was some way with the new API to register an entry into the encoders/decoders data structure for a specific string "type" (ie. in the same way that 'utf8' and 'json' exist). Hope I'm making sense. If not, no big dramas, but worth an ask. |
@dominictarr in regards to subindex, I started off using byteup, but I struggled to get sublevel play well with Buffers as a keyencodings (like bytewise). I eventually just encoded the key as a hex string. You're right, the byteup reference is not actually used in subindex any more. |
oh right - that makes sense. I would be opposed to registering encodings,
etc. I am planning on refactoring level-sublevel so that it has real support for binary encoded keys, bytewise, etc. Although, ranges is a higher priority for me. Once that is cleaned up it will remove a lot of crud from level-sublevel. |
Adds support for custom encodings.
Just pass in an object with
{encode: function, decode: function, buffer: boolean}
instead of a string encoding type, for
keyEncoding
,valueEncoding
orencoding
options.So, this could probably be cleaned up, to use more abstraction - like:
var key =this.toKey(key_)
but I did it this way, because was simplest to make smallest change to the current code.
closes #51