Skip to content
A msgpack v5 implementation for node.js, with extension points / msgpack.org[Node]
JavaScript HTML
Branch: master
Clone or download

Latest commit

thephoenixofthevoid and mcollina Improve internals and add some features (#78)
* Feature: Manually encode unsuppoted object.

* Refactoring: inlining hasMinBufferSize and simplifying decode(buf)

Changes to be committed:
  modified:   lib/decoder.js

* Used while where it actually meant to.

* In decodeMap(...) simplify counting amount of bytes consumed

Well, provided that initialOffset is the value passed as an argument:
console.log(headerLength + totalBytesConsumed === offset - initialOffset)
// true   --- always

* Imporove maintainability

1. getSize now is implemented through hash not switch statement.
   More concise and appear to be much faster.

2. Created helpers.js file, and functions that logically unrelated
   to decoding/encoding will go there.

3. Where possible prefer flat code that nested branches

4. In decodeExt dramatically simplify all the things.

Changes to be committed:
	modified:   lib/decoder.js
	new file:   lib/helpers.js

* Getting rid of "offset + 1" artificial expression

It's caused by not incrementing offset after reaading first byte.
To handle this change I first of all introduce "initialOffset" and
change everywhere offset to "initial offset". This commit does
this part of the job.

Then I will clean 1-by-1 every affected function and branch.

* Keep on getting intent more clear

1. Remove "else" statements that go after "return"
2. Where appropriate (another 1-4 bytes have been read) offset is incremented
3. So that expressions like "initialOffset + 2" --- "... + 5" could be replaced by just "offset"

Changes to be committed:
  modified:   lib/decoder.js

* Implemented all integers and all kinds of floats

1. function decodeSigned (buf, offset, size)
2. function decodeFloat (buf, offset, size)
3. function decodeExt (buf, offset, type, size, headerSize)

* Added missing Ranges as TODO commends

Currently:

| RANGE      |  STATUS             | function name (if exists) or type name  |
|------------|---------------------|-----------------|
| 0x00, 0x7f |  MISSING COMPLETELY | positive fixint
| 0x80, 0x8f |  MISSING COMPLETELY | fixmap
| 0x90, 0x9f |  MISSING COMPLETELY | fixarr
| 0xa0, 0xbf |  MISSING COMPLETELY | fixstr
| 0xc0, 0xc3 |  DONE               | decodeConstants(...)
| 0xc4, 0xc6 |  OLD CODE           | decodeBuffers(...)
| 0xc7, 0xc9 |  WIP NOW            | decodeExt(...)
| 0xca, 0xcb |  DONE               | decodeFloat
| 0xcc, 0xcf |  DONE               | decodeUnsignedInt()
| 0xd0, 0xd3 |  DONE               | decodeSigned()
| 0xd4, 0xd8 |  DONE               | decodeFixExt()
| 0xd9, 0xdb |  OLD CODE           | decodeStr(...)
| 0xdc, 0xdd |  OLD CODE           | decodeArray(...)
| 0xde, 0xdf |  OLD CODE           | decodeMap(...)
| 0xe0, 0xff |  MISSING COMPLETELY | negative fixint

* function decodeStr (buf, offset, size)

* Added decodeBuffers

* gitignore updated

* if (inRange(0xc7, 0xc9)) return decodeExts(buf, offset, size - 2)

* decodeFixExt + Moved functions outside of closure

* It was just a line!

* Update .gitignore

* decodeArray

* small arrays and

* decodeMaps + ordering

* fixmap fixarr fixstr

* Inline buf.readUIntBE(offset, size) method where getLength (buf, offset, size) used

* function decodeTimestamp (buf, size, headerSize) moved to helper

* Use array syntax to handle results

* Few changes

* Avoid using length var at top level

* Avoid using result var at top level

* remove dist folder from git

* Restore dist files

* Restore dist min files

* Finally restore min version

* Inline decodeExts helper

* Revert transform unsupported

* Clean up index.js

* Import IncompleteBufferError from helpers.js and not reexport in decoder.js

* Remove passing transformUnsupported option from index.js

* Small fixes

* Small fixes

* Fix travis builds

* Fix travis builds 2

* Fix travis builds 3

* function getHeaders

* function getHeader

* move some functions out of closure

* Better handling of TimestampEncoding

* encodeFloat function simplified

* Feature: Sort Keys

* Feature: support encoding/decoding Maps

* Feature: support encoding/decoding Maps

* Test descr updated

* Move all Timestamp-related into separate file

* Use bl constuctor instead of append method

* Unify headers methods

* Import helpers from helpers.js

* fix build

* restructure DataCodec module
Latest commit 3cdd1d4 May 24, 2019

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
benchmarks standard Feb 11, 2016
dist Bumped v4.2.1. Sep 17, 2018
lib Improve internals and add some features (#78) May 24, 2019
test Improve internals and add some features (#78) May 24, 2019
.gitignore Refactoting of decoder for better maintainability (#76) Apr 26, 2019
.jshintrc Sanitized all files for undefined variables in loops. Jul 29, 2014
.travis.yml Updated dependencies. Removed testling. May 17, 2018
CONTRIBUTING.md Fixed typos in CONTRIBUTING.md Jan 10, 2015
LICENSE Initial commit Jul 17, 2014
README.md Improve internals and add some features (#78) May 24, 2019
example.js Update dependencies. Jul 17, 2017
index.js Improve internals and add some features (#78) May 24, 2019
package.json Bumped v4.2.1. Sep 17, 2018
spec.html Encoding fixints. Jul 17, 2014
spec.md Encoding fixints. Jul 17, 2014

README.md

msgpack5  Build Status

A msgpack v5 implementation for node.js and the browser, with extension point support.

Install

npm install msgpack5 --save

Usage

var msgpack = require('msgpack5')() // namespace our extensions
  , a       = new MyType(2, 'a')
  , encode  = msgpack.encode
  , decode  = msgpack.decode

msgpack.register(0x42, MyType, mytipeEncode, mytipeDecode)

console.log(encode({ 'hello': 'world' }).toString('hex'))
// 81a568656c6c6fa5776f726c64
console.log(decode(encode({ 'hello': 'world' })))
// { hello: 'world' }
console.log(encode(a).toString('hex'))
// d5426161
console.log(decode(encode(a)) instanceof MyType)
// true
console.log(decode(encode(a)))
// { value: 'a', size: 2 }

function MyType(size, value) {
  this.value = value
  this.size  = size
}

function mytipeEncode(obj) {
  var buf = new Buffer(obj.size)
  buf.fill(obj.value)
  return buf
}

function mytipeDecode(data) {
  var result = new MyType(data.length, data.toString('utf8', 0, 1))
    , i

  for (i = 0; i < data.length; i++) {
    if (data.readUInt8(0) != data.readUInt8(i)) {
      throw new Error('should all be the same')
    }
  }

  return result
}

In the Browser

This library is compatible with Browserify.

If you want to use standalone, grab the file in the dist folder of this repo, and use in your own HTML page, the module will expose a msgpack5 global.

<script type="text/javascript"
        src="./msgpack5.min.js">
</script>

To build

	npm run build

API

API


msgpack(options(obj))

Creates a new instance on which you can register new types for being encoded.

options:

  • forceFloat64, a boolean to that forces all floats to be encoded as 64-bits floats. Defaults to false.
  • sortKeys, a boolean to force a determinate keys order
  • compatibilityMode, a boolean that enables "compatibility mode" which doesn't use str 8 format. Defaults to false.
  • disableTimestampEncoding, a boolean that when set disables the encoding of Dates into the timestamp extension type. Defaults to false.

encode(object)

Encodes object in msgpack, returns a bl.


decode(buf)

Decodes buf from in msgpack. buf can be a Buffer or a bl instance.

In order to support a stream interface, a user must pass in a bl instance.


registerEncoder(check(obj), encode(obj))

Register a new custom object type for being automatically encoded. The arguments are:

  • check, a function that will be called to check if the passed object should be encoded with the encode function
  • encode, a function that will be called to encode an object in binary form; this function must return a Buffer which include the same type for registerDecoder.

registerDecoder(type, decode(buf))

Register a new custom object type for being automatically decoded. The arguments are:

  • type, is a greater than zero integer identificating the type once serialized
  • decode, a function that will be called to decode the object from the passed Buffer

register(type, constructor, encode(obj), decode(buf))

Register a new custom object type for being automatically encoded and decoded. The arguments are:

  • type, is a greater than zero integer identificating the type once serialized
  • constructor, the function that will be used to match the objects with instanceof
  • encode, a function that will be called to encode an object in binary form; this function must return a Buffer that can be deserialized by the decode function
  • decode, a function that will be called to decode the object from the passed Buffer

This is just a commodity that calls registerEncoder and registerDecoder internally.


encoder(options)

Builds a stream in object mode that encodes msgpack.

Supported options:

  • wrap, objects should be passed to encoder in wrapped object {value: data}. Wrap option should be used if you need to pass null to encoder.

decoder(options)

Builds a stream in object mode that decodes msgpack.

Supported options:

  • wrap, decoded objects returned in wrapped object {value: data}. Wrap option should be used if stream contains msgpack nil.

LevelUp Support

msgpack5 can be used as a LevelUp valueEncoding straight away:

var level = require('level')
  , pack  = msgpack()
  , db    = level('foo', {
      valueEncoding: pack
    })
  , obj   = { my: 'obj' }

db.put('hello', obj, function(err) {
  db.get('hello', function(err, result) {
    console.log(result)
    db.close()
  })
})

Related projects

Disclaimer

This library is built fully on JS and on bl to simplify the code. Every improvement that keeps the same API is welcome.

Acknowledgements

This project was kindly sponsored by nearForm.

This library was originally built as the data format for JSChan.

License

MIT

You can’t perform that action at this time.