transit format for erlang
Erlang Makefile
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
bench
config
eqc_test
include
src
test Update all exemplars, add vector_special_numbers test. Apr 13, 2015
.eqc_ci
.gitignore
.travis.yml
EQC_CI_LICENCE.txt
LICENSE
Makefile
README.md
dialyzer.ignore-warnings
rebar.config
rebar.lock

README.md

Build Status

transit-erlang

transit-format implementation in Erlang.

Test and developed on Erlang/OTP R17+.

NOTE: Things are still experimental and subject to change.

Usage

rebar get-deps compile
erl -pa ebin deps/*/ebin
A = transit:write(#{<<"a">> => <<"b">>, 3 => 4}, #{ format => json }).
%% => <<"[\"^ \",\"a\",\"b\",3,4]">>
transit:read(A, [{format, json}]).
%% => #{<<"a">> => <<"b">>, 3 => 4}

%%% JSON Verbose mode
transit:write(#{<<"a">> => <<"b">>, 3 => 4}, #{ format => json_verbose }).
%% => <<"{\"~i3\":4,\"a\":\"b\"}">>

%%% msgpack
transit:write(#{<<"a">> => <<"b">>, 3 => 4}, #{ format => msgpack }).
%% => <<149,162,94,32,161,97,161,98,163,126,105,51,4>>

Benchmarks

These benchmarks are run on a Lenovo Thinkpad W540 with a 16 Gigabyte RAM configuration and the following CPU core:

Intel(R) Core(TM) i7-4900MQ CPU @ 2.80GHz

Timings run 300 rounds of encoding of the file transit-format/examples/0.8/example.json and then we divide down to get the encoder time for each round. This then forms the base benchmark.

Erl Commit Renderer Vsn Test Timing ms
17.3 3d3b04e JSON #{iso => 31.987, read => 9.976, write => 20.810}
17.3 c976ce6 JSON #{iso => 29.248, read => 8.883, write => 18.700}
17.3 9d678c8 JSON #{iso => 26.6893,read => 7.454,write => 18.178}
17.5 6dfbb23 327f0d2 JSON #{iso => 18.381, read => 5.240,write => 13.407}
18.0-rc1 6dfbb23 327f0d2 JSON #{iso => 18.172, read => 5.304,write => 12.824}
17.3 3d3b04e MsgPack #{iso => 15.911, read => 4.901, write => 12.072}
17.3 c976ce6 MsgPack #{iso => 11.713, read => 3.258, write => 9.051}
17.3 9d678c8 MsgPack #{iso => 11.2637, read => 2.8620, write => 9.0897}
18.0-rc1 6dfbb23 327f0d2 MsgPack #{iso => 11.910, read => 2.800, write => 8.822}
17.5 6dfbb23 c666a5f MsgPack #{iso => 11.818, read => 2.801, write => 9.000}
17.3 3d3b04e JSON_Verbose #{iso => 34.236, read => 9.724, write => 25.638}
17.3 c976ce6 JSON_Verbose #{iso => 36.613, read => 9.572, write => 27.120}
17.3 9d678c8 JSON_Verbose #{iso => 33.36954, read => 8.59574, write => 29.23906}
17.5 6dfbb23 327f0d2 JSON_Verbose #{iso => 19.714, read => 5.250, write => 13.440}

The JSX library can decode at different speeds depending on the version. An older version of JSX decoded in 5.630ms but a newer version is able to decode the same data in 3.504ms. MsgPack decodes at roughly 0.9ms for the same data. These provide some minimum bounds in decoder speed.

Current limitations

  • Points-in-time before the date 1/1 1970 are not encoded and decoded correctly.

Default type mapping

We currently handle the types in the given table with the given mappings.

Rationale for the mapping: The problem we face in Erlang w.r.t transit is that we can't really map external data directly into atom() types. The reason is the atom-table is limited and an enemy can easily outrun it. Other language implementations are not with this limit, so they will just use keywords as they go along, ignoring all limitations of them in Erlang. Thus, we opt for a solution where the low-level mapping is to map a lot of things into binary types, but tag them as we do so to discriminate them.

We chose to handle a "naked" binary() as an UTF-8 string.

Mapping override

In order to handle data in a neat Erlang-esque way, it is possible to supply a translation table in the decoder direction. This table is used to handle scalar types and map them into other data. The intended use is to support binary() → atom() conversion for keywords and symbols. But it also useful for direct decoding of other types.

9> transit:write({kw, <<"foo">>}).                                                                        
<<"[\"~#'\",\"~:foo\"]">>
10> transit:read(v(9), #{ format => json, translate_fun => fun ({kw, <<"foo">>}) -> foo; (X) -> X end }).
foo

We are currently not able to support:

  • Link types
Transit type Write accepts Read returns
null undefined undefined
string binary() binary()
boolean true, false true, false
integer integer() integer()
decimal float() float()
big integer integer() integer()
time {timepoint, now()}; now() {timepoint, now()
keyword {kw, binary()}; atom() {kw, binary()}
symbol {sym, binary()} {sym, binary()}
uri {uri, binary()} {uri, binary()}
uuid {uuid, binary()} {uuid, binary()}
bytes {binary, binary()} {binary, binary()}
special number nan, infinity, neg_infinity nan, infinity, neg_infinity
array list list
list {list, list(transit())} {list, list(transit())}
set sets, gb_sets, ordsets sets
map map map