Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

some fixes and fix atom issue #4

Open
wants to merge 15 commits into from

3 participants

@benoitc

fixes from fdmanana branch + atom issue fixed

fdmanana and others added some commits
@fdmanana fdmanana Fix encoding of integers and floats
Fixes both the incorrect codes and the badarg errors since the
inputs to list_to_float and list_to_integer are binaries and not
lists.
589b09f
@fdmanana fdmanana A more flexible erl_nif_compat.h version
Ensure that future otp releases get the same compatibility macros as those defined
for OTP R14A/B (the minor and major version macro values for R14B02 changed).
Also added a few more compat macros:

* enif_realloc_binary_compat
* enif_release_binary_compat
* enif_priv_data_compat
* enif_make_string_compat
* enif_make_uint_compat
92dc485
@fdmanana fdmanana Re-added old error return values
Mostly to avoid the test failures. Also because they add a little bit
of extra useful information (number of bytes consumed until the parsing
error occurred).
5f2a813
@fdmanana fdmanana Fix decoding of floats
For some odd reason, Erlang's list_to_float needs a list that contains a '.'.
For example, list_to_float("1e2") fails - it must be invoked as
list_to_float("1.0e2")

Conflicts:

	src/json.erl
9400917
@fdmanana fdmanana On encode and decode, return a tagged tuples
This is mostly to comply to the previous API and make the
tests pass.
a7ed188
@fdmanana fdmanana Fix the numbers test 1a3b241
@fdmanana fdmanana Fix the arrays test 913b0af
@fdmanana fdmanana Fix encoding of property lists
Keys and values were swapped.
a2aff62
@fdmanana fdmanana Use compatibility macros
To ensure compatibility with at least R13B04
05ce33c
@fdmanana fdmanana Like mochijson2, accept Erlang atoms as JSON strings 808b671
@fdmanana fdmanana erl_nif_compat.h: explicitly define OTP_R14A, OTP_R14B and OTP_R14B01…
… macros
fd58243
@fdmanana fdmanana Ensure R13B04 compatibility
enif_is_tuple was introduced in R14, so use enif_get_tuple to check
whether the term is a tuple or not.
71687ac
@fdmanana fdmanana White space cleanup only 8780e30
@fdmanana fdmanana Use yajl error messages 921b16f
@b3no b3no fix encoding. 3cb69be
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Mar 29, 2011
  1. @fdmanana

    Fix encoding of integers and floats

    fdmanana authored
    Fixes both the incorrect codes and the badarg errors since the
    inputs to list_to_float and list_to_integer are binaries and not
    lists.
  2. @fdmanana

    A more flexible erl_nif_compat.h version

    fdmanana authored
    Ensure that future otp releases get the same compatibility macros as those defined
    for OTP R14A/B (the minor and major version macro values for R14B02 changed).
    Also added a few more compat macros:
    
    * enif_realloc_binary_compat
    * enif_release_binary_compat
    * enif_priv_data_compat
    * enif_make_string_compat
    * enif_make_uint_compat
  3. @fdmanana

    Re-added old error return values

    fdmanana authored
    Mostly to avoid the test failures. Also because they add a little bit
    of extra useful information (number of bytes consumed until the parsing
    error occurred).
  4. @fdmanana

    Fix decoding of floats

    fdmanana authored
    For some odd reason, Erlang's list_to_float needs a list that contains a '.'.
    For example, list_to_float("1e2") fails - it must be invoked as
    list_to_float("1.0e2")
    
    Conflicts:
    
    	src/json.erl
  5. @fdmanana

    On encode and decode, return a tagged tuples

    fdmanana authored
    This is mostly to comply to the previous API and make the
    tests pass.
  6. @fdmanana

    Fix the numbers test

    fdmanana authored
  7. @fdmanana

    Fix the arrays test

    fdmanana authored
  8. @fdmanana

    Fix encoding of property lists

    fdmanana authored
    Keys and values were swapped.
  9. @fdmanana

    Use compatibility macros

    fdmanana authored
    To ensure compatibility with at least R13B04
Commits on Mar 31, 2011
  1. @fdmanana
  2. @fdmanana
  3. @fdmanana

    Ensure R13B04 compatibility

    fdmanana authored
    enif_is_tuple was introduced in R14, so use enif_get_tuple to check
    whether the term is a tuple or not.
  4. @fdmanana

    White space cleanup only

    fdmanana authored
  5. @fdmanana

    Use yajl error messages

    fdmanana authored
Commits on Apr 1, 2011
  1. @b3no

    fix encoding.

    b3no authored
This page is out of date. Refresh to see the latest.
View
81 c_src/decode.c
@@ -19,6 +19,32 @@ typedef struct {
#define CANCEL 0
+static ERL_NIF_TERM
+make_error(yajl_handle handle, ErlNifEnv* env)
+{
+ char* yajlError = (char*) yajl_get_error(handle, 0, NULL, 0);
+ ERL_NIF_TERM errMsg;
+
+ if(yajlError != NULL)
+ {
+ errMsg = enif_make_string(env, yajlError, ERL_NIF_LATIN1);
+ yajl_free_error(handle, (unsigned char*) yajlError);
+ }
+ else
+ {
+ errMsg = enif_make_string(env, "unknown parse error", ERL_NIF_LATIN1);
+ }
+
+ return enif_make_tuple(env, 2,
+ enif_make_atom(env, "error"),
+ enif_make_tuple(env, 2,
+ enif_make_uint(env, handle->bytesConsumed),
+ errMsg
+ )
+ );
+}
+
+
static void
add_to_head(void* vctx, ERL_NIF_TERM newhead)
{
@@ -44,26 +70,46 @@ static int
decode_number(void * ctx, const char * numberVal, unsigned int numberLen)
{
// scan in the input to see if it's a float or int
-
+
int numberType = 0; // 0 means integer, 1 means float
unsigned int i;
- ErlNifBinary bin;
-
+ ErlNifBinary bin;
+ int missingDot = 1;
+ unsigned int expPos;
+
for(i=0; i<numberLen; i++) {
switch (numberVal[i]) {
case '.':
+ missingDot = 0;
+ numberType = 1; // it's a float
+ goto loopend;
case 'E':
case 'e':
+ expPos = i;
numberType = 1; // it's a float
goto loopend;
}
}
loopend:
- if(!enif_alloc_binary_compat(ENV(ctx), numberLen, &bin))
+ if ((numberType == 1) && missingDot)
{
- return CANCEL;
+ if(!enif_alloc_binary_compat(ENV(ctx), numberLen + 2, &bin))
+ {
+ return CANCEL;
+ }
+ memcpy(bin.data, numberVal, expPos);
+ bin.data[expPos] = '.';
+ bin.data[expPos + 1] = '0';
+ memcpy(bin.data + expPos + 2, numberVal + expPos, numberLen - expPos);
+ }
+ else
+ {
+ if(!enif_alloc_binary_compat(ENV(ctx), numberLen, &bin))
+ {
+ return CANCEL;
+ }
+ memcpy(bin.data, numberVal, numberLen);
}
- memcpy(bin.data, numberVal, numberLen);
add_to_head(ctx, enif_make_tuple(ENV(ctx), 2,
enif_make_int(ENV(ctx), numberType),
enif_make_binary(ENV(ctx), &bin)));
@@ -164,7 +210,7 @@ check_rest(unsigned char* data, unsigned int size, unsigned int used)
return CANCEL;
}
}
-
+
return CONTINUE;
}
@@ -178,10 +224,10 @@ reverse_tokens(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
unsigned int used;
ErlNifBinary bin;
ERL_NIF_TERM ret;
-
+
ctx.env = env;
ctx.head = enif_make_list_from_array(env, NULL, 0);
-
+
if(!enif_inspect_iolist_as_binary(env, argv[0], &bin))
{
ret = enif_make_badarg(env);
@@ -190,7 +236,7 @@ reverse_tokens(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
status = yajl_parse(handle, bin.data, bin.size);
used = handle->bytesConsumed;
-
+
// Parsing something like "2.0" (without quotes) will
// cause a spurious semi-error. We add the extra size
// check so that "2008-20-10" doesn't pass.
@@ -218,20 +264,7 @@ reverse_tokens(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
goto done;
case yajl_status_error:
- {
- char* unknownError = "Unknown Parse Error";
- char* yajlError = (char*)yajl_get_error(handle, 0, NULL, 0);
- if (!yajlError) { /* can't alloc the error! */
- yajlError = unknownError;
- }
- ret = enif_make_tuple(env, 2,
- enif_make_atom(env, "error"),
- enif_make_string(env, yajlError, ERL_NIF_LATIN1));
-
- if (yajlError != unknownError) {
- yajl_free_error(handle, (unsigned char*)yajlError);
- }
- }
+ ret = make_error(handle, env);
goto done;
case yajl_status_insufficient_data:
View
45 c_src/encode.c
@@ -23,7 +23,7 @@ static int
ensure_buffer(void* vctx, unsigned int len) {
encode_ctx* ctx = (encode_ctx*)vctx;
if ((ctx->bin.size - ctx->fill_offset) < len) {
- if(!enif_realloc_binary(&(ctx->bin), (ctx->bin.size * 2) + len)) {
+ if(!enif_realloc_binary_compat(ctx->env, &(ctx->bin), (ctx->bin.size * 2) + len)) {
return NOMEM;
}
}
@@ -34,7 +34,7 @@ static void
fill_buffer(void* vctx, const char* str, unsigned int len)
{
encode_ctx* ctx = (encode_ctx*)vctx;
-
+
if (ctx->error || (ctx->error = ensure_buffer(vctx, len))) {
return;
}
@@ -49,7 +49,7 @@ encode_string(void* vctx, ERL_NIF_TERM binary)
{
encode_ctx* ctx = (encode_ctx*)vctx;
ErlNifBinary bin;
-
+
if(!enif_inspect_binary(ctx->env, binary, &bin)) {
return NOMEM;
}
@@ -59,7 +59,7 @@ encode_string(void* vctx, ERL_NIF_TERM binary)
}
yajl_string_encode2(fill_buffer, ctx, bin.data, bin.size);
fill_buffer(ctx, "\"", 1);
-
+
return ctx->error;
}
@@ -76,36 +76,30 @@ final_encode(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
{
ERL_NIF_TERM head = argv[0];
ERL_NIF_TERM term;
-
double number;
-
encode_ctx ctx;
+
ctx.env = env;
ctx.fill_offset = 0;
ctx.error = 0;
-
- if (!enif_alloc_binary(100, &ctx.bin)) {
+
+ if (!enif_alloc_binary_compat(env, 100, &ctx.bin)) {
return no_mem_error(env);
}
-
+
while(enif_get_list_cell(env, head, &term, &head)) {
ErlNifBinary termbin;
-
+ const ERL_NIF_TERM* array;
+ int arity;
+ int code;
+
// We scan the list, looking for things to write into the binary, or
// encode and then write into the binary. We encode values that are
// tuples tagged with a type and a value: {Type, Value} where Type
// is a an Integer and Value is what is to be encoded
-
- if (enif_is_tuple(env, term)) {
+
+ if (enif_get_tuple(env, term, &arity, &array)) {
// It's a tuple to encode and copy
- const ERL_NIF_TERM* array;
- int arity;
- int code;
- if (!enif_get_tuple(env, term, &arity, &array)) {
- // shouldn't be possible to get here.
- ctx.error = BADARG;
- goto done;
- }
if (arity != 2 || !enif_get_int(env, array[0], &code)) {
// not arity 2 or the first element isn't an int
ctx.error = BADARG;
@@ -137,7 +131,6 @@ final_encode(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
// increment the length
ctx.fill_offset += strlen((char*)ctx.bin.data+ctx.fill_offset);
}
-
} else if (enif_inspect_binary(env, term, &termbin)) {
// this is a regular binary, copy the contents into the buffer
fill_buffer(&ctx, (char*)termbin.data, termbin.size);
@@ -153,16 +146,16 @@ final_encode(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
}
done:
if (ctx.error == NOMEM) {
- enif_release_binary(&ctx.bin);
+ enif_release_binary_compat(env, &ctx.bin);
return no_mem_error(env);
} else if (ctx.error == BADARG) {
- enif_release_binary(&ctx.bin);
+ enif_release_binary_compat(env, &ctx.bin);
return enif_make_badarg(env);
}
-
+
// Resize the binary to our exact final size
- if(!enif_realloc_binary(&(ctx.bin), ctx.fill_offset)) {
- enif_release_binary(&ctx.bin);
+ if(!enif_realloc_binary_compat(env, &(ctx.bin), ctx.fill_offset)) {
+ enif_release_binary_compat(env, &ctx.bin);
return no_mem_error(env);
}
// make the binary term which transfers ownership
View
57 c_src/erl_nif_compat.h
@@ -7,19 +7,59 @@ extern "C" {
#include "erl_nif.h"
-#if ERL_NIF_MAJOR_VERSION == 1 && ERL_NIF_MINOR_VERSION == 0
+
+#if ERL_NIF_MAJOR_VERSION == 0 && ERL_NIF_MINOR_VERSION == 1
+#define OTP_R13B03
+#elif ERL_NIF_MAJOR_VERSION == 1 && ERL_NIF_MINOR_VERSION == 0
+#define OTP_R13B04
+#elif ERL_NIF_MAJOR_VERSION == 2 && ERL_NIF_MINOR_VERSION == 0
+#define OTP_R14A
+#define OTP_R14B
+#define OTP_R14B01
+#elif ERL_NIF_MAJOR_VERSION == 2 && ERL_NIF_MINOR_VERSION == 1
+#define OTP_R14B02
+#endif
+
+
+#ifdef OTP_R13B03
+
+#define enif_open_resource_type_compat enif_open_resource_type
+#define enif_alloc_resource_compat enif_alloc_resource
+#define enif_release_resource_compat enif_release_resource
+#define enif_alloc_binary_compat enif_alloc_binary
+#define enif_alloc_compat enif_alloc
+#define enif_release_binary_compat enif_release_binary
+#define enif_free_compat enif_free
+#define enif_get_atom_compat enif_get_atom
+#define enif_priv_data_compat enif_get_data
+#define enif_make_uint_compat enif_make_ulong
+
+#define enif_make_string_compat(E, B, Enc) \
+ enif_make_string(E, B)
+
+#endif /* R13B03 */
+
+
+#ifdef OTP_R13B04
#define enif_open_resource_type_compat enif_open_resource_type
#define enif_alloc_resource_compat enif_alloc_resource
#define enif_release_resource_compat enif_release_resource
#define enif_alloc_binary_compat enif_alloc_binary
+#define enif_realloc_binary_compat enif_realloc_binary
+#define enif_release_binary_compat enif_release_binary
#define enif_alloc_compat enif_alloc
#define enif_free_compat enif_free
#define enif_get_atom_compat enif_get_atom
+#define enif_priv_data_compat enif_priv_data
+#define enif_make_string_compat enif_make_string
+#define enif_make_uint_compat enif_make_uint
#endif /* R13B04 */
-#if ERL_NIF_MAJOR_VERSION == 2 && ERL_NIF_MINOR_VERSION == 0
+
+/* OTP R14 and future releases */
+#if !defined(OTP_R13B03) && !defined(OTP_R13B04)
#define enif_open_resource_type_compat(E, N, D, F, T) \
enif_open_resource_type(E, NULL, N, D, F, T)
@@ -33,6 +73,12 @@ extern "C" {
#define enif_alloc_binary_compat(E, S, B) \
enif_alloc_binary(S, B)
+#define enif_realloc_binary_compat(E, S, B) \
+ enif_realloc_binary(S, B)
+
+#define enif_release_binary_compat(E, B) \
+ enif_release_binary(B)
+
#define enif_alloc_compat(E, S) \
enif_alloc(S)
@@ -42,12 +88,15 @@ extern "C" {
#define enif_get_atom_compat(E, T, B, S) \
enif_get_atom(E, T, B, S, ERL_NIF_LATIN1)
-#endif /* R14 */
+#define enif_priv_data_compat enif_priv_data
+#define enif_make_string_compat enif_make_string
+#define enif_make_uint_compat enif_make_uint
+#endif /* R14 and future releases */
#ifdef __cplusplus
}
#endif /* __cplusplus */
-#endif /* ERL_NIF_COMPAT_H_ */
+#endif /* ERL_NIF_COMPAT_H_ */
View
27 src/json.erl
@@ -29,7 +29,7 @@ decode(IoList) ->
encode(EJson) ->
try
RevList = encode_rev(EJson),
- final_encode(lists:reverse(lists:flatten([RevList])))
+ {ok, final_encode(lists:reverse(lists:flatten([RevList])))}
catch throw:Error ->
Error
end.
@@ -48,6 +48,8 @@ encode_rev(I) when is_integer(I) ->
list_to_binary(integer_to_list(I));
encode_rev(S) when is_binary(S) ->
{0, S};
+encode_rev(S) when is_atom(S) ->
+ {0, list_to_binary(atom_to_list(S))};
encode_rev(F) when is_float(F) ->
{1, F};
encode_rev({Props}) when is_list(Props) ->
@@ -69,9 +71,18 @@ encode_array_rev([Val | Rest], Acc) ->
encode_proplist_rev([], Acc) ->
[<<"}">> | Acc];
encode_proplist_rev([{Key,Val} | Rest], [<<"{">>]) ->
- encode_proplist_rev(Rest, [{0, Key}, <<":">>, encode_rev(Val), <<"{">>]);
+ encode_proplist_rev(
+ Rest, [encode_rev(Val), <<":">>, {0, as_binary(Key)}, <<"{">>]);
encode_proplist_rev([{Key,Val} | Rest], Acc) ->
- encode_proplist_rev(Rest, [{0, Key}, <<":">>, encode_rev(Val), <<",">> | Acc]).
+ encode_proplist_rev(
+ Rest, [encode_rev(Val), <<":">>, {0, as_binary(Key)}, <<",">> | Acc]).
+
+as_binary(B) when is_binary(B) ->
+ B;
+as_binary(B) when is_list(B) ->
+ list_to_binary(B);
+as_binary(A) when is_atom(A) ->
+ list_to_binary(atom_to_list(A)).
@@ -90,12 +101,12 @@ make_ejson([2 | RevEvs], [ObjValues, PrevValues | RestStack]) ->
make_ejson([3 | RevEvs], Stack) ->
% 3 ObjectEnd
make_ejson(RevEvs, [[] | Stack]);
+make_ejson([{0, Value} | RevEvs], [Vals | RestStack] = _Stack) ->
+ % {0, IntegerString}
+ make_ejson(RevEvs, [[list_to_integer(binary_to_list(Value)) | Vals] | RestStack]);
make_ejson([{1, Value} | RevEvs], [Vals | RestStack] = _Stack) ->
- % {1 , IntegerString}
- make_ejson(RevEvs, [[list_to_integer(Value) | Vals] | RestStack]);
-make_ejson([{2, Value} | RevEvs], [Vals | RestStack] = _Stack) ->
- % {2, FloatString}
- make_ejson(RevEvs, [[list_to_float(Value) | Vals] | RestStack]);
+ % {1, FloatString}
+ make_ejson(RevEvs, [[list_to_float(binary_to_list(Value)) | Vals] | RestStack]);
make_ejson([{3, String} | RevEvs], [[PrevValue|RestObject] | RestStack] = _Stack) ->
% {3 , ObjectKey}
make_ejson(RevEvs, [[{String, PrevValue}|RestObject] | RestStack]);
View
2  test/003-numbers.t
@@ -19,7 +19,7 @@ good() ->
{<<"309230948234098">>, 309230948234098},
{<<"1.0">>, 1.0, <<"1">>},
{<<"0.3">>, 0.3},
- {<<"2.4234324">>, 2.4234324, <<"2.42343">>},
+ {<<"2.4234324">>, 2.4234324, <<"2.4234324">>},
{<<"-3.1416">>, -3.1416},
{<<"1E4">>, 10000.0, <<"10000">>},
{<<"1.0E+01">>, 10.0, <<"10">>},
View
2  test/005-arrays.t
@@ -15,7 +15,7 @@ good() ->
{<<"[\t[\n]\r]">>, [[]], <<"[[]]">>},
{<<"[\t123, \r true\n]">>, [123, true], <<"[123,true]">>},
{<<"[1,\"foo\"]">>, [1, <<"foo">>]},
- {<<"[1199344435545.0,1]">>, [1199344435545.0,1], <<"[1.19934e+12,1]">>},
+ {<<"[1199344435545.0,1]">>, [1199344435545.0,1], <<"[1199344435545,1]">>},
{
<<"[\"\\u00A1\",\"\\u00FC\"]">>,
[<<194, 161>>, <<195, 188>>],
View
2  test/cases/array_close.erl
@@ -1 +1 @@
-{error,{1,invalid_token}}.
+{error,{1,"parse error: unallowed token at this point in JSON text\n"}}.
View
2  test/cases/bogus_char.erl
@@ -1 +1 @@
-{error,{97,invalid_string}}.
+{error,{97,"lexical error: invalid string in json text.\n"}}.
View
2  test/cases/invalid_utf8.erl
@@ -1 +1 @@
-{error,{11,invalid_utf8}}.
+{error,{11,"lexical error: invalid bytes in UTF8 string.\n"}}.
View
2  test/cases/leading_zero_in_number.erl
@@ -1 +1 @@
-{error,{16,bad_token_after_map_value}}.
+{error,{16,"parse error: after key and value, inside map, I expect ',' or '}'\n"}}.
View
2  test/cases/lonely_minus_sign.erl
@@ -1 +1 @@
-{error,{82,missing_integer_after_minus}}.
+{error,{82,"lexical error: malformed number, a digit is required after the minus sign.\n"}}.
View
2  test/cases/map_close.erl
@@ -1 +1 @@
-{error,{1,invalid_token}}.
+{error,{1,"parse error: unallowed token at this point in JSON text\n"}}.
View
2  test/cases/missing_integer_after_decimal_point.erl
@@ -1 +1 @@
-{error,{3,missing_integer_after_decimal}}.
+{error,{3,"lexical error: malformed number, a digit is required after the decimal point.\n"}}.
View
2  test/cases/missing_integer_after_exponent.erl
@@ -1 +1 @@
-{error,{3,missing_integer_after_exponent}}.
+{error,{3,"lexical error: malformed number, a digit is required after the exponent.\n"}}.
View
2  test/cases/non_utf8_char_in_string.erl
@@ -1 +1 @@
-{error,{125,invalid_utf8}}.
+{error,{125,"lexical error: invalid bytes in UTF8 string.\n"}}.
View
2  test/cases/string_invalid_escape.erl
@@ -1 +1 @@
-{error,{62,invalid_escaped_char}}.
+{error,{62,"lexical error: inside a string, '\\' occurs before a character which it may not.\n"}}.
View
2  test/cases/string_invalid_hex_char.erl
@@ -1 +1 @@
-{error,{44,invalid_hex_char}}.
+{error,{44,"lexical error: invalid (non-hex) character occurs after '\\u' inside string.\n"}}.
View
2  test/cases/string_with_invalid_newline.erl
@@ -1 +1 @@
-{error,{66,invalid_json_char}}.
+{error,{66,"lexical error: invalid character inside string.\n"}}.
Something went wrong with that request. Please try again.