Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Use enif_consume_timeslice and don't monopolize scheduler thread #49

Closed
wants to merge 2 commits into from

5 participants

@urbanserj

No description provided.

@davisp
Owner

Wow. That's much smaller than I was expecting to convert to using the new timeslice function. I'd like to see a couple changes before merging this though.

First, don't make this a compile time switch. Just convert the whole thing over and use a single #ifdef to change the definition of consume_timeslice that return's 0 if the function doesn't exist. I'd probably put this into util.c as jiffy_consume_timeslice so that if we have to call platform dependent functions then its easier to keep the ifdef'ed functions in a single spot.

Secondly, the calculation used to give a percentage to enif_consume_timeslice is confusing to read. As I read it, its basically calculating 1 byte of encoded or decoded binary as 0.1 percent of the total timeslice. There's also no range check to make sure it ends up between 1 and 100. In theory a long string could put this way beyond 100. In theory at least the time per byte can vary significantly. I'd probably vote in favor of just estimating based on elapsed wall-clock time assuming we can find a sub-millisecond resolution clock on each platform. Or perhaps we could fallback to some bytes consumed method if not.

Third, the processed byte calculation looks subtly broken when encoding bignums (via enc_unknown) because of how the iolist resets d->i to zero. I didn't read too hard there but I didn't see any accounting for that.

All in all, this looks fairly solid. Definitely a lot cleaner than I was expecting.

@urbanserj

Hi, Paul.

First, don't make this a compile time switch. Just convert the whole thing over and use a single #ifdef to change the definition of consume_timeslice that return's 0 if the function doesn't exist. I'd probably put this into util.c as jiffy_consume_timeslice so that if we have to call platform dependent functions then its easier to keep the ifdef'ed functions in a single spot.

I have removed compile time switch #ifdef TIMESLICE and replaced macro consume_timeslice by jiffy_consume_timeslice function. I have also moved resources allocations in yield.

There's also no range check to make sure it ends up between 1 and 100.

This check is not necessary because it is present at enif_consume_timeslice: https://github.com/erlang/otp/blob/OTP_R16B01/erts/emulator/beam/erl_nif.c#L1451-L1452. But i decided to add this check for not to have problems with debug builds of beam.

I'd probably vote in favor of just estimating based on elapsed wall-clock time assuming we can find a sub-millisecond resolution clock on each platform. Or perhaps we could fallback to some bytes consumed method if not.

I used getticks from cycle.h, which is MIT licensed. For platforms x86 and x86_64 it uses the instruction RDTSC. I believe that it's a very good metric in our case.

I can squash commits into one if you wish.

Do you agree with everything else?

Thanks.

@davisp
Owner

@urbanserj First off, this is quite awesome. You made all the changes I requested just fine and I'm planning on merging this but I'm still trying to reason my way through the calculation for the time slice call.

I'm new to cycle.h but as I read the file itself it seems quite adamant that we shouldn't be trying to convert it to a time unit:

https://github.com/urbanserj/jiffy/blob/364b9ef7aae0ced615591b0055ba8e1ce65ae7cb/c_src/cycle.h#L40-L42

I did some googling on various time functions to see if we couldn't cover most platforms and I was reminded on Windows' terribleness with time.

I'm thinking about switching back to something along the lines of your original patch but phrasing it slightly different. For both decoding and encoding we'll add an option that is the amount of data decoded or encoded that's handled before yielding back to Erlang. Then before yielding we just call enif_consume_timeslice(env, 100) to call it a full time slice. This seems to make a lot more sense to me rather than attempting to try and play games with Erlang's idea of a time slice as a unit of time.

How does that sound?

@urbanserj

1 millisecond from enif_consume_timeslice documentation is a time estimate, but not a strict claim. Number of reductions in beam is not tunable, and it's the same for various platforms, including xeon or raspberry pi, isn't it?

Indeed, cycle.h documentation says that we shouldn't treat ticks as time, but degree of number of ticks is an approximation of nanoseconds. As I sad before, it's a good metric in our case.

Then before yielding we just call enif_consume_timeslice(env, 100) to call it a full time slice.

Before first call of jiffy, process can produce 1999 reductions. If jiffy calls enif_consume_timeslice only once w/ 100, it will be bad for system responsiveness. Maybe it's a good idea to call enif_consume_timeslice on every 10 percents on first call, and than use 100 as the second argument.

For both decoding and encoding we'll add an option that is the amount of data decoded or encoded that's handled before yielding back to Erlang.

I suppose that since noone can change number of reductions in beam, this setting won't be popular.

I did some research using erlang:system_monitor(Pid, [{long_schedule, 1}]), but I haven't got any significant results yet.

@davisp
Owner
@urbanserj

Please check out some commits I've made yesterday.

@davisp
Owner

@urbanserj Awesome work! If you squash this into a single commit I'll pull it down and start running it through tests on R14 and R15 and hopefully figure out a reproducible test case that demonstrates it doesn't screw up the scheduling algorithm.

@urbanserj

I've squashed it into a single commit.

@knutin

Hey,

I've been playing around a bit with these changes and ran into a problem causing the beam to segfault. When encoding a bignum, jiffy returns an iolist and the continuation seems to not handle this very well.

Here's how to reproduce it (at least on R16B02):

1> jiffy:encode([trunc(math:pow(2, 64)) || _ <- lists:seq(1, 1000)]).
[1]    11130 segmentation fault (core dumped)  erl -pa ebin

As a bonus, here is something very strange:

1> jiffy:encode([trunc(math:pow(2, 64)) || _ <- lists:seq(1, 960)]), ok.
ok
2> jiffy:encode([trunc(math:pow(2, 64)) || _ <- lists:seq(1, 960)]), ok.
[1]    11372 segmentation fault (core dumped)  erl -pa ebin
@davisp
Owner

@knutin I can duplicate locally on R16B01 as well. Its also not deterministic and appears to be happening back in the VM after the NIF returns which usually means its a fun misuse of the APIs. I'll look at it for a bit to see if I can find it.

@davisp
Owner

Interesting note, this doesn't seem to reproduce it. Granted this should hit the new resumable logic.

[jiffy:encode(trunc(math:pow(2, 64))) || _ <- lists:seq(1, 1000)], ok.
@davisp
Owner

I'm seeing a couple things that look suspect as well as a couple ways to make this a bit less complex to reason through. I'll try and find some time in the next few days to get this finished up. Apologies to @urbanserj for not getting to this sooner.

@urbanserj

Thank you for reporting a bug, @knutin!

Segfaults were caused by not returning e->iolist in enc_yield. I've added a patch that fixed it. In order to avoid this type of errors we must return each explotable value of ERL_NIF_TERM type on continuation.

@davisp
Owner

Ah, good find!

@lpgauth

Will this make into master? 0.8.6 :)

@devinus

I see this as a huge problem in jiffy right now. The database driver we're using uses jiffy to decode and encode documents, and anything slightly larger is going to take longer than 1 millisecond.

@davisp
Owner

@devinus Yeah, I need to get on this. Apologies to all for letting it slip as long as I have. I'm on a business trip right now but I'll try and find time to focus on it when I'm trying to kill time.

@jhs jhs referenced this pull request from a commit in jhs/build-couchdb
@jhs jhs Upgrade to Erlang/OTP R16B03-1
This is due to Russell Branca's notes, attached below.

There has been some discussion on what versions of Erlang CouchDB
should support, and what versions of Erlang are detrimental to
use. Sadly there were some pretty substantial problems in the R15 line
and even parts of R16 that are landmines for CouchDB. This post will
describe the current state of things and make some potential
recommendations on approach.

It was discovered by Basho that R15* and R16B are susceptible to
scheduler collapse. There's quite a bit of discussion and
information in several threads [1] [2] [3] [4] [5].

So what is scheduler collapse? Erlang schedulers can be put to sleep
when there is not sufficient work to occupy all schedulers, which
saves on CPU and power consumption. When the schedulers that are still
running go through enough reductions to pass the work balancing
threshold, they can trigger a rebalance of work that will wake up
sleeping schedulers. The other mechanism for sharing scheduler load is
work stealing. A scheduler that does not have any work to do can
steal work from other schedulers. However a scheduler that has gone
to sleep cannot steal work, it has to be woken up separately.

Now the real problem of scheduler collapse occurs when you take
sleeping schedulers and long running NIFs and BIFs that do not report
an appropriate amount of reductions. When you have NIFs and BIFs that
don't report an appropriate amount of reductions, you can get into a
situation where a long running function call will only show up as
taking one reduction, and never hit the work balance threshold,
causing that scheduler to be blocked during the operation and no
additional schedulers getting woken up.

I keep mentioning "NIFs and BIFs" because it's important to note that
it is _not_ just user defined NIFs that are problematic, but also a
number of Erlang BIFs that don't properly report reductions.
Particularly relevant to CouchDB are the BIFs `term_to_binary` and
`binary_to_term` which do _not_ behave properly, and each report a
single reduction count, regardless of the size of the value passed to
them. Given that every write CouchDB makes goes through
`term_to_binary`, this is definitely not good.

This problem is systemic to all versions of R15 and R16B. In R16B01,
two changes were made to alleviate the problem. First, in `OTP-11163`
`term_to_binary` now uses an appropriate amount of reductions and will
yield back to the scheduler. The second important change was the
introduction of the `+sfwi` (Scheduler Forced Wakeup Interval) flag
[6] which allows you to specify a time interval for a new watchdog
process to check scheduler run queues and wake up sleeping schedulers
if need be. These two changes help significantly, although from what I
understand, they do not fully eliminate scheduler collapse.

*NOTE*: the `+sfwi` is _not_ enabled by default, you must specify a
greater than zero time interval to enable this. *WE NEED TO ENABLE
THIS SETTING.* We should figure out a way to conditionally add this
to vm.args or some such.

On a side note, Basho runs R15B01 because they backported the `+sfwi`
feature to R15B01 [7] [8]. They recommend running with `+sfwi 500` for
a 500ms interval. It might be worth testing out different values, but
500 seems like a good starting point. For Riak 2.0, they will be
building against R16B03-1 and 17.0 as their set of patches to R16B02
landed in R16B03-1 [9] [10].

So R16B01 sorted out the scheduler collapse issues, but unfortunately
it also broke monitors, which immediately disqualifies this release as
something we should recommend to users. The issues was fixed in
`OTP-11225` in R16B02.

I don't know of any catastrophic problems on the order of those
described above in either of these releases. Basho fixed a number of
unrelated bugs in R16B02 [9] [10] that have since landed in R16B03-1,
which indicates we should probably prefer R16B03-1 over R16B02. R16B03
is also disqualified because it broke SSL and `erl_syntax`, resulting
in the patched R16B03-1.

R14B01, R14B03, and R14B04 are known good stable releases of Erlang,
and in my opinion the only known stable releases > R13 that don't
present issues for CouchDB (I think R16B02/R16B03-1 are too new to
declare stable yet). As for R14B02, there are some bad `ets` issues
with that release.

It's worth pointing out that there are two known bugs in R14B01, as
Robert Newson explains:

```
There are two bugs in R14B01 that we do encounter, however. 1) Another
32/64 bit oops causes the vm to attempt to allocate huge amounts of
ram (terabytes, or more) if it ever tries to allocate more than 2gib
of ram at once. When this happens, the vm dies and is restarted. It’s
annoying, but infrequent. 2) Sometimes when closing a file, the
underlying file descriptor is *not* closed, though the erlang process
exits. This is rare but still quite annoying.
```

The 17.0 release brings in a number of interesting changes to help the
scheduler collapse situation. `OTP-11648` improves reduction cost and
yielding of `term_to_binary`. It also utilizes `OTP-11388` which
allows for NIFs and BIFs to have more control over when and how they
are garbage collected (we should do some investigation on the
usefulness of this for NIFs like Jiffy).

The 17.0 release also updates `binary_to_term` in `OTP-11535` to
behave properly with reductions and yielding similar to
`term_to_binary`. This marks the 17.0 release as an important one for
CouchDB as now `term_to_binary` and `binary_to_term` both behave
properly.

One other interesting item introduced in the 17.0 release is the
concept of dirty schedulers [12] [13]. This is an experimental feature
providing CPU and I/O schedulers specifically for NIFs that are known
to take longer that 1ms to run. In general, we want to make sure the
NIFs we use will yield and report reductions properly, but for
situations where that isn't feasible, we may want to look into using
dirty schedulers down the road when it's a non experimental feature.

In my opinion we need to take the Erlang release issues more seriously
than we currently do and provide strong recommendations to users on
what versions of Erlang we support. I suggest we loosely take an
approach similar to Debian, and make three recommendations:

  * OldStable: [R14B01, R14B03, R14B04 (NOTE: _not_ R14B02)]
  * Unstable: [R16B03-1 recommended, R16B02 acceptable]
  * Experimental: [17.0]

I'm not suggesting permanently having three Erlang releases
recommended like this, but it currently seems appropriate. I think
long term we should target 17.x as our preferred Erlang release, and
then make a CouchDB 3.0 release that is backwards incompatible with
anything less than 17.0 so that we can switch over to using maps.

The narrowness of the acceptable releases list is going to cause some
problems. Debian Wheezy runs R15B01, which as established above, is
not good to run with unless you have the `+sfwi` patch, and I'm sure
there are many other distros running R15 and R16B or R16B01. I think
it would be useful to users to have a set of packages with a proper
Erlang CouchDB release allowing us to bless specific versions of
Erlang and bundle it together, but I know this idea goes against the
recent change in stance on working with distributions, and I don't
know the ASF stance on this issue well enough to comment on the
legality of it. That said, it does seem like the logical approach
until we get a range of stable releases spread out through the
distros.

We need to make sure that all NIFs we use that could potentially take
longer than 1ms to run properly yield and report reductions. For
Jiffy, there is already a good start on this work [11]. We'll want to
look into what needs to be done for the rest of the NIFs.

There's quite a bit of information here, and plenty more in the
footnotes, so I hope this gives a good overview of the current state
of Erlang releases and helps us to make informed decisions on what
approach to take with Erlang releases.

[1] http://comments.gmane.org/gmane.comp.lang.erlang.bugs/3564

[2] http://erlang.org/pipermail/erlang-questions/2013-April/073490.html

[3] http://erlang.org/pipermail/erlang-questions/2012-October/069503.html

[4] http://erlang.org/pipermail/erlang-questions/2012-October/069585.html

[5] http://permalink.gmane.org/gmane.comp.lang.erlang.bugs/3573

[6] http://erlang.org/pipermail/erlang-patches/2013-June/004109.html

[7] https://gist.github.com/evanmcc/a599f4c6374338ed672e

[8] http://data.story.lu/2013/06/23/riak-1-3-2-released

[9] basho/otp@erlang:maint...OTP_R16B02_basho4

[10] https://groups.google.com/forum/#!topic/nosql-databases/XpFKVeUBdn0

[11] davisp/jiffy#49

[12] erlang/otp@c1c03ae

[13] http://www.erlang.org/doc/man/erl_nif.html#dirty_nifs
849fa7e
@davisp
Owner

Just released 0.10.1 which includes this functionality. Huge thanks to @urbanserj for the work putting together this patch. I ended up moving things around slightly but its all directly based on his work.

@davisp davisp closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Aug 28, 2013
  1. @urbanserj
Commits on Oct 8, 2013
  1. @urbanserj
This page is out of date. Refresh to see the latest.
View
95 c_src/decoder.c
@@ -47,7 +47,6 @@ typedef struct {
jiffy_st* atoms;
ERL_NIF_TERM arg;
- ErlNifBinary bin;
int is_partial;
@@ -59,22 +58,37 @@ typedef struct {
char* st_data;
int st_size;
int st_top;
+
+ int is_resource;
+ size_t reds;
} Decoder;
+
void
-dec_init(Decoder* d, ErlNifEnv* env, ERL_NIF_TERM arg, ErlNifBinary* bin)
+dec_init_bin(Decoder* d, ErlNifEnv* env, ERL_NIF_TERM arg, ErlNifBinary* bin)
+{
+ d->arg = arg;
+
+ d->p = (char*) bin->data;
+ d->u = bin->data;
+ d->len = bin->size;
+
+ d->env = env;
+}
+
+int
+dec_init(Decoder* d, ErlNifEnv* env, ERL_NIF_TERM arg,
+ ERL_NIF_TERM opts, ErlNifBinary* bin)
{
int i;
+ ERL_NIF_TERM val;
d->env = env;
d->atoms = enif_priv_data(env);
- d->arg = arg;
d->is_partial = 0;
- d->p = (char*) bin->data;
- d->u = bin->data;
- d->len = bin->size;
+ dec_init_bin(d, env, arg, bin);
d->i = 0;
d->st_data = (char*) enif_alloc(STACK_SIZE_INC * sizeof(char));
@@ -87,11 +101,26 @@ dec_init(Decoder* d, ErlNifEnv* env, ERL_NIF_TERM arg, ErlNifBinary* bin)
d->st_data[0] = st_value;
d->st_top++;
+
+ d->reds = REDUCTIONS;
+
+ if(!enif_is_list(env, opts)) {
+ return 0;
+ }
+ while(enif_get_list_cell(env, opts, &val, &opts)) {
+ if(!get_reductions(env, val, d->atoms, &d->reds)) {
+ return 0;
+ }
+ }
+
+ d->is_resource = 0;
+ return 1;
}
void
-dec_destroy(Decoder* d)
+dec_destroy(ErlNifEnv* env, void* dec)
{
+ Decoder* d = dec;
if(d->st_data != NULL) {
enif_free(d->st_data);
}
@@ -604,26 +633,60 @@ make_array(ErlNifEnv* env, ERL_NIF_TERM list)
return ret;
}
+static ERL_NIF_TERM
+dec_yield(Decoder* d, ERL_NIF_TERM objs, ERL_NIF_TERM curr)
+{
+ Decoder* dec = d;
+ if(!d->is_resource) {
+ dec = enif_alloc_resource(d->atoms->res_decoder, sizeof(Decoder));
+ *dec = *d;
+ dec->is_resource = 1;
+ }
+ ERL_NIF_TERM val = enif_make_resource(d->env, dec);
+ return enif_make_tuple4(d->env,
+ d->atoms->atom_partial, val, objs, curr);
+}
+
ERL_NIF_TERM
decode(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
{
Decoder dec;
Decoder* d = &dec;
-
ErlNifBinary bin;
- ERL_NIF_TERM objs = enif_make_list(env, 0);
- ERL_NIF_TERM curr = enif_make_list(env, 0);
+ ERL_NIF_TERM objs;
+ ERL_NIF_TERM curr;
ERL_NIF_TERM val;
ERL_NIF_TERM ret;
- if(argc != 1) {
+ if(argc != 2) {
return enif_make_badarg(env);
} else if(!enif_inspect_binary(env, argv[0], &bin)) {
return enif_make_badarg(env);
}
- dec_init(d, env, argv[0], &bin);
+ int arity;
+ ERL_NIF_TERM* args;
+ if(enif_get_tuple(env, argv[1], &arity, (const ERL_NIF_TERM **) &args)) {
+ jiffy_st *priv = enif_priv_data(env);
+ if(arity != 3 ) {
+ return enif_make_badarg(env);
+ }
+ if(!enif_get_resource(env, args[0], priv->res_decoder, (void **) &d)) {
+ return enif_make_badarg(env);
+ }
+ objs = args[1];
+ curr = args[2];
+ dec_init_bin(d, env, argv[0], &bin);
+ } else {
+ objs = enif_make_list(env, 0);
+ curr = enif_make_list(env, 0);
+ if (!dec_init(d, env, argv[0], argv[1], &bin)) {
+ return enif_make_badarg(env);
+ }
+ }
+
+ size_t processed = d->i;
//fprintf(stderr, "Parsing:\r\n");
while(d->i < bin.size) {
@@ -897,6 +960,11 @@ decode(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
ret = dec_error(d, "invalid_internal_state");
goto done;
}
+ if(dec_curr(d) != st_done) {
+ if(jiffy_consume_timeslice(env, d->reds, d->i, &processed)) {
+ return dec_yield(d, objs, curr);
+ }
+ }
}
if(dec_curr(d) != st_done) {
@@ -908,7 +976,8 @@ decode(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
}
done:
- dec_destroy(d);
+ jiffy_consume_timeslice(env, d->reds, d->i, &processed);
+ dec_destroy(env, d);
return ret;
}
View
63 c_src/encoder.c
@@ -36,12 +36,16 @@ typedef struct {
int iolen;
ERL_NIF_TERM iolist;
+ size_t iosize;
ErlNifBinary* curr;
char* p;
unsigned char* u;
size_t i;
+
+ int is_resource;
+ size_t reds;
} Encoder;
@@ -61,7 +65,7 @@ static char* shifts[NUM_SHIFTS] = {
int
-enc_init(Encoder* e, ErlNifEnv* env, ERL_NIF_TERM opts, ErlNifBinary* bin)
+enc_init(Encoder* e, ErlNifEnv* env, ERL_NIF_TERM opts)
{
ERL_NIF_TERM val;
@@ -71,6 +75,7 @@ enc_init(Encoder* e, ErlNifEnv* env, ERL_NIF_TERM opts, ErlNifBinary* bin)
e->pretty = 0;
e->shiftcnt = 0;
e->count = 0;
+ e->reds = REDUCTIONS;
if(!enif_is_list(env, opts)) {
return 0;
@@ -83,15 +88,16 @@ enc_init(Encoder* e, ErlNifEnv* env, ERL_NIF_TERM opts, ErlNifBinary* bin)
e->pretty = 1;
} else if(enif_compare(val, e->atoms->atom_force_utf8) == 0) {
// Ignore, handled in Erlang
- } else {
+ } else if(!get_reductions(env, val, e->atoms, &e->reds)) {
return 0;
}
}
e->iolen = 0;
e->iolist = enif_make_list(env, 0);
- e->curr = bin;
- if(!enif_alloc_binary(BIN_INC_SIZE, e->curr)) {
+ e->iosize = 0;
+ e->curr = enif_alloc(sizeof(ErlNifBinary));
+ if(!e->curr || !enif_alloc_binary(BIN_INC_SIZE, e->curr)) {
return 0;
}
@@ -101,15 +107,19 @@ enc_init(Encoder* e, ErlNifEnv* env, ERL_NIF_TERM opts, ErlNifBinary* bin)
e->u = (unsigned char*) e->curr->data;
e->i = 0;
+ e->is_resource = 0;
+
return 1;
}
void
-enc_destroy(Encoder* e)
+enc_destroy(ErlNifEnv* env, void* enc)
{
+ Encoder *e = enc;
if(e->curr != NULL) {
enif_release_binary(e->curr);
}
+ enif_free(e->curr);
}
ERL_NIF_TERM
@@ -189,6 +199,7 @@ enc_unknown(Encoder* e, ERL_NIF_TERM value)
e->iolist = enif_make_list_cell(e->env, value, e->iolist);
e->iolen++;
+ e->iosize += e->i;
// Reinitialize our binary for the next buffer.
e->curr = bin;
@@ -493,13 +504,26 @@ enc_comma(Encoder* e)
return 1;
}
+static ERL_NIF_TERM
+enc_yield(Encoder* e, ERL_NIF_TERM stack)
+{
+ Encoder* enc = e;
+ if(!e->is_resource) {
+ enc = enif_alloc_resource(e->atoms->res_encoder, sizeof(Encoder));
+ *enc = *e;
+ enc->is_resource = 1;
+ }
+ ERL_NIF_TERM val = enif_make_resource(e->env, enc);
+ return enif_make_tuple4(e->env, e->atoms->atom_partial, val, stack, e->iolist);
+}
+
+
ERL_NIF_TERM
encode(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
{
Encoder enc;
Encoder* e = &enc;
- ErlNifBinary bin;
ERL_NIF_TERM ret;
ERL_NIF_TERM stack;
@@ -514,11 +538,26 @@ encode(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
return enif_make_badarg(env);
}
- if(!enc_init(e, env, argv[1], &bin)) {
- return enif_make_badarg(env);
+ jiffy_st *priv = enif_priv_data(env);
+ if(!enif_get_resource(env, argv[0], priv->res_encoder, (void **) &e)) {
+ if(!enc_init(e, env, argv[1])) {
+ return enif_make_badarg(env);
+ }
+ stack = enif_make_list(env, 1, argv[0]);
+ } else {
+ int arity;
+ ERL_NIF_TERM* args;
+ if(!enif_get_tuple(env, argv[1], &arity, (const ERL_NIF_TERM **) &args)) {
+ return enif_make_badarg(env);
+ } else if(arity != 2) {
+ return enif_make_badarg(env);
+ }
+ stack = args[0];
+ e->iolist = args[1];
+ e->env = env;
}
- stack = enif_make_list(env, 1, argv[0]);
+ size_t processed = e->iosize + e->i;
while(!enif_is_empty_list(env, stack)) {
if(!enif_get_list_cell(env, stack, &curr, &stack)) {
@@ -690,6 +729,9 @@ encode(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
goto done;
}
}
+ if(jiffy_consume_timeslice(env, e->reds, e->iosize + e->i, &processed)) {
+ return enc_yield(e, stack);
+ }
}
if(!enc_done(e, &item)) {
@@ -704,6 +746,7 @@ encode(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
}
done:
- enc_destroy(e);
+ jiffy_consume_timeslice(env, e->reds, e->i, &processed);
+ enc_destroy(env, e);
return ret;
}
View
8 c_src/jiffy.c
@@ -23,11 +23,17 @@ load(ErlNifEnv* env, void** priv, ERL_NIF_TERM info)
st->atom_uescape = make_atom(env, "uescape");
st->atom_pretty = make_atom(env, "pretty");
st->atom_force_utf8 = make_atom(env, "force_utf8");
+ st->atom_reductions = make_atom(env, "reductions");
// Markers used in encoding
st->ref_object = make_atom(env, "$object_ref$");
st->ref_array = make_atom(env, "$array_ref$");
+ st->res_encoder = enif_open_resource_type(env, "jiffy", "encoder",
+ enc_destroy, ERL_NIF_RT_CREATE, NULL);
+ st->res_decoder = enif_open_resource_type(env, "jiffy", "decoder",
+ dec_destroy, ERL_NIF_RT_CREATE, NULL);
+
*priv = (void*) st;
return 0;
@@ -54,7 +60,7 @@ unload(ErlNifEnv* env, void* priv)
static ErlNifFunc funcs[] =
{
- {"nif_decode", 1, decode},
+ {"nif_decode", 2, decode},
{"nif_encode", 2, encode}
};
View
10 c_src/jiffy.h
@@ -6,6 +6,8 @@
#include "erl_nif.h"
+#define REDUCTIONS 1000
+
typedef struct {
ERL_NIF_TERM atom_ok;
ERL_NIF_TERM atom_error;
@@ -19,17 +21,25 @@ typedef struct {
ERL_NIF_TERM atom_uescape;
ERL_NIF_TERM atom_pretty;
ERL_NIF_TERM atom_force_utf8;
+ ERL_NIF_TERM atom_reductions;
ERL_NIF_TERM ref_object;
ERL_NIF_TERM ref_array;
+
+ ErlNifResourceType *res_encoder;
+ ErlNifResourceType *res_decoder;
} jiffy_st;
ERL_NIF_TERM make_atom(ErlNifEnv* env, const char* name);
ERL_NIF_TERM make_ok(jiffy_st* st, ErlNifEnv* env, ERL_NIF_TERM data);
ERL_NIF_TERM make_error(jiffy_st* st, ErlNifEnv* env, const char* error);
+int get_reductions(ErlNifEnv *env, ERL_NIF_TERM term, jiffy_st* st, size_t* val);
+int jiffy_consume_timeslice(ErlNifEnv *env, size_t reds, size_t cur, size_t* proc);
ERL_NIF_TERM decode(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]);
ERL_NIF_TERM encode(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]);
+void enc_destroy(ErlNifEnv* env, void* e);
+void dec_destroy(ErlNifEnv* env, void* d);
int int_from_hex(const unsigned char* p);
int int_to_hex(int val, char* p);
View
29 c_src/util.c
@@ -24,3 +24,32 @@ make_error(jiffy_st* st, ErlNifEnv* env, const char* error)
{
return enif_make_tuple2(env, st->atom_error, make_atom(env, error));
}
+
+int
+get_reductions(ErlNifEnv *env, ERL_NIF_TERM term, jiffy_st* st, size_t* val)
+{
+ int arity;
+ const ERL_NIF_TERM *tuple;
+
+ return enif_get_tuple(env, term, &arity, &tuple) &&
+ arity == 2 &&
+ enif_compare(tuple[0], st->atom_reductions) == 0 &&
+ enif_get_int(env, tuple[1], (int*) val) &&
+ val >= 0;
+}
+
+int
+jiffy_consume_timeslice(ErlNifEnv *env, size_t reds, size_t cur, size_t* proc) {
+#if ERL_NIF_MAJOR_VERSION >= 2 && ERL_NIF_MINOR_VERSION >= 4
+#define PERCENTS 10
+ if (reds > 0 && cur - *proc >= reds / PERCENTS) {
+ int percents = 100 * (cur - *proc) / reds;
+ percents = (percents < 1) ? 1 : (
+ (percents > 100) ? 100 :
+ percents );
+ *proc = cur;
+ return enif_consume_timeslice(env, percents);
+ }
+#endif
+ return 0;
+}
View
34 src/jiffy.erl
@@ -2,13 +2,17 @@
% See the LICENSE file for more information.
-module(jiffy).
--export([decode/1, encode/1, encode/2]).
+-export([decode/1, decode/2, encode/1, encode/2]).
-define(NOT_LOADED, not_loaded(?LINE)).
-on_load(init/0).
-decode(Data) when is_binary(Data) ->
- case nif_decode(Data) of
+decode(Data) ->
+ decode(Data, []).
+
+
+decode(Data, Options) when is_binary(Data) ->
+ case nif_decode_loop(Data, Options) of
{error, _} = Error ->
throw(Error);
{partial, EJson} ->
@@ -16,8 +20,8 @@ decode(Data) when is_binary(Data) ->
EJson ->
EJson
end;
-decode(Data) when is_list(Data) ->
- decode(iolist_to_binary(Data)).
+decode(Data, Options) when is_list(Data) ->
+ decode(iolist_to_binary(Data), Options).
encode(Data) ->
@@ -26,7 +30,7 @@ encode(Data) ->
encode(Data, Options) ->
ForceUTF8 = lists:member(force_utf8, Options),
- case nif_encode(Data, Options) of
+ case nif_encode_loop(Data, Options) of
{error, invalid_string} when ForceUTF8 == true ->
FixedData = jiffy_utf8:fix(Data),
encode(FixedData, Options -- [force_utf8]);
@@ -99,9 +103,25 @@ init() ->
not_loaded(Line) ->
erlang:nif_error({not_loaded, [{module, ?MODULE}, {line, Line}]}).
-nif_decode(_Data) ->
+nif_decode_loop(Data, Options) ->
+ case nif_decode(Data, Options) of
+ {partial, Decoder, Objs, Curr} ->
+ nif_decode_loop(Data, {Decoder, Objs, Curr});
+ Other ->
+ Other
+ end.
+
+nif_decode(_Data, _Options) ->
?NOT_LOADED.
+nif_encode_loop(Data, Options) ->
+ case nif_encode(Data, Options) of
+ {partial, Encoder, Stack, IoList} ->
+ nif_encode_loop(Encoder, {Stack, IoList});
+ Other ->
+ Other
+ end.
+
nif_encode(_Data, _Options) ->
?NOT_LOADED.
Something went wrong with that request. Please try again.