Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Float to list 2 #9

Closed
wants to merge 1 commit into from
Closed

Float to list 2 #9

wants to merge 1 commit into from

Conversation

saleyn
Copy link
Contributor

@saleyn saleyn commented Jan 17, 2011

This patch implements a float_to_list/2 BIF that solves a problem of float_to_list/1 not allowing to specify
the number of digits after the decimal point when formatting floats. Additionally this new BIF is optimized to run 5-10x faster than float_to_list/1 for floats under 2^52.

float_to_list(Float, Options) -> string()

Float = float()
Options = [Option]
Option = {precision, Precision::integer()} | compact

Text representation of a float formatted using given options

Returns a string which corresponds to the text
representation of Float using fixed decimal point formatting.
When precision option is specified
the returned value will contain at most Precision number of
digits past the decimal point. When compact option is provided
the trailing zeros at the end of the list are truncated.

The simple test module included below illustrates the performance difference.

  1> test:test().
  float_to_list(123.400000)                   = {1.14305,
                                                 "1.23400000000000005684e+02"}
  float_to_list(123.400000, [])               = {0.124313,"123.4000"}
  float_to_list(123.400000, [{precision, 4}]) = {0.130213,"123.4000"}
  float_to_list(123.400000, 4)                = {0.125899,"123.4000"}
  ok

Test module:

-module(test).
-export([test/0, test/1]).

test() ->
    test(123.4).
test(N) ->
    Self = self(),
    spawn(fun() -> tc(Self, 1000000, fun() -> float_to_list(N) end) end),
    receive
    Msg0 -> io:format("float_to_list(~f)                   = ~p\n", [N, Msg0])
    end,
    spawn(fun() -> tc(Self, 1000000, fun() -> float_to_list(N, []) end) end),
    receive
    Msg1 -> io:format("float_to_list(~f, [])               = ~p\n", [N, Msg1])
    end,
    spawn(fun() -> tc(Self, 1000000, fun() -> float_to_list(N, [{precision, 4}]) end) end),
    receive
    Msg2 -> io:format("float_to_list(~f, [{precision, 4}]) = ~p\n", [N, Msg2])
    end,
    spawn(fun() -> tc(Self, 1000000, fun() -> float_to_list(N, 4) end) end),
    receive
    Msg3 -> io:format("float_to_list(~f, 4)                = ~p\n", [N, Msg3])
    end.


tc(Pid, N, F) when N > 0 ->
    loop(Pid, N, N, F, erlang:now()).

loop(Pid, 1, N, F, Time1) ->
    Res = F(),
    Int = timer:now_diff(erlang:now(), Time1),
    Pid ! {Int / N, Res};
loop(Pid, N, X, F, Time1) ->
    F(),
    loop(Pid, N-1, X, F, Time1).

pekadan pushed a commit to pekadan/otp that referenced this pull request Aug 20, 2012
pekadan pushed a commit to pekadan/otp that referenced this pull request Nov 14, 2012
fdmanana added a commit to fdmanana/otp that referenced this pull request Dec 28, 2012
When using the async thread pool and compressed files, when
an efile driver port instance is shutdown, the efile_drv
stop callback closes the file descriptor (a gzFile instance
actually) - this is dangerous if at the same time there's
an async thread performing an operation against the file,
for example calling invoke_read(), which can result in a
segmentation fault, or calling invoke_close() which double
closes the gzFile and this in turn causes 2 consecutive calls
to driver_free() against same gzFile instance (resulting in
later unexpected crashes in erl_bestfit_alloc.c for example).

The following test program makes the emulator crash when using
the async thread pool:

-module(t2).

-export([t/1]).

t(N) ->
    file:delete("foo.bar"),
    % Use of 'compressed' option, for creating/writing the file,
    % is irrelevant. It only matters when opening it later for
    % reads - a non-compressed file open with the 'compressed'
    % option goes through an internal gzFile handle (instead of
    % a plain integer fd), just like a compressed file.
    %{ok, Fd} = file:open("foo.bar", [raw, write, binary]),
    {ok, Fd} = file:open("foo.bar", [raw, write, binary, compressed]),
    ok = file:write(Fd, <<"qwerty">>),
    ok = file:close(Fd),
    Pid = spawn_link(fun() ->
       process_flag(trap_exit, true),
       loop(N)
    end),
    Ref = erlang:monitor(process, Pid),
    receive
    {'DOWN', Ref, _, _, _} ->
         ok
    end.

loop(0) ->
    ok;
loop(N) ->
    Server = self(),
    Pid = spawn(fun() ->
        {ok, Fd} = file:open("foo.bar", [read, raw, binary, compressed]),
        Server ! continue,
        % Comment the file:read/2 call to make the file:close/1 call much
        % more likely to crash or end up causing efile_drv to close twice
        % the fd (gzFile), which will make the emulator crash later in the
        % best fit allocator (erl_bestfit_alloc.c).
        _ = file:read(Fd, 5),
        file:close(Fd)
    end),
    receive continue -> ok end,
    exit(Pid, shutdown),
    loop(N - 1).

Running this test when using the async thread pool:

  shell> erl +A 4
  Erlang R15B03 (erts-5.9.3.1) [source] [64-bit] [smp:4:4] [async-threads:4] [hipe] [kernel-poll:false]

  Eshell V5.9.3.1  (abort with ^G)
  1> c(t2).
  {ok,t2}
  2> t2:t(500000).
  Segmentation fault (core dumped)

When not using the async thread pool, there are no issues:

  shell> erl
  Erlang R15B03 (erts-5.9.3.1) [source] [64-bit] [smp:4:4] [async-threads:0] [hipe] [kernel-poll:false]

  Eshell V5.9.3.1  (abort with ^G)
  1> c(t2).
  {ok,t2}
  2> t2:t(500000).
  ok
  3>

An example stack trace when the crash happens because there's
an ongoing read operation is:

  Thread 1 (Thread 0x7f021cf2c700 (LWP 10687)):
  #0  updatewindow (strm=0x2691bf8, out=5) at zlib/inflate.c:338
  erlang#1  0x00000000005a2ba0 in inflate (strm=0x2691bf8, flush=0) at zlib/inflate.c:1141
  erlang#2  0x000000000055c46a in erts_gzread (file=0x2691bf8, buf=0x7f0215b29e80, len=5) at drivers/common/gzio.c:523
  erlang#3  0x00000000005849ef in invoke_read (data=0x26b2228) at drivers/common/efile_drv.c:1114
  erlang#4  0x000000000050adcb in async_main (arg=0x7f021bf5cf40) at beam/erl_async.c:488
  erlang#5  0x00000000005c21a0 in thr_wrapper (vtwd=0x7fff69c6ff10) at pthread/ethread.c:106
  erlang#6  0x00007f021c573e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
  erlang#7  0x00007f021c097cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
  erlang#8  0x0000000000000000 in ?? ()

And when there's an ongoing close operation when the driver
is stopped:

  Thread 1 (Thread 0x7fe5f5654700 (LWP 747)):
  #0  0x0000000000459b64 in bf_unlink_free_block (block=0x10b2a70, allctr=<optimized out>, flags=<optimized out>) at beam/erl_bestfit_alloc.c:792
  erlang#1  bf_unlink_free_block (flags=0, block=0x10b2a70, allctr=0x873380) at beam/erl_bestfit_alloc.c:822
  erlang#2  bf_get_free_block (allctr=0x873380, size=<optimized out>, cand_blk=<optimized out>, cand_size=<optimized out>, flags=0) at beam/erl_bestfit_alloc.c:869
  erlang#3  0x000000000044f0dd in mbc_alloc_block (alcu_flgsp=<synthetic pointer>, blk_szp=<synthetic pointer>, size=200, allctr=0x873380) at beam/erl_alloc_util.c:1198
  erlang#4  mbc_alloc (allctr=0x873380, size=200) at beam/erl_alloc_util.c:1345
  erlang#5  0x000000000045449b in do_erts_alcu_alloc (size=200, extra=0x873380, type=165) at beam/erl_alloc_util.c:3442
  erlang#6  erts_alcu_alloc_thr_pref (type=165, extra=<optimized out>, size=192) at beam/erl_alloc_util.c:3520
  erlang#7  0x000000000055c0bf in gz_open (mode=0x5d98b2 "rb", path=0x1103418 "foo.bar") at drivers/common/gzio.c:164
  erlang#8  erts_gzopen (path=0x1103418 "foo.bar", mode=0x5d98b2 "rb") at drivers/common/gzio.c:307
  erlang#9  0x0000000000584e47 in invoke_open (data=0x1103330) at drivers/common/efile_drv.c:1857
  erlang#10 0x000000000050adcb in async_main (arg=0x7fe5f698af80) at beam/erl_async.c:488
psyeugenic pushed a commit to psyeugenic/otp that referenced this pull request Jan 23, 2013
When using the async thread pool and compressed files, when
an efile driver port instance is shutdown, the efile_drv
stop callback closes the file descriptor (a gzFile instance
actually) - this is dangerous if at the same time there's
an async thread performing an operation against the file,
for example calling invoke_read(), which can result in a
segmentation fault, or calling invoke_close() which double
closes the gzFile and this in turn causes 2 consecutive calls
to driver_free() against same gzFile instance (resulting in
later unexpected crashes in erl_bestfit_alloc.c for example).

The following test program makes the emulator crash when using
the async thread pool:

-module(t2).

-export([t/1]).

t(N) ->
    file:delete("foo.bar"),
    % Use of 'compressed' option, for creating/writing the file,
    % is irrelevant. It only matters when opening it later for
    % reads - a non-compressed file open with the 'compressed'
    % option goes through an internal gzFile handle (instead of
    % a plain integer fd), just like a compressed file.
    %{ok, Fd} = file:open("foo.bar", [raw, write, binary]),
    {ok, Fd} = file:open("foo.bar", [raw, write, binary, compressed]),
    ok = file:write(Fd, <<"qwerty">>),
    ok = file:close(Fd),
    Pid = spawn_link(fun() ->
       process_flag(trap_exit, true),
       loop(N)
    end),
    Ref = erlang:monitor(process, Pid),
    receive
    {'DOWN', Ref, _, _, _} ->
         ok
    end.

loop(0) ->
    ok;
loop(N) ->
    Server = self(),
    Pid = spawn(fun() ->
        {ok, Fd} = file:open("foo.bar", [read, raw, binary, compressed]),
        Server ! continue,
        % Comment the file:read/2 call to make the file:close/1 call much
        % more likely to crash or end up causing efile_drv to close twice
        % the fd (gzFile), which will make the emulator crash later in the
        % best fit allocator (erl_bestfit_alloc.c).
        _ = file:read(Fd, 5),
        file:close(Fd)
    end),
    receive continue -> ok end,
    exit(Pid, shutdown),
    loop(N - 1).

Running this test when using the async thread pool:

  shell> erl +A 4
  Erlang R15B03 (erts-5.9.3.1) [source] [64-bit] [smp:4:4] [async-threads:4] [hipe] [kernel-poll:false]

  Eshell V5.9.3.1  (abort with ^G)
  1> c(t2).
  {ok,t2}
  2> t2:t(500000).
  Segmentation fault (core dumped)

When not using the async thread pool, there are no issues:

  shell> erl
  Erlang R15B03 (erts-5.9.3.1) [source] [64-bit] [smp:4:4] [async-threads:0] [hipe] [kernel-poll:false]

  Eshell V5.9.3.1  (abort with ^G)
  1> c(t2).
  {ok,t2}
  2> t2:t(500000).
  ok
  3>

An example stack trace when the crash happens because there's
an ongoing read operation is:

  Thread 1 (Thread 0x7f021cf2c700 (LWP 10687)):
  #0  updatewindow (strm=0x2691bf8, out=5) at zlib/inflate.c:338
  erlang#1  0x00000000005a2ba0 in inflate (strm=0x2691bf8, flush=0) at zlib/inflate.c:1141
  erlang#2  0x000000000055c46a in erts_gzread (file=0x2691bf8, buf=0x7f0215b29e80, len=5) at drivers/common/gzio.c:523
  erlang#3  0x00000000005849ef in invoke_read (data=0x26b2228) at drivers/common/efile_drv.c:1114
  erlang#4  0x000000000050adcb in async_main (arg=0x7f021bf5cf40) at beam/erl_async.c:488
  erlang#5  0x00000000005c21a0 in thr_wrapper (vtwd=0x7fff69c6ff10) at pthread/ethread.c:106
  erlang#6  0x00007f021c573e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
  erlang#7  0x00007f021c097cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
  erlang#8  0x0000000000000000 in ?? ()

And when there's an ongoing close operation when the driver
is stopped:

  Thread 1 (Thread 0x7fe5f5654700 (LWP 747)):
  #0  0x0000000000459b64 in bf_unlink_free_block (block=0x10b2a70, allctr=<optimized out>, flags=<optimized out>) at beam/erl_bestfit_alloc.c:792
  erlang#1  bf_unlink_free_block (flags=0, block=0x10b2a70, allctr=0x873380) at beam/erl_bestfit_alloc.c:822
  erlang#2  bf_get_free_block (allctr=0x873380, size=<optimized out>, cand_blk=<optimized out>, cand_size=<optimized out>, flags=0) at beam/erl_bestfit_alloc.c:869
  erlang#3  0x000000000044f0dd in mbc_alloc_block (alcu_flgsp=<synthetic pointer>, blk_szp=<synthetic pointer>, size=200, allctr=0x873380) at beam/erl_alloc_util.c:1198
  erlang#4  mbc_alloc (allctr=0x873380, size=200) at beam/erl_alloc_util.c:1345
  erlang#5  0x000000000045449b in do_erts_alcu_alloc (size=200, extra=0x873380, type=165) at beam/erl_alloc_util.c:3442
  erlang#6  erts_alcu_alloc_thr_pref (type=165, extra=<optimized out>, size=192) at beam/erl_alloc_util.c:3520
  erlang#7  0x000000000055c0bf in gz_open (mode=0x5d98b2 "rb", path=0x1103418 "foo.bar") at drivers/common/gzio.c:164
  erlang#8  erts_gzopen (path=0x1103418 "foo.bar", mode=0x5d98b2 "rb") at drivers/common/gzio.c:307
  erlang#9  0x0000000000584e47 in invoke_open (data=0x1103330) at drivers/common/efile_drv.c:1857
  erlang#10 0x000000000050adcb in async_main (arg=0x7fe5f698af80) at beam/erl_async.c:488
pekadan pushed a commit to pekadan/otp that referenced this pull request Mar 24, 2013
garazdawi referenced this pull request in garazdawi/otp Jun 29, 2020
sverker added a commit to sverker/otp that referenced this pull request Dec 7, 2020
and robustify the NIFs against faulty usage.

* Add ctx->is_functional to know if ENGINE_init has been called
  and if ENGINE_finish needs to be called.
* Add cleanup to destructor as fallback if engine_free/finish_nif
  has not been called.
* Set ctx->engine=NULL when freed
  and check it after all calls to enif_get_resource.

Removed memory leaks reported by address-sanitizer:

Test case engine_SUITE:ctrl_cmd_string

Direct leak of 192 byte(s) in 1 object(s) allocated from:
    #0 0x7f955fb64b40 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb40)
    #1 0x7f9556c0accb in CRYPTO_malloc crypto/mem.c:222
    #2 0x7f9556c0acfe in CRYPTO_zalloc crypto/mem.c:230
    #3 0x7f9556bd227e in ENGINE_new crypto/engine/eng_lib.c:34
    erlang#4 0x7f9556bd345c in ENGINE_by_id crypto/engine/eng_list.c:301
    erlang#5 0x7f9556fc1965 in engine_by_id_nif otp/lib/crypto/c_src/engine.c:138
    erlang#6 0x562b780d4e8a in process_main x86_64-unknown-linux-gnu/asan/smp/beam_cold.h:184
    erlang#7 0x562b7810f1eb in sched_thread_func beam/erl_process.c:8486
    erlang#8 0x562b787a1b9b in thr_wrapper pthread/ethread.c:122
    erlang#9 0x7f955f0a26da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
@sverker sverker mentioned this pull request Dec 7, 2020
ferd pushed a commit to ferd/otp that referenced this pull request Sep 20, 2021
* Simple trailing maybes without cond work

* Works in all corner cases ... once

* Make it work more than once

* Refactored and simplified

* Refactor, fold all corner cases
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant