An error occurred in the auto-repair when opening dets

[768645186]

Describe the bug
When a dets file does not execute dets:close() correctly and the dets file is opened again, all or part of the data within the dets is cleared.

To Reproduce
Open a dets file in an erlang node and copy the dets file without executing dets:close(). Using another node to open that dets file, all the data is present normally under erlang version 19, whereas under version 21 or even higher, the data is empty or part of the data is missing.

Expected behavior
Since erlang version 19 is able to repair all data properly, higher versions of erlang should also be able to.

Affected versions
version 21 and hihger

[garazdawi]

Very little has changed in dets since Erlang 19. Can you provide an testcase that works when using erlang 19 and breaks in later versions?

[768645186]

Very little has changed in dets since Erlang 19. Can you provide an testcase that works when using erlang 19 and breaks in later versions?

An erlang node has role_1.dets open, at which point the erlang node may be writing data and is not closed. In the meantime, I backed up this dets table and used otp-19 and otp-21 to open the backed up dets table and the following occurs:
Erlang/OTP 21 [erts-10.3.5.14] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [dtrace]
Eshell V10.3.5.14 (abort with ^G)
1> dets:open_file(role_1, [{file, "./role_1.dets"}, {keypos, 12}, {type, set}]).
dets: file "./role_1.dets" not properly closed, repairing ...
{ok,role_1}
2> dets:info(role_1).
[{type,set},
{keypos,12},
{size,0},
{file_size,5464},
{filename,"./role_1.dets"}]

Erlang/OTP 19 [erts-8.3.5.6] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
Eshell V8.3.5.6 (abort with ^G)
1> dets:open_file(role_1, [{file, "./role_1.dets"}, {keypos, 12}, {type, set}]).
dets: file "./role_1.dets" not properly closed, repairing ...
{ok,role_1}
2> dets:info(role_1).
[{type,set},
{keypos,12},
{size,54},
{file_size,27792856},
{filename,"./role_1.dets"}]

[garazdawi]

Can you reproduce the same behaviour if you create the dets file using 25 and repair it using 27?

[768645186]

Can you reproduce the same behaviour if you create the dets file using 25 and repair it using 27?

I tried to open dets with otp-25 and write data continuously, at some point cp three copies test_1.dets, test_2.dets, test_3.dets. the commands I used were cp -rf test.dets test_1.dets && cp -rf test_1.dets test_2.dets && cp -rf test_1.dets test_3.dets.

Use otp-25 to open test_1.dets, data is cleared
Opened test_2.dets with otp-27, data is empty
Opened test_3.dets with otp-19, data is correct.

Here are my tests.

-module(test).
-behaviour(gen_server).
-export([start_link/0, init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2, code_change/3]).
-export([]).

-record(state, {}).

start_link() ->
	gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).

init([]) ->
    State = #state{},
    dets:open_file(test, [{file, "./test.dets"}, {keypos, 1}, {type, set}]),
    erlang:send_after(100, self(), insert),
    {ok, State}.

handle_call(_Request, _From, State) ->
    {reply, false, State}.

handle_cast(_Msg, State) ->
    {noreply, State}.

handle_info(insert, State) ->
    dets:insert(test, {rand(1, 100), lists:seq(1,10000)}),
    erlang:send_after(100, self(), insert),
    {noreply, State};
handle_info(_Info, State) ->
    {noreply, State}.

terminate(_Reason, _State) ->
    ok.

code_change(_OldVsn, State, _Extra) ->
    {ok, State}.
    rand(Min, Max) ->
    M = Min - 1,
    rand:uniform(Max - M) + M.

[qinghanranran]

I also encountered a similar issue. After some investigation, it seems that the problem started when the FileHeader#fileheader.no_colls in dets_v9:fsck_input was changed from is_integer to is_list, leading to different behaviors.

[qinghanranran]

Can you reproduce the same behaviour if you create the dets file using 25 and repair it using 27?

After modifying the dets_v9:fsck_input/4 function by changing is_list(FileHeader#fileheader.no_colls) to is_integer(FileHeader#fileheader.no_colls) and recompiling, the data is being transformed correctly. However, I'm not sure if this is the sole reason for the resolution of the problem.

[garazdawi]

Can you upload a "broken" file that should get repaired but does not so that we can have a look at it?

[qinghanranran]

Can you upload a "broken" file that should get repaired but does not so that we can have a look at it?

aaa.zip
Of course

[garazdawi]

The repair work for me:

Erlang/OTP 25 [erts-13.2.2.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]

Eshell V13.2.2.1  (abort with ^G)
1> dets:open_file("aaa").
dets: file "aaa" not properly closed, repairing ...
{ok,#Ref<0.3620824307.94896131.156162>}

[qinghanranran]

The repair work for me:

Erlang/OTP 25 [erts-13.2.2.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]

Eshell V13.2.2.1  (abort with ^G)
1> dets:open_file("aaa").
dets: file "aaa" not properly closed, repairing ...
{ok,#Ref<0.3620824307.94896131.156162>}

You can use dets:info see the result, the size is 0

[qinghanranran]

The repair work for me:

Erlang/OTP 25 [erts-13.2.2.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]

Eshell V13.2.2.1  (abort with ^G)
1> dets:open_file("aaa").
dets: file "aaa" not properly closed, repairing ...
{ok,#Ref<0.3620824307.94896131.156162>}

Can you upload a "broken" file that should get repaired but does not so that we can have a look at it?

aaa.zip Of course

The repair work for me:

Erlang/OTP 25 [erts-13.2.2.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]

Eshell V13.2.2.1  (abort with ^G)
1> dets:open_file("aaa").
dets: file "aaa" not properly closed, repairing ...
{ok,#Ref<0.3620824307.94896131.156162>}

If it is correctly processed, the result should be 155 pieces of data.

[garazdawi]

yes of course, my bad. Thanks for the clarification, I'll dig a bit and see what I find.