TABLE_PAT_KEY with Int64 key creates broken table

[kou]

Here are Groonga commands that reproduces this problem:

table_create pat TABLE_PAT_KEY Int64

load --table pat
[
{"_key": 0},
{"_key": 1}
]

delete pat --key 1

load --table pat
[
{"_key": 3}
]

select pat
# -> [[0,1444229153.27227,0.00015711784362793],[[[2],[["_id","UInt32"],["_key","Int64"]],[1,0],[1,0]]]]

The last select reports duplicated records.

[s-yata]

I can reproduce this problem on my environment, but grn_ts reports a different result as follows:

> select pat
[[0,1444267858.98347,7.41481781005859e-05],[[[2],[["_id","UInt32"],["_key","Int64"]],[1,0],[1,0]]]]

> select pat --filter '?true'
[[0,1444267876.78139,8.82148742675781e-05],[[[2],[["_id","UInt32"],["_key","Int64"]],[1,0],[3,3]]]]

--filter 'all_records()' reports the same as grn_ts.

> select pat --filter 'all_records()'
[[0,1444267898.69612,0.0135645866394043],[[[2],[["_id","UInt32"],["_key","Int64"]],[1,0],[3,3]]]]

In addition, --filter 'true' reports another result.

> select pat --filter 'true'
[[0,1444268039.73808,0.00961446762084961],[[[1],[["_id","UInt32"],["_key","Int64"]],[1,0]]]]

I'm not sure why these report different results.

[kou]

My conjecture:

• select pat: It uses cursor by key.
• select pat --filter '?true': It uses cursor by ID.
• select pat --filter 'all_records()': It uses cursor by ID.
• select pat --filter 'true': It uses cursor by key. And it reduces duplicated entries by putting records to grn_hash.

[s-yata]

The duplication error reproduces if the key type is signed integer (IntXX).
It does not reproduce for unsigned integers (UIntXX) and Float.

> table_create pat TABLE_PAT_KEY UInt64
> # Execute the same commands...
> select pat
[[0,1444403895.67027,0.000429868698120117],[[[2],[["_id","UInt32"],["_key","UInt64"]],[1,0],[3,3]]]]

[s-yata]

The last comment is wrong.
The duplication error reproduces if the key type is not signed integer (IntXX) as follows:

table_create pat TABLE_PAT_KEY UInt8
load --table pat
[ {"_key": 128}, {"_key": 129} ]
delete pat --key 129
load --table pat
[ {"_key": 131} ]
select pat
# -> [[0,1444407219.58947,0.000639200210571289],[[[2],[["_id","UInt32"],["_key","UInt8"]],[1,128],[1,128]]]]

The following is another example.

table_create pat TABLE_PAT_KEY ShortText
load --table pat
[ {"_key": "あ0"}, {"_key": "あ1"} ]
delete pat --key "あ1"
load --table pat
[ {"_key": "あ3"} ]
select pat
# -> [[0,1444407344.34656,0.000884532928466797],[[[2],[["_id","UInt32"],["_key","ShortText"]],[1,"あ0"],[1,"あ0"]]]]

The common point is that the MSB of the first byte is 1 and the keys inserted by the first load differs at the MLB of the last byte.

[s-yata]

I've confirmed that grn_pat_delete() breaks grn_pat.
Also, grn_pat gets broken even if the MSB of the first byte is 0.

After deletion, nodes of the grn_patare as follows:

id	check	lr[0]	lr[1]
0	0	0	1
1	0	1	1

id	check	lr[0]	lr[1]
0	0	0	2
2	0	2	2

Please note that "broken" means that grn_pat_cursor returns the same ID more than once.

[s-yata]

• Related commits
  • bae2ed1
  • f69eddc
• TODO
  • Add a command test or unit test for this problem.
  • Test whether or not this problem reproduces with an older Groonga.

[s-yata]

I've confirmed that this problem reproduces with an old commit bae2ed1 .