retrieving all keys in db with a cursor causes subsequent cursors to not see some keys

[lebedov]

When I run the following, the second printf displays 483 rather than 500. Removing the first while() loop that retrieves all of the keys causes the second printf to display 500. Is this expected?

#include "unqlite.h"
#include <stdio.h>

#define N       500
#define KEYLEN  8
#define DATALEN 100

int main(void) {

  unqlite *db;
  unqlite_kv_cursor *cursor;
  char key[KEYLEN];
  char data[DATALEN];
  char key_array[N][KEYLEN];
  int ikey;
  unqlite_int64 idata;
  int i, j, rc;

  unqlite_open(&db, "test.db", UNQLITE_OPEN_CREATE);

  for (i=0; i<N; i++) {
    sprintf(key, "key_%i", i);
    for (j=0; j<DATALEN; j++)
      data[j] = 'x';
    unqlite_kv_store(db, key, sizeof(key), data, sizeof(data));
  }

  unqlite_close(db);

  unqlite_open(&db, "test.db", UNQLITE_OPEN_CREATE);

  unqlite_kv_cursor_init(db, &cursor);
  unqlite_kv_cursor_reset(cursor);

  j = 0;
  while (unqlite_kv_cursor_valid_entry(cursor)) {
    unqlite_kv_cursor_key(cursor, &key_array[j], &ikey);
    unqlite_kv_cursor_next_entry(cursor);
    j += 1;
  }
  unqlite_kv_cursor_release(db, cursor);
  printf("%i\n", j);

  unqlite_kv_cursor_init(db, &cursor);
  unqlite_kv_cursor_reset(cursor);

  j = 0;
  while (unqlite_kv_cursor_valid_entry(cursor)) {
    unqlite_kv_cursor_key(cursor, &key, &ikey);
    unqlite_kv_fetch(db, key, sizeof(key), &data, &idata);
    unqlite_kv_cursor_next_entry(cursor);
    j += 1;
  }
  unqlite_kv_cursor_release(db, cursor);
  printf("%i\n", j);
}

I'm building the above against unqlite 1.1.6 with Apple LLVM 7.3.0 on MacOS 10.11.6.

[coleifer]

I noticed this as well when I tried with 1000 records that 24 were missing (twice what @lebedov noted when testing 500 keys). Interestingly, the same keys seemed to be missing each time:

key_270, key_271, key_371, key_390, key_416, key_417, key_490, key_516, key_517, key_531, key_533, key_534, key_535, key_536, key_921, key_922, key_923, key_933, key_940, key_941, key_950, key_951, key_960, key_970

[coleifer]

I had been kind of hoping this was a bug in my binding rather than in UnQLite, but it appears that @lebedov reproduced it using the C APIs. This is extremely disconcerting: a key/value database may mysteriously not return keys that are actually present when iterating with a cursor.

I hope you are able to follow-up on this issue soon, as I would like to warn any users of my Python bindings about this bug and potentially direct them to a more suitable database if necessary.

[coleifer]

Seems related to #3 and discussed also here: https://unqlite.org/forum/thread.php?file=__8

[symisc]

Be sure to test against the last patched version of the library. We have faced this issue and a patch was submitted to the Github repo.

Sent from my android device.

-----Original Message-----
From: Charles Leifer notifications@github.com
To: symisc/unqlite unqlite@noreply.github.com
Sent: Wed, 17 Aug 2016 22:19
Subject: Re: [symisc/unqlite] retrieving all keys in db with a cursor causes subsequent cursors to not see some keys (#41)

I noticed this as well when I tried with 1000 records that 24 were missing (twice what @lebedov noted when testing 500 keys). Interestingly, the same keys seemed to be missing each time:

key_270, key_271, key_371, key_390, key_416, key_417, key_490, key_516, key_517, key_531, key_533, key_534, key_535, key_536, key_921, key_922, key_923, key_933, key_940, key_941, key_950, key_951, key_960, key_970

You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
#41 (comment)

[lebedov]

The problem still occurs when I build the test program against the latest revision in master (b87eecb).

[coleifer]

Same here, seeing it fail with 1.1.6 amalgamation as well. In-memory DBs seem unaffected, but DBs on the filesystem are the one's that have the issue.

[symisc]

Could you send your test case

Sent from my android device.

-----Original Message-----
From: "Lev E. Givon" notifications@github.com
To: symisc/unqlite unqlite@noreply.github.com
Cc: Mrad Chems Eddine chm@symisc.net, Comment comment@noreply.github.com
Sent: Wed, 17 Aug 2016 22:36
Subject: Re: [symisc/unqlite] retrieving all keys in db with a cursor causes subsequent cursors to not see some keys (#41)

The problem still occurs when I build the test program against the latest revision in master (b87eecb).

You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#41 (comment)

[coleifer]

It's stupid simple.

Create a database on the filesystem with 500 keys, 8byte keys, 100byte values (just to replicate what @lebedov noticed). Then close and re-open the database.

After re-opening the database, use the cursor APIs to iterate through twice in a row. The first time you'll see 500...the second time you'll see somewhat less. I'm seeing 490 at the moment.

Here's the Python test-case I just added (which is failing):

coleifer/unqlite-python@110e967...master

[coleifer]

Interestingly, with Vedis, after inserting 500 records I get an error on the very first iteration :/ -- even after updating my sources.

[coleifer]

Apparently this bug has been around since 2014? https://unqlite.org/forum/thread.php?file=__8&page=1#2hymzp2f2kfv

[symisc]

Yes, since they share the same pager & transaction manager. I really suspect that layer.
Anyway, I have to reproduce this case upon back to the office next Monday.
Mind if you could send me the test code?

Sent from my android device.

-----Original Message-----
From: Charles Leifer notifications@github.com
To: symisc/unqlite unqlite@noreply.github.com
Cc: Mrad Chems Eddine chm@symisc.net, Comment comment@noreply.github.com
Sent: Wed, 17 Aug 2016 22:58
Subject: Re: [symisc/unqlite] retrieving all keys in db with a cursor causes subsequent cursors to not see some keys (#41)

Interestingly, with Vedis, after inserting 500 records I get an error on the very first iteration :/ -- even after updating my sources.

You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#41 (comment)

[coleifer]

Well, there's #41 (comment) which @lebedov was kind enough to share. And I've linked you to the test-case I checked into the Python bindings I wrote here.

Here's the pseudocode:

create db UnQLite('some_file')
value = 500 bytes of data
for i in 0 to 499
   set 8-byte key based on "i" to a constant 500 byte value
close db
reopen db
using cursor, iterate through db, counting records. Should be 500.
release cursor
using a new cursor, iterate through db, counting records. Will be less than 500.

[Yuras]

@lebedov Are you using the amalgamation from master? AFAIK it doesn't contain the fix for #3, try with sources in src dir. Also it could be an instance of #5 which is not fixed in master, you may want to try the fix I proposed for it.

[lebedov]

Yes - I was using the amalgamated unqlite.c file. Building the test program directly against the code in src appears to make the problem go away.

[symisc]

Yep, I'll merge all of that once back to the office

On 18-Aug-16 15:37, Lev E. Givon wrote:

Yes - I was using the amalgamated |unqlite.c| file. Building the test
program directly against the code in |src| appears to make the problem
go away.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#41 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEZu8N2uEbminp2FSosXrWFwmNvzMiBcks5qhG4dgaJpZM4Jmnmq.

Mrad - Symisc Systems, Suarl, http://symisc.net

[coleifer]

It'd be nice if there was a Makefile that could be used to generate the amalgamation.

[mxmauro]

Hi @symisc You said this issue is fixed but code is latest commit is from several months ago. Is there another repository? Thanks.

[symisc]

Source Code (Amalgamation & Github) Update with the 1.1.7 release which fixes all the know bugs.

https://unqlite.org/downloads.html