KV iterator questions.

[jdgordon]

Hi,

A few quick questions around the use of the KV iterator.

1. After the iterator has been initialised and walked to a certain item, is the iterator still safe to use after and KV's have been removed /added/modified?
2. Is there any ordering guarentees with the iterators? I assume no, and that it simply walks the blob structure? presumably the order is mostly by modification time.

Lastly, do you have any design documentation of how the KV database is stored in flash and how you deal with flash wear levelling?

Thanks
Jonathan

[armink]

Hi, Jonathan

The iterator is NOT thread-safe when KV's have been removed/added/modified. Please add the db_lock when using iterator. like this:

static void db_lock(fdb_db_t db)
{
    pthread_mutex_lock((pthread_mutex_t *)db->user_data);
}

static void db_unlock(fdb_db_t db)
{
    pthread_mutex_unlock((pthread_mutex_t *)db->user_data);
}

int main(void)
{
        ........
        fdb_kvdb_control(&kvdb, FDB_KVDB_CTRL_SET_LOCK, (void *)db_lock);
        fdb_kvdb_control(&kvdb, FDB_KVDB_CTRL_SET_UNLOCK, (void *)db_unlock);
        ........
        result = fdb_kvdb_init(&kvdb, "env", "fdb_kvdb1", &default_kv, &kv_locker);
}

    db_lock();
    while (fdb_kv_iterate(&data->kvdb, &iterator))
    {
        // LOG_DEBUG("enter record_from_database ...。\n");
        cur_kv = &(iterator.curr_kv);
        data_size = (size_t)cur_kv->value_len;
        memset(data_buf, 0, 128);
        fdb_blob_read((fdb_db_t)&data->kvdb, fdb_kv_to_blob(cur_kv, fdb_blob_make(&blob, data_buf, data_size)));
/*
        * balabala do what ever you like with blob...
        */
        // FDB_INFO("get kv  key--%s value---%s   value_size is %d  \n", cur_kv->name, data_buf, (int)data_size);
        printf("get kv  key--%s value---%s   value_size is %d \n", cur_kv->name, data_buf, (int)data_size);
    }
    db_unlock();

Here are some early design documents, but it's Chinese, I hope it will help you. Please see https://github.com/armink/EasyFlash/blob/master/docs/zh/design.md

[jdgordon]

Thank you!

I'm not concerned about thread safety. I have an (single threaded) application which wants to use the KV db as a simple FIFO. The majority of the access pattern will be dozens of inserts over a long time period, followed by a shorter burst of "get a blob"/"remove blob".
During that read period there is less likely to have any writes, so I was thinking of storing the iterator and reusing it on each request for the next blob. Do I need to reinit the iterator on each removal as well as writes?

I'll try to draw out what I'm trying to describe. (each [x] is a blob)

db looks like -> [a][b][c][d][e][f]

1. We start doing reads, lets say we have 3 reads before any removes or new writes, iter @ [c]
2. we need to remove [b]. At this point is the iter still valid?
3. we add a new[h] while the iter is @ [c], I assume the iter must be reset
4. After some time the db would look like [d][e][f][g][h]

Is using the iterator for this purpose a bad idea? After writing this out I'm thinking it may just be safer to use the key lookup and accept the performance penalty?

And thanks for the link to the documentation. I dont understand chinese but google translate seems to do a good job of translating!

[armink]

If you are using KVDB in single thread mode. The iterator is still safe.

You can try the linux demo https://github.com/armink/FlashDB/tree/master/demos/linux , and run iterator in main thread.

[jdgordon]

Thanks!