Feasibility of modifying to use Amazon DynamoDB in place of filesystem

[DallanQ]

DynamoDB is a simple key-value store. How much work would it be for me to modify JDBM3 to use DynamoDB with key-values of <block-id, block-contents> in place of filesystem calls, so I could use JDBM3 in a multi-user environment?

DynamoDB doesn't supports transactions, but it does support conditional updates. I'm ok with disabling transaction support and caching. But I'm wondering what would happen during a b-tree leaf node split that required the parent be split as well, if the server crashed half-way through the split. Would it leave the b-tree in an unusable state, or would whatever was just added simply be lost?

I'm sure glad to see that you're doing this. I want to be able to store sorted collections in DynamoDB, and I was afraid that I was going to have to implement the b-tree logic myself. I know that DynamoDB supports sorted collections using two-part keys, but there are performance implications, so I'd rather not use them if I can avoid it.

[jankotek]

Hi,

JDBM currently have those layers (from consumer down to raw disk)

1. collection with index (BTree, HTree)
2. instance cache
3. serialization
4. primitive record key-value store (long as key, byte[] as value).
5. disk store (ram, zip, file)

And if I understand correctly, this proposal is about JDBM using alternative implementations at level 4. I think this is very good idea and will definitely implement it.

My idea is: JDBM will export one more interface called RecordManager with methods like this:

long insert(byte[]) //insert new record, returs recid
void delete(long)  //delete record by recid
void update(long,byte[]) //update record with given recid with new data
boolean supportTransaction()
void roolback()
void commit()

User may supply alternative implementation in DBMaker. In this case JDBM will use other db as back-end, but user will still get instance cache, collections, serialization and other honey JDBM provides.

Only problem is that I am not going to write wrapper for DynamoDB (have no access to Amazon WS). Reference wrapper implementation will use LevelDB or Tokyo Cabinet. JDBM will also still support only single transaction per store.

ETA 2 weeks.

[DallanQ]

Thank you for the reply! I'll check back in a couple of weeks.

[jankotek]

Hi,
JDBM now has very basic support for alternative key-value stores. Checkout RecordManager interface to give you some basic understanding. But it is still not finished.

https://github.com/jankotek/JDBM3/blob/master/src/main/java/net/kotek/jdbm/RecordManager.java

[jankotek]

I decided not to implement this. Multiple key-value would complicate design too much. I reverted previous GIT commit

[DallanQ]

ok, thanks for letting me know.