#scalajs-rx-idb

Indexed Database reactive (Rx) wrapper written in scala.js using monifu, uTest.

• dependency : "com.viagraphs" %%% "scalajs-rx-idb" % "0.0.8-SNAPSHOT"
• Scala.js version : 0.6.5

This is a special version that does not require uPicke. All classes serialised to indexDB must be pure javascript object, ie. the objects and all the fields must recursively inherit js.Any

Primarily it is trying to be :

##type safe

• a key validation type class doesn't let you store keys of unsupported types
• there is an abstraction over CRUD operations allowing seamlessly work with both scala collections and idb key ranges over store or index

##user friendly because

• there is too much mutability and confusion regarding request result value, versioning, transactions and error handling in IndexedDb API
• no living soul wants to spend 3 hours trying to reliably check whether a database exists
• it should prevent lock starvation that I spent literally days to put up with already
• it should supervise transaction boundaries. There are a few edge cases though I haven't covered yet, I asked a question on SO
• Rx based API has a clean contract by definition

##handling asynchrony, resilience and back-pressure the Rx way because

• IndexedDb API imho leads to inevitable callback hell and I couldn't really say when it crashes and why
• it makes it easier to implement new features like profiling
• you get a full control over returned data streams in form of higher-order functions
• thanks to Monifu's back pressure implementation you get a way to asynchronously processing results requested lazily with the possibility to cancel.

In other words, doing complicated stuff with IndexedDb directly is not that easy as one might expect. I came to conclusion that IndexedDb is rather a db engine that is meant to be used by Databases built around it

NOTE

• Just the main operations are tested so far, it's a work in progress, there is no time to test edge cases
• The performance might get a little worse in comparison with direct IDB access
  • But after you spend some time with IDB you'll know that loosing a few milliseconds is always better than lock starvation that might put the entire application down or waste hours of troubleshooting
• This library is suitable for bulk operations rather than requests targeting one record. That's why all methods are passed either Iterable or KeyRange

Struggles

• There was a significant resistance doing abstraction over a javascript application that itself doesn't even have a common interface for IDBObjectStore and IDBIndex even though both being a Store, sharing common interface
• I was fighting scalac a lot
  • Scalac type inference sometimes really surprises (unpleasantly) - I had to combine path dependent types with type classes which is always a bad thing (I'll personally never do it again)
  • Scalac doesn't look into descendants of associated types - which also made my day
  • After I rewrote API to path dependent style to spare user from having to explicitly specify types all the time - I found out that I should have probably rewritten the entire application from scratch because due to these unexpected issues the interface turned out to be a little less self explanatory than I intended it to be, I'd like to rewrite it a little in future

Important API - Store

There is lot to abstract over in regards to querying IDB, especially key autogeneration, key being on value's keypath, KeyRanges, Indexes, operations on Last and First record etc. I could use scala Marcos to generate the API based on DB Schema, but unfortunately I decided not to, there are just 4 methods that basically do everything based on type of input.

//      v - either store Value OR (Key,Value) type     v - type class abstracting over the possibility of key being on value keypath, autogenerated or explicitly specified
def add[I, C[X] <: Iterable[X]](values: C[I])(implicit p: StoreKeyPolicy[I], tx: Tx[C]): Observable[(K,V)]
//         ^ - type constructor of any type that is iterable                  ^ - type class for ad-hoc polymorphism regarding transaction handling 

//       v - type constructor that might be either an Iterable or KeyRange of Keys                           
def get[C[_]](keys: C[K])(implicit e: Tx[C]): Observable[(K,V)]
//                                 ^ - type class allows you to add a custom logic for the request, there is just an evidence for Iterable and KeyRange

//                                                   v - usually an observable of Key Value pairs is returned, delete just completes
def delete[C[_]](keys: C[K])(implicit e: Tx[C]): Observable[Nothing]

// update works similar to add except it supports KeyRange - beware you must supply KeyRange entries
def update[I, C[_]](input: C[I])(implicit p: StoreKeyPolicy[I], e: Tx[C]): Observable[(K,V)]

Examples

• The best place to look at examples is IndexedDbSuite

• Note that the crud operations accept either anything that is Iterable or any com.viagraphs.idb.Store.Key

• working with iterables (check the tests. The code below must be adapted to be purely javascript objects. See test suite.)

val obj1 = Map("x" -> 0) // store values might be anything that upickle manages to serialize
val obj2 = Map("y" -> 1)
val db = IndexedDb( // you may create new db, open, upgrade or recreate existing one
  OpenDb("dbName", db => db.createObjectStore("storeName", lit("autoIncrement" -> true)))
)
val store = db.openStore[Int,Map[String, Int]]("storeName") //declare Store's key and value type information
// db requests should be combined with `onCompleteNewTx` combinator which honors idb transaction boundaries
store.add(List(obj1, obj2)).onCompleteNewTx { appendTuples =>
  assert(appendTuples.length == 2)
  val (keys, values) = appendTuples.unzip
  assert(values.head == Map("x" -> 0))
  store.get(keys).onCompleteNewTx { getTuples =>
    val (keys2, _) = getTuples.unzip
    store.delete(keys2).onCompleteNewTx { empty =>
      store.count.onCompleteNewTx { counts =>
        assert(counts(0) == 0)
        db.close()
      }
    }
  }
}

• working with key ranges

val store = db.openStore[Int, Int](storeName)
store.add(1 to 10).onCompleteNewTx { tuples =>
  store.delete(store.lastKey).onCompleteNewTx { empty =>
    store.count.map { count =>
      assert(count == 9)
    }
    store.delete(store.firstKey).onCompleteNewTx { empty =>
      store.count.map { count =>
        assert(count == 8)
      }
      store.delete(store.rangedKey(IDBKeyRange.bound(3,5), Direction.Prev)).onCompleteNewTx { empty =>
        store.count.map { count =>
          assert(count == 5)
        }
        db.close()
      }
    }
  }
}

• working with Index

val db = IndexedDb(recreateDB(dbName))
val store = db.openStore[Int,AnInstance](dbName)
val index = store.index[String]("testIndex")
store.add(List(obj)).onCompleteNewTx { appendTuples =>
  index.get(List("index")).onCompleteNewTx { tuples =>
    db.close()
  }
}