What is YapDatabase

Robbie Hanson edited this page May 21, 2014 · 10 revisions

YapDatabase is a comprised of 2 main features:

  • a collection/key/value store built atop sqlite for iOS & Mac (the foundation)
  • a plugin architecture that provides for advanced functionality such as Views, Secondary Indexes, Full Text Search, etc.

It has the following features:

  • Concurrency. You can read from the database while another thread is simultaneously making modifications to the database. So you never have to worry about blocking the main thread, and you can easily write to the database on a background thread. And, of course, you can read from the database on multiple threads simultaneously.

  • Built-In Caching. A configurable object cache is built-in. Of course sqlite has caching too. But it's caching raw bytes, and we're dealing with objects. So having a built-in cache means you can skip the deserialization process, and get your objects even faster.

  • Collections. Sometimes a single key isn't enough. Sometimes a collection & key is better. No worries. YapDatabase supports collections out of the box.

  • Metadata. Ever wanted to store extra data along with your object? Like maybe a timestamp of when it was downloaded. Or a fully separate but related object? You're in luck. Metadata support comes standard. Along with its own separate configurable cache too!

  • Performance. Fetch thousands of objects on the main thread without dropping a frame.

  • Objective-C. A simple to use Objective-C API means you'll be up and running in no time.

  • Extensions. More than just a key/value store, YapDatabase comes with an extensions architecture built-in. You can even create your own extensions.

  • Views. Need to filter, group & sort your data? No problem. YapDatabase comes with Views. And you don't even need to write esoteric SQL queries. Views work using blocks and Objective-C code. Plus they automatically update themselves, and they make animating tables really easy.

  • Secondary Indexing. Speed up your queries by indexing important properties. And then use SQL style queries to quickly find your items.

  • Full Text Search. Built atop sqlite's FTS module (contributed by google), you can add extremely speedy searching to your app with minimal effort.


CONCURRENCY

It all starts with sqlite. The powerful embedded database builtin to iOS and Mac OS X, and the database used by Apple within Core Data. Sqlite has been around since 2000, which means there's plenty of outdated information on the web about it. Like this often touted bit of mis-information:

WRONG: Writing to a sqlite database locks the file and blocks others threads from concurrently reading it.

TRUTH: Starting with v3.7 sqlite added "write ahead logging". Here's what the sqlite website says about WAL:

  1. WAL is significantly faster in most scenarios.
  2. WAL provides more concurrency as readers do not block writers and a writer does not block readers. Reading and writing can proceed concurrently.

We built our database from the ground up to explicitly take advantage of WAL and the latest improvements in sqlite. You get concurrency out of the box. So reading from the database won't block. Ever. You can still only have a single write transaction at a time, but we handle all those details. So if you've used sqlite directly before, you can say goodbye to worrying about "busy" errors.


CACHING

In addition to concurrency, we added built-in caching. This means your code will hit the disk less often, and skip the deserialization step too. And we didn't settle for NSCache. That wasn't nearly fast enough for us. We built our own caching architecture. And it's often an order of magnitude faster than NSCache.

It's configurable too. Configure the limits per database and per connection at any time.

Plus it's integrated into the concurrency model. So update an object on your background connection and the cache(s) on other connections are automatically updated at the proper time.


COLLECTIONS & METADATA

Plain-old key/value is a little restrictive. Many modern "key/value" stores such as MongoDb or Riak have an extra level called a collection or a bucket, etc. YapDatabase follows suite. So instead of using key/value, you can use collection/key/value. This comes in handy when you have a bunch of related but separate objects. It's almost like having tons of small databases, but without all the extra overhead.

Next we added metadata.

You can store an associated "metadata" object alongside your regular object. The metadata can be anything you need. From a simple timestamp (e.g. when the object was downloaded, or when it expires), to something more complicated like pre-calculated computational results, or perhaps a small subset of regularly used ivars from a much bigger object graph.

Naturally metadata has its own separate dedicated cache. Configure it however you need.


PERFORMANCE + OBJECTIVE-C

YapDatabase strives to strike the right balance between providing an easy to use objective-c API, and being as close to sqlite as possible. In sqlite there are 3 layers:

  • the database file itself
  • a "connection" to the database
  • a transaction, executed on a connection, to read and/or write

YapDatabase mirrors these layers in objective-c and then heavily optimizes on top of it. The end result is a sense of familiarity for those with sqlite experience, yet no sqlite knowledge is required. The only thing visible is a clean Objective-C API, and only sqlite guru's will notice how the API secretly enforces performance enhancing techniques.


EXTENSIONS

A key/value database by itself may be nice, but it might not provide all the functionality you need as an application developer. And that's where extensions come in.

The key/value architecture of YapDatabase is just the foundation for an advanced plugin system. These plugins are called "extensions". And they integrate seamlessly.

Views allow you Sort, Group & Filter your data. Perfect for tableViews, collectionViews, and more. And they even provide NSFetchedResultsController style updates to make it easy to animate changes to your tableViews/collectionViews.

Secondary Indexes help to optimize your queries so you can find your item(s) faster. They even support SQL style queries, just like you're used to.

Full Text Search provides blazing fast search using SQLite's FTS module (written by Google).

More extensions are planned. And you can even write your own.