Rust backlog

[rnewman]

We're starting on a Rust rewrite of the existing Clojure codebase, using the rust branch.

With an eye to parallelism, the following will get us to parity and a bit further, at which point we'll start on the list of more speculative work.

Large portions of this will be easier than the first time around: the ClojureScript implementation required lots of deep thinking that won't need to be repeated.

Starter Projects:

• Learning Rust. This is a totally valid use of time.
• [@bgrins] Get Rust builds and tests working in CI.
  • Land initial code (Update test boilerplate for running on travis #134)
  • Update readme with build and test instructions (Include instructions for building and testing with cargo #135)
• [@jsantell] Begin SQLite wrapper
  • Survey of Rust SQLite libraries.
  • [@bgrins] dependency
  • open a DB at a path and run the default pragmas
  • exec/query abstractions
  • How do we update and deliver SQLite?
• [@joewalker] Build an EDN parser.
• [@jsantell] Build a Rust node native module. End goal: call a Rust function from JS.
• Build a Rust library for Android. Call it via JNI.
• Build a Rust library for iOS. Sign it. Call it from Swift.
• [@bgrins] Command-line tool. Right now it wouldn't do much, but we could put the skeleton in place and do something like print user_version for a DB path… Stub out cli crate #136
  • Evaluate exposing as a crate within a subfolder
• [@bgrins] Figure out an out-of-band test solution for Rust. In-code tests are great, but they're going to get really verbose when we have piles of SQL generation tests. (see https://github.com/mozilla/datomish/tree/rust/tests)
• SQLite generator libraries a la honeysql: [query] Add some algebraic interface for generating SQL #273
  • Research.
  • Import and use.
• Add a logger, something like https://github.com/slog-rs/slog

Moving on:

• Core (the big blocker for much of the rest):
  • Core valueType work: type codes and translation, value conversion.
  • Core schema work. Basic Rust type definitions. Traits for translations.
  • SQLite wrapper: opener, part reading, etc.
  • Bootstrapper.
  • Conn + DB.
• Querying:
  • [@rnewman] Datalog query parser: port from DataScript. Parse simple find expressions #160.
  • Parsed query (FindRel) -> abstract query (ConjoiningClauses). Porting datomish.query.
    • This will involve building up much of 'Core', above, brick by brick.
  • Abstract query -> SQL.
  • Abstract query -> projectors.
  • Querying history and logs.
• Transactor:
  • Transactor expansion: schema + shorthand -> datoms.
  • Transactor writer.
  • Transactor loop.
  • Prepared statements/statement caching.
  • Excision.
  • noHistory.
• Schema management module (porting).
• Reader pool (query executor).
  • Query interruption via closing connection.
  • Parallel reads.
  • Connection affinity to exploit sqlite page cache.
  • Prepared statements for repeated queries.
    • Invalidation when schema contents change.
    • Handling of query transformation in the presence of bound values, which might change between uses of the prepared statement.
• Tooling. We can invest in this as much as we want; it's leverage.
  • A command-line query tool, just like sqlite3.
  • A REPL that's capable of doing programmatic read/write tasks. JS bindings? Rust REPL?
  • A developer tool/embedded server: showing query translations and plans, timing query execution, exploring schema…
  • GraphQL interface generation from Datomish schema.
  • Bulk loading and import. Make this independent by expanding into a datom stream wrt a given schema.
  • Export: textual/EDN, RDF?
• Transaction listeners over a socket.
  • Log dumper standalone tool. Watch writes happening as they occur.
• Syncing:
  • Transaction log replication (out) and replay (in).
    • Code and algorithms exist.
    • Define how IDs are mapped in and out, and how those might be efficiently represented in chunks.
    • Handle application of noHistory attributes.
    • Handle remote excision.
  • Multi-master transaction log handling (syncing).

[chrmod]

If possible, the Rust implementation may be compiled to Javascript via Ecmascripten. In this way Cljs implementation could be fully abandoned.

[rnewman]

Abandoned by us, quite probably. But because there's a native component — SQLite itself — it will probably be convenient to treat Datomish as simply a larger native component, and we'll tack a JS API on the front for those callers. I'm not sure how compiling SQLite itself to WebAssembly would work, given its reliance on low-level filesystem calls, but someone might be motivated to explore that.

I can imagine that some Clojure/ClojureScript developers might be interested in continued development of the existing codebase; if so, great, and they can continue to do so. Running pure Clojure on top of JDBC is a different but workable stack for the exact same code.