Why Prisma should avoid polyfilling (for now)

[matthewmueller]

I want to share my thoughts on why starting with database-level features first is important for us from a product and strategic perspective.

Where we are today

The diagram below shows the life a query in both Photon and Lift. I've oversimplified the Datasource server into Handler and Storage.

• Handler: Handlers receive query requests, do stuff and respond with results
• Storage: Is the raw data that the server manages. This could be file or even another database.

You can apply this simplification to Postgres, MySQL, SQLite, Mongo, Redis and HTTP APIs.

┌─────┬───────────┐                           ┌──────────────────────────┐
│ App │           │    ┌─────────────────┐    │ Datasource Server        │
├─────┘┌──────────┤    │                 │    ├──────────┐   ┌───────────┤
│      │          │    │      Query      │    │          │   │           │
│      │  Photon  │◀──▶│     Engine      │◀──▶│ Handler  │◀─▶│  Storage  │
│      │          │    │                 │    │          │   │           │
│      └──────────┤    └─────────────────┘    ├──────────┘   └───────────┤
│                 │                           │                          │
└─────────────────┘                           └──────────────────────────┘

┌────┬────────────┐                           ┌──────────────────────────┐
│CLI │            │    ┌─────────────────┐    │ Datasource Server        │
├────┘ ┌──────────┤    │                 │    ├──────────┐   ┌───────────┤
│      │          │    │    Migration    │    │          │   │           │
│      │   Lift   │◀──▶│     Engine      │◀──▶│ Handler  │◀─▶│  Storage  │
│      │          │    │                 │    │          │   │           │
│      └──────────┤    └─────────────────┘    ├──────────┘   └───────────┤
│                 │                           │                          │
└─────────────────┘                           └──────────────────────────┘

To be a viable query builder and migration tool, there are certain features that we just need to have. Some of these features include:

• CRUD queries: create, read, update, delete
• data types: strings, integers, floats, dates, booleans, json, uuid, list types
• relations: one-to-one, one-to-many & many-to-many relationships
• constraints: uniques, non-null, check constraints
• indexes: optimizing access patterns. indexes may use known constraints to choose faster algorithms.
• default values: if we receive a null value, use a default value or run an expression
• expressions: now(), generate_uuid(), etc.
• procedures: calling stored scripts that you can run against the database.
• triggers: run a procedure if a row or table changes.

The datasource's Handler offers these features on top of pure Storage.

Databases like Postgres, MySQL & Mongo support many of these features natively, while databases like SQLite and Redis support some of these features. APIs like Stripe support very few of these features.

With Prisma's architecture, we have the opportunity to "polyfill" data sources with additional features they don't support natively. We can polyfill in the Query Engine and in the Migration Engine.

This opportunity presents a question: when should we polyfill?

In the past, Prisma polyfilled many parts of the database. This made sense at the time when Prisma managed the database. Introspection changes the game. We now have to deal with an existing database's warts and all. This has led us to re-evaluate polyfilling for features like native scalar lists and cascading deletes.

Our current decision-making is feature-based, but I think we need to think higher-level. We need to develop a default position, where we prefer one decision over the other until the tradeoffs are too great. It's our own version of "innocent until proven guilty."

I believe that our default position should be: "database native until proven otherwise".

Why should we adopt "database native until proven otherwise"?

From a product perspective, we should be database native for the following reasons. Native allows:

• Our introspection, migrate, re-introspect loop works without human intervention.
• Incrementally adopt Prisma because Photon can run alongside existing database clients.
• Write less code because we're taking advantage of the database's features.
• Test less code because we lean on the database's tests. This is more important than it sounds. ^[1]
• We're (likely) more performant because we're doing less and using the database's optimized algorithms.

From a strategic perspective, we should be database native for the following reasons:

• Prisma can plug into existing applications, speeding up our growth inside companies.
• It buys us time. Being database native does not exclude us from adding polyfills to the query engine in the future. These polyfills will be requirements for companies with write-heavy or performance sensitive applications. Conveniently for us, these companies often have a lot of money.
• Less reinvention. Engineering just needs to hook into existing database features rather than clone them in Rust.
• It adds constraints. This might sound like a downside but it's actually a blessing. Right now we can do anything because we have the Query & Migration Engine. Being able to do something doesn't mean that we should do it.
• We stand firmly on the shoulders of database systems that have been around for decades.

What does this change mean for us?

Database-first requires a change in mindset:

• We aim to be idiomatic with the database at the expense of Graphcool/Prisma 1 features that are undoubtably nice to have.
• If we can't figure out a way to do something natively in the database, we'll probably punt on this feature until later.
• We adopt the mindset of the Query Engine and Migration Engine being lightweight connection managers and coordinators. The Query Engine is more like PGBouncer than Prisma 1. For example, I don't think we should even do query validation (outside of GraphQL parsing), let the database handle that.

This mindset is similar to going serverless, where you innovate within the constraints of AWS. We'll innovate within the limits of our target datasources.

This also means we'll need to finalize parts of our Schema like custom types which are essential for being database native.

Tradeoffs

I've made quite a few tradeoffs in this proposal:

• We'll need to refactor and remove chunks of code from our existing query engine and migration engine.
• Concepts like String[] are not possible everywhere.
• Expressions like cuid() would be out of scope and not universally available.
  • If we're adamate about having expressions like this, in Postgres we can write a plpgsql function to provide this.
• Cascading deletes just do what the database does

Again, I lean on the comfort that we can come back to Prisma-level features when companies with big pockets come knocking. Companies like Facebook and Github do not use foreign keys because they're too slow. I'd love to solve their problem at the Prisma-level when the time comes.

Footnotes

[1]: I encourage you to watch this video on SQLite to get a better sense of just how
battle-tested databases like SQLite are.

[divyenduz]

"To shim or not to shim" is almost always no-shim for me. People choose (architects and decision makers) DBs for reasons (native features of the DB), if we shim any of them, that would instantly become a hard sell or an uphill battle. We can in-future be super selective about shimming certain things but no-shim as the default sounds great to me.