[RFC 0002] MVP #2

kiwicopple · 2021-01-29T05:16:19Z

We would like to create a basic CLI for Supabase. At this stage we can assume that the user won't need to create orgs/projects from the CLI, but they will need to:

develop locally
push database migrations to their Postgres database

Rendered

We would like to create a basic CLI for Supabase. At this stage we can assume that the user won't need to create orgs/projects from the CLI, but they will need to: - develop locally - push database migrations to their Postgres database

darora · 2021-02-01T15:23:12Z

rfcs/0002-mvp.md

+
+### Other questions
+- what is the prefix? should it be `supabase` (clearer) or `supa` (might have a few people calling it "supper/suppa")
+- Which CLI tool/framework/libraries should we use?


If we use golang (static binaries, easy to distribute), https://github.com/spf13/cobra is used by most (all?) of the nicer modern CLIs (e.g. kubectl, I think also flyctl under the wraps)

Looks great, although I suspect we will need to use postgres-meta/pg-api for a lot of this functionality.

Unfortunately it creates a dependency on NodeJS

https://github.com/vadimdemedes/ink looks good, apparently Prisma used to use it but I couldn't figure out why they migrated away from it.

@soedirgo It had issues on Windows for us (Prisma) and we simplified things a lot so it was not needed anymore.

BeeeQueue · 2021-02-02T14:36:09Z

rfcs/0002-mvp.md

+```
+
+
+### Generators (bonus)


This is a fantastic idea that I think will help a lot of people!

For some inspiration you can take a look at the fantastic GraphQL Code Generator project!

kiwicopple · 2021-02-14T05:21:23Z

rfcs/0002-mvp.md

+# Models: 
+# This is my suggestion for a "local first" database which has all of the rules/relationships we need to build a a realtime user store with automatic synchronization via our Realtime engine.
+# A bit like https://github.com/mobxjs/mst-gql (but hopefully much simpler)
+supa gen typescript store 


@joshnuss 's ActiveRecord library could be a good starting place for JS:
https://github.com/joshnuss/supabase-active-record

This reverts commit 920a3e2.

kiwicopple · 2021-02-14T08:14:00Z

rfcs/0002-mvp.md

+├── models
+│   ├── public.table                    # Personal preference to explicitly add the schema, and we should prefer dot notation over nesting.
+│   │   ├── data.{sql|csv|json}         # For convenience, table specific data can be stored here too.   
+│   │   ├── model.sql                   # This is the core. We should also figure out a nice, readable format for SQL files. Should include constraints and indexes and table/column comments.


I did a small mock of this to figure out how to structure the file. Some initial thoughts here:
https://github.com/supabase/cli/pull/3/files#r575771489

kiwicopple · 2021-02-14T08:23:59Z

rfcs/0002-mvp.md

+### Dump
+
+```sh
+supa db dump                # dumps the currently active (local) database to disk


I used postgres-meta to create a small mock here: #3

I'm not entirely sure if it's the right approach. On the one hand, we have full control to manage the layout/structure of the dump. On the other hand, we will need to cover all the nuances of Postgres.

Perhaps it's better rely on pgdump from the outset. doing some sort of system where we either:

Option 1:

Run pg_dump on the entire database

Run some sort of AST parser on the full dump

Clean it up and process it into separate files

or Option 2:

Run pg_dump on individual tables/functions/types etc (eg: pg_dump -Fp -t $table -f $table.dump;). We can use postgres-meta to fetch the "structure" of the database

Run some some sort of "prettier" formatter to make it look readable

I'm leaning towards option 2, as it also give the develop better control. For example they might want to dump just their "functions".

For example they might want to dump just their "functions".

AFAICT, pg_dump cannot dump only functions(SO ref).

So if we go the above way, we'd need a combination of 1 and 2

Maybe there's another way.

If you look at the Schema Diff output image in #2 (comment), you'll notice that it also gives the source DDL. And all the database objects are neatly separated.

The Flask endpoint of the pgadmin schema diff process gives this big json as its response. The diff related to the above image is located here.

Notice that the ddl output also has COMMENTs included and it's pretty printed(has newlines).

Checking the big json, all the different types of db objects(event triggers, tables, functions, etc) are there. If we could (ab)use that output, we'd have most of what we need already organized.

kiwicopple · 2021-02-14T08:29:09Z

rfcs/0002-mvp.md

+```sh
+supa db dump                # dumps the currently active (local) database to disk
+supa db restore             # restores the (local) database from disk
+supa migrate                # generate the migrations file


I'm still not sure how we will handle going from a declarative structure, into a set of migration scripts.

There are tools like migra which work on a schema comparison - this might be the way to do it as long as the limitations aren't too restrictive.

eg: "custom types/domains Basic support (drop-and-create only, no alter)". ref

I'm still not sure how we will handle going from a declarative structure, into a set of migration scripts.

Ideally the developer should not look at migrations scripts and instead it should focus on declarative* SQL. So perhaps we could keep the migrations off git?(perhaps on a table)

Of course there's a limit of how declarative SQL can be. Column renames for example always have to use ALTER.

metagration also has an interesting approach. It allows generating up/down migration scripts with stored procedures. It also has its own table for tracking applied migrations.

Maybe we can use metagration plus migra for providing a template migration that can be edited.

metagration also creates a restore point for down migrations, so it's possible to recover dropped columns for example(no need for a restore from a backup).

OK cool - metagration looks interersting.

I think it's the declarative structure -> up/down scripts that's the tricky part here though. The tools I found:

migra (python)

https://github.com/gimenete/dbdiff (JS, looks limited)

https://github.com/joncrlsn/pgdiff (Go)

https://github.com/fordfrog/apgdiff (Java)

https://github.com/eulerto/pgquarrel (C)

Or we can of course code this directly into postgres-meta, perhaps using migra as an example since it seems the most feature complete. It will be a lot of work though.

Unfortunately they're not open to provide a CLI mode: https://redmine.postgresql.org/issues/6304 (you'll need to sign in to see this). They mention that it would take a lot of work.

I still think the pgadmin diff is the best choice. It's already the most complete and has dedicated support. It would provide the best DX.

I've already managed to run it from source. It's a Flask application plus a Backbone.js frontend(seems they're migrating to React).

The functionality for the schema diff was started here: pgadmin-org/pgadmin4@45f2e35. That gives me an idea of the modules involved for making it work.

How about if I try isolating that python code from the GUI and create a package of our own?

How about if I try isolating that python code from the GUI

Succeeded doing that. Code here: https://gist.github.com/steve-chavez/8c6825abee788394cb5424380c06487c.

Now I would need to wrap the code in a CLI.

Nice job Steve - that's an awesome effort.

Now I would need to wrap the code in a CLI

Yeah, it would be very cool if we can use it here somehow.

I have the CLI ready on this branch: https://github.com/steve-chavez/pgadmin4/blob/cli/web/cli.py.

I'm now wrapping it on a docker container so it can be tried without building the whole pgadmin4 package.

Docker container here: https://hub.docker.com/r/supabase/pgadmin-schema-diff/

Usage:

docker run supabase/pgadmin-schema-diff \ 'postgres://postgres@host:5432/diff_source' \ 'postgres://postgres@host:5432/diff_target' \ > diff_db.sql

rfcs/0002-mvp.md

steve-chavez · 2021-02-15T23:27:45Z

rfcs/0002-mvp.md

+```sh
+│   # Data: database fixed/seed data.
+├── data
+│   ├── some_data.{sql|csv|json}        


Regarding branching, this would allow us to just rely on different hardcoded master data for test/staging/preview environments right?

It does seem easier compared to something like snapshotting the db with ZFS. Much more lighter on resources as well.

Yeah exactly - this would give "reproducible builds". It's not as powerful as ZFS cloning, but it is less "magic"

ZFS cloning might solve a different use-case in the future for managing terrabyte-scale databases

rfcs/0002-mvp.md

steve-chavez · 2021-02-15T23:37:38Z

rfcs/0002-mvp.md

+- `postgresql://postgres@localhost:5432/master`
+- `postgresql://postgres@localhost:5432/develop`
+- `postgresql://postgres@localhost:5432/chore_add_types`


On these different envs, would the users also get different realtime/postgrest instances?

For preview workflows, the users would like to interact with the full stack right?

This completely slipped my mind. Yes they are going to need it. Hmm, how will we keep the costs down?

Perhaps it will have to be "local only" for now? It should be manage-able with the docker files

Hmm, how will we keep the costs down?

A temporary postgrest with db-pool = 1(one connection max) should be lightweight, it would fit on the same instance. Realtime would need to limit its pool as well, but we'd definitely need to set its systemd MemoryLimit to be really low. But I'm not sure how low it can get.

Perhaps it will have to be "local only" for now? It should be manage-able with the docker files

Yes, local-only it's a good start. Other envs can be done in a later RFC - they need more thought.

We can also take advantage of systemd socket activation, which essentially lets a service sleep until a request comes. That coupled with a systemd timer to stop the service after some idle time(so question), would make resource usage minimum.

…umn? That would give the CLI a more framework-like experience.

soedirgo · 2021-10-13T11:56:25Z

Closing this as we're transitioning to the Go CLI.

RFC0002: MVP

bb422ae

We would like to create a basic CLI for Supabase. At this stage we can assume that the user won't need to create orgs/projects from the CLI, but they will need to: - develop locally - push database migrations to their Postgres database

kiwicopple changed the title ~~RFC0002: MVP~~ [RFC 0002] MVP Jan 29, 2021

fix: formatting

a65274a

darora reviewed Feb 1, 2021

View reviewed changes

kiwicopple mentioned this pull request Feb 2, 2021

Update Type Implementation (supabase-js side) supabase/supabase-js#125

Closed

BeeeQueue reviewed Feb 2, 2021

View reviewed changes

kiwicopple commented Feb 14, 2021

View reviewed changes

kiwicopple added 2 commits February 14, 2021 15:53

Mocking up some ideas for dumping a database

920a3e2

Revert "Mocking up some ideas for dumping a database"

3316639

This reverts commit 920a3e2.

kiwicopple commented Feb 14, 2021

View reviewed changes

steve-chavez reviewed Feb 15, 2021

View reviewed changes

rfcs/0002-mvp.md Outdated Show resolved Hide resolved

steve-chavez reviewed Feb 15, 2021

View reviewed changes

rfcs/0002-mvp.md Outdated Show resolved Hide resolved

steve-chavez reviewed Feb 15, 2021

View reviewed changes

kiwicopple added 3 commits February 17, 2021 21:44

Functions: stick to SQL for this RFC, plv8

9994d54

Test: generate some simple pgtap tests by default? Like table has_col…

b217cff

…umn? That would give the CLI a more framework-like experience.

Types should all be kept in a single file

f4f36db

soedirgo closed this Oct 13, 2021

soedirgo deleted the rfc/mvp branch December 3, 2021 03:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC 0002] MVP #2

[RFC 0002] MVP #2

kiwicopple commented Jan 29, 2021 •

edited

darora Feb 1, 2021

kiwicopple Feb 2, 2021

soedirgo Feb 14, 2021

Jolg42 Mar 2, 2021

BeeeQueue Feb 2, 2021

kiwicopple Feb 14, 2021

kiwicopple Feb 14, 2021

kiwicopple Feb 14, 2021

steve-chavez Mar 13, 2021

kiwicopple Feb 14, 2021

steve-chavez Feb 16, 2021

steve-chavez Feb 16, 2021

steve-chavez Feb 16, 2021

kiwicopple Feb 17, 2021

steve-chavez Mar 12, 2021

steve-chavez Mar 15, 2021

kiwicopple Mar 15, 2021

steve-chavez Mar 19, 2021

steve-chavez Mar 25, 2021 •

edited

steve-chavez Feb 15, 2021

kiwicopple Feb 17, 2021

steve-chavez Feb 15, 2021

kiwicopple Feb 17, 2021

steve-chavez Feb 18, 2021

steve-chavez Mar 25, 2021

soedirgo commented Oct 13, 2021

		```


		### Generators (bonus)

[RFC 0002] MVP #2

[RFC 0002] MVP #2

Conversation

kiwicopple commented Jan 29, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steve-chavez Mar 25, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

soedirgo commented Oct 13, 2021

kiwicopple commented Jan 29, 2021 •

edited

steve-chavez Mar 25, 2021 •

edited