New dev workflow

[tailhook]

First step stays as is for now:

edgedb project init

TBD: Do we want edgedb watch to run project init?
Upsides:

• Single command for everything

Downsides:

• Whenever user needs to customize options in non-interactive mode (i.e. cli options) they will only work in project init not watch.
• We don't teach users that project init is a thing. So they don't know how to script this thing by running individual non-interactive commands. They don't know how to customize init for their project's guide (although, we should be able to put everything needed in edgedb.toml)

I think in other things we optimize for easier start rather than learning, so we probably want to edgedb watch to ask for running edgedb project init automatically.

Then in a terminal separate from your main application, start the watch command:

> edgedb watch
Initialized. Monitoring "/work/myproject".

If Initialized. Monitoring is printed and no error, you're good to go:

1. Edit and save dbschema/* files
2. Your changes to the schema should be applied in 3-5 seconds. In case they are not, take a look at the console that runs edgedb watch. You should either see a spinner (in case migration took more time than expected) or the error.

You are free to run git pull/git switch or edit files in dbschema (including migrations) in any way you want. As long as changes are valid and non-conflicting edgedb watch will pick up new files seamlessly.

Comitting Schema

When you're ready to share schema with your team do:

$ edgedb migration create

At this point all your changes since your last migration will be verified again. You have an opportunity:

1. To mark column/type renames as such
2. To give fill expressions for properties

Now commit both schema and migrations:

$ git add ./dbschema
$ git commit

[elprans]

First step stays as is for now

Yeah, I think the safer option would be to keep them separate for now. Other than that, this looks beautiful in its simplicity.

[raddevon]

@tailhook Just started testing this, and I've run into a question. I changed my schema from this:

module default {
  type Person {
    required property name -> str;
    property phone -> int64;
  }
}

to this:

module default {
  type Person {
    required property name -> str;
    property phone -> str;
  }
}

which produces a schema error:

Schema migration error: SchemaError: property 'phone' of object type 'default::Person' cannot be cast automatically from scalar type 'std::int64' to scalar type 'std::str'
  Hint: You might need to specify a conversion expression in a USING clause

What's the ideal workflow for me at this point? Is the USING clause mentioned in the hint representable in my .esdl file, or should I be stopping my watch, creating a migration, and then editing it?

[raddevon]

Automatic drop-create could be a problem if I have data in that property I want to preserve. The automatic casting sounds like a reasonable solution if it's feasible.

I almost suggested prompting for a cast expression like we do in a proper migration of this type, but it feels a bit weird to suddenly go "interactive" within the watch command. That gives the user the most flexibility, but I think it could easily be missed and leave the user not knowing what's going on.

[arogozhnikov]

Making field required blocks migrations and just as well may leave user clueless.

Unless you want migration hints to be a part of SDL, I don't see a better way than prompts.

You implicitly expect that user controls when the file is saved (intellij IDEs don't provide this control) - removing character or reordering lines may drop a column. That's confusing, as if database spontaneously drops data.

Entering relevant test data is more job than changing schema or running manual migration, so to "just drop" it every time one needs a good tool to fill with mock data according to current schema. Developer in parallel can edit schema and mock data.

[arogozhnikov]

And developer still needs to monitor watch output, because in some cases migration would be stall (errors, new required fields)

[arogozhnikov]

how about 1) trying automatic cast first and 2) falling back to prompts if failed?

[tailhook]

That's confusing, as if database spontaneously drops data.

Well we don't expect that there is anything valuable in the data in the the dev mode. So we are assuming it's fine to happen at times.

If the drop is intentional, it's also more likely that you will drop columns just added a few minutes ago, than columns having long history of production data, because production data columns usually have a deprecation period for backwards compatibility.

I'm not sure how bad is that with Intellij though. As I usually use the editors that provide some control even if autosave. But I think we can't detect whether it's a temporary removing of columns or permanent one just from the text of files. The only thing we can do is wait more when files changed. But that probably means tens of seconds or minutes, which is too long to make the watch function useful.

Delaying just drops specifically might be possible, but can also be unexpected and thus annoying.

Making field required blocks migrations and just as well may leave user clueless.

We're working on making it more visible: #4625

On the other hand, it's almost always not useful to make field required if there are some objects (I mean you will have the same problem on the production database and any kind of fill expression is rarely better than an empty value).

how about 1) trying automatic cast first and 2) falling back to prompts if failed?

The problem with that, is that prompts conflict with additional changes, I mean:

1. Prompt needs to be canceled or replaced by another one while you're typing something and editor saved something in the background. This requires quite complex safeguards to be useful.
2. Every time you update migrations in files (e.g. git pull/merge/switch or edit migrations manually) we might need to ask the same questions. And that might be more annoying than the initial problem we're trying to solve. Technically we can also cache the answers, but matching them can be technically complex. And managing the cache (i.e. when user explicitly wants to change conversion expression) can be another annoying thing.
3. Even if we want to implement (1) unfortunately we don't have cancelable prompts (and that sounds hard to implement in our specific case).

So it's unlikely we can do that in 3.0 milestone. Also, while we understand those theoretical implications we want people to play more with simpler version and see how often manual workarounds are needed and how annoying they are before making something more complex and error-prone.

You can, however, run edgedb migration create when error occurs and do whatever you need manually. Currently this means managing created migration file manually (i.e. you can remove it and create a new bigger migration before commit). Maybe we can add edgedb migration create --dev-mode or make edgedb migrate --dev-mode interactive.

[arogozhnikov]

current watch migrates database, but does not check/reexport .graphql files.

I'd expect if schema can be watched, so all the dependent queries (this can be optional).

[tailhook]

I'm sorry, I'm not sure what "reexporting .graphql" files mean.

EdgeQL -> Javascript/python/.. codegen runs with their own watch for now (if that's implemented in respective client library).