Allow partials during plan / run

[mattiasthalen]

Sometimes I need to run partial intervals for all models, e.g., during initial plan.
Instead of setting this for all models, I would like to do it as an option.
sqlmesh plan --allow-partials
sqlmesh run --allow-partials

[tobymao]

i think we could add this for run, but can you explain more about your use case? would love to learn more about how your using this and why you need allow partials, this will help us build context to how folks are using sqlmesh

[mattiasthalen]

I face the issue when initialising a project. It does the first load of all data into snapshots (SCD2), then the downstream models read from those with incremental by time range on _sqlmesh__loaded_at, a field I add during snapshot. That opens the scenario of the snapshots being created 10:04, and the downstream models are still in this partial interval.

[tobymao]

sorry, i'm confused, that's why sqlmesh doesn't allow partials. the incremental by time range shouldn't load any data that isn't complete.

[mattiasthalen]

Yes, but let's say I'm developing a daily incremental model. I won't be able to validate the result until the day after. Which means that I need to first set the model to FULL to see the data and later switch to incremental.

[izeigerman]

@mattiasthalen why is that? Can't you restate a partial interval for the incremental by time model just to test it out?