Recreate all changed models in development environment

[franzoni315]

One interesting feature would be to always recreate modified models from the development environment.

Consider the dag:
model_1 -> model_2

Imagine you modified model_1 and model_2. If you run dbt run -s model_2, you are going to use model_1 from production environment, but I think one would want to use model_1 from dev environment in most cases.

[LewisDavies]

Thanks for opening an issue! I completely agree that this workflow would be better but I don't think it's possible fora package right now.

The selected_resources variable tells the package which models & tests have been selected, but not which have changed since the last run. To do so would require a production manifest or info from version control.

I'll leave this issue open in case the situation changes or anyone has ideas to share.

[LewisDavies]

@franzoni315 I have something that might work for you - would you be willing to test it out? My approach only works on Snowflake and BigQuery atm.

The package can now use a dev relation if it was created or updated after the prod version. In your example above, it would use dev.model_1 until the next time prod.model_1 is updated. This can be controlled by changing a variable, so it's easy to disable if necessary.

If you'd like to give it a try, add this to packages.yml, set the prefer_recent variable and update your ref macro:

packages:
  - git: https://github.com/LewisDavies/upstream-prod.git
    revision: prefer-recent

Let me know how it goes for you 🙂

[franzoni315]

Hi, @LewisDavies ! Sorry for the delay, I did not see your update. I will see if I can try this next week!

[franzoni315]

I tested it here, and it seems to be working properly! My test was to reference a table that was in my staging environment for a long time, and the compiled code returned the model in production environment. Then, I ran the model again in order to update the table, and the compiled code returned the model in staging environment, as I expected!

AFAIK your strategy was to look at the update date of the model on the database, right? It is not looking if the code of the model is different. Therefore, if a daily job updates the production table, then tomorrow the dbt compile process would choose the production table again, correct?

[LewisDavies]

Awesome, glad it worked! That's right, the package is checking when each table was last modified / created and picks the most recent one. So yes, in your example the production table would be chosen tomorrow after the daily job has run.

Does this approach make sense to you? I'm hopeful that even if the dev table needs to be updated daily, most projects will still see efficiency gains from not building it on every subsequent run. I'd love to hear of any suggestions you have or workflows this doesn't cover though.

[LewisDavies]

@franzoni315 I realised that I've been sitting on these changes for a while so, since it worked for you, I've merged them (#19) and will release a new version soon. I'll close this issue but please feel free to comment with any other questions or feedback you have.