Random fixes from deploying at customer

[eakmanrq]

• The skeleton project doesn't include an __init__.py when generating the macro directory and I got an error when running. Now we generate this by default.
• There was a bug in part of our code where we assumed all tables existed which isn't the case for external models
• I was having errors when updating the snapshots table. Databricks Delta does not allow concurrent updates to a single underlying file. Therefore I have to partition the table by the condition to force a file per update which also means this table now has a file per partition which isn't good from a performance perspective. I believe a reason for why I didn't hit this before was because I ran in single node mode and that limited the concurrency.
  • Note: I decided to implement this at the engine level. Eventually we will be adding support for setting and understanding the actual format of the tables we are creating and it would make sense to couple this to that once it is created. It would have significantly increased the scope of this PR if I tried to add this now.

[izeigerman]

Doesn't the same problem apply to the environments table? Shouldn't we partition it as well?

[izeigerman]

I don't really like the fact that the state sync logic leaks into something as low lever as engine adapter