Add support for creating/dropping schema's

[Fokko]

We needed support for creating and dropping schema's.

[drewbanin]

I think you're going to want to delete the two methods shown here too:
https://github.com/fishtown-analytics/dbt-spark/blob/b4db0548266f260c3b75903ce08ad140805b355a/dbt/adapters/spark/impl.py#L36-L46

We disabled this functionality intentionally for two reasons:

1. We wanted to be extra sure that dbt didn't errantly drop the wrong schemas while this plugin was in early development
2. We imagined most applications of this plugin creating external schemas, ie. supplying a LOCATION like s3://path/to/folder when creating schemas

Are you running dbt-spark against an s3 datastore, or against something else? I bet there's a good way for us to configure the location in the create schema query fwiw

[Fokko]

Thanks for the background information. We're using DBT with Databricks on the Azure cloud. We're pleasantly surprised how well DBT works out of the box (with a few patches, such as #34).

We use Azure Data Lake (ADLS) as persistent storage. By setting hive.metastore.warehouse.dir to the ADLS storage bucket, all the data will be persisted on this location. We're using temporary clusters to process the pipeline, and this property is simply part of the cluster configuration. For AWS this will be very similar, you set this property on EMR/Databricks, and point it to a s3:// bucket.

[jtcohen6]

@Fokko Thanks for this! For both reasons @drewbanin mentioned, we were hesitant to build database/schema creation and deletion as native dbt methods when we were just starting on Spark and had a testing sample size of 1. This is a totally sane addition, and it sounds like it hasn't caused any issues in testing.

I don't know enough about all the different Spark vendors/implementations to fully understand the possible mappings between a schema/database and a location in external storage, if a file path location is not specified when running create schema. It seems that there are reasonable ways to set default values in the cluster config or associated metastore. When I run this branch in Databricks (backed by AWS), it just works.

Outside of integration tests and CI builds (in dbt Cloud), I don't think dbt ever initiates the deletion of a schema. If we wanted to be extra safe, we could get away without drop_schema. Given my goal of getting dbt-spark to feature parity with the core adapters, though, we may as well define it all the same.

I'm immensely appreciative of your contributions to this plugin over the past few months. I know you have a few PRs open, which I'm hoping to work through later this week.

[Fokko]

Thanks @jtcohen6 for the extensive response. For now, I've reverted the drop schema. I see how we need to be defensive here. In the cloud, almost every table is an external table since you don't want to have any tables on Hive itself, on the cluster. In the cloud, the clusters are temporary, so you don't want to keep it on HDFS, but store it on S3, GCS, or ADLS.

I've been extensively testing DBT with Azure/Databricks/ADLS. And @Dandandan is testing extensively on AWS using EMR and Glue. We're happy to share the experiences. This change helps us to create databases for users.

[Dandandan]

Thanks @jtcohen for testing this out and reviewing the open PRs!

For EMR, we will also use setting hive.metastore.warehouse.dir, though overriding this location in a config setting might also be useful for us for targeting different target locations, maybe we will look into adding support for that at a later moment.

[Dandandan]

I saw the configurable location is also being added by Niels in this PR #43 ; these two together would be perfect additions to the spark plugin for us!

[jtcohen6]

@Fokko Sorry, I should have been more clear! I think it's fine to include drop_schema here. I was justifying (mostly to myself) why it's not as dangerous now as we felt it may have been several months ago.

I'm immensely grateful to both of you for testing so extensively, and so glad to have you as contributors!

Related work:

• I agree that, since Expose location, clustered_by to dbt-spark #43 will allow overriding the location of specific tables, the location defined in the schema/database is not quite as all-important as we originally thought.
• There's a current bug with partial runs (including dbt seed) if you have custom schemas defined, where dbt tries to inspect metadata for a schema it has not yet created. My proposed changes in Leverage "show table extended" for Relation type metadata #50 would fix.

[Fokko]

Thanks for the clarification @jtcohen6. I've reverted my latest commit. I think this is good to go :-)