Harness for deploying and testing (dependent) UDFs

[jklukas]

I have previously worked with https://github.com/PeriscopeData/redshift-udfs which provides a structure for defining python UDFs for Redshift along with tests, and a harness for deploying the functions and running tests.

It would be nice to have something similar for this repo. For the BigQuery case, this is made more complicated by the fact that persistent UDFs are not yet generally available.

Here's a potential way forward:

• Change udf definition files under udf/ to use syntax for creating persistent udfs
• Write a small python package for parsing through the udf files to determine dependencies (find usages of udf_*) and build a DAG
• Write a small python package for executing the udf definition files in the order they appear in the DAG
• Write a test harness that can run creation of the persistent udfs as part of running tests in a generated temporary dataset, then runs tests defined similarly to how we have tests defined for tables
• Write a small python lib for parsing the files under sql/ and identifying usage of udfs there; we would then inject temporary udf definitions and output a generated directory; this allows us to more reliably use udfs in our production etl queries without having to duplicate them directly into the source files

The above seems like a significant chunk of work, so likely not an immediate priority.

[relud]

• Change udf definition files under udf/ to use syntax for creating persistent udfs
  ...
• Write a small python package for executing the udf definition files in the order they appear in the DAG

We could instead have the dependency-checker do the same for test queries, and prepend them to the query for running tests, in which case it doesn't matter what order they are specified in.

[jklukas]

We now have a harness for resolving dependencies between UDFs and running tests. I'm going to say that deploying persistent UDFs is probably not a priority right now, so we've met the spirit of this issue.