Sneaking Interactivity into Static SQL Statements
To enable conditional functionality in otherwise static SQL scripts in order to facilitate the following:
- Automated SQL testing
- Data integrity testing in production
- Idempotent preconditions in SQL scripts
- Simple parameterization
Full 360 has been building data warehouses in AWS since 2007. At first we used flow based ETL tools for data integration, but we always ended up running into the limitations of functionality and cloud/devops friendliness.
We soon moved on to building data ingestion processes with ruby, and eventually jruby (because of easy JDBC driver support) running in Docker containers. SneaQL is the distillation of our best practices using jruby and templatized SQL when interacting with columnar databases.