The S3 lambda consumer is designed to be triggered when a binary file is written to the S3 intermediary bucket. It downloads the file straight to memory and then transforms the data to Apache Parquet format before writing it to the final S3 bucket in the same file path. This is illustrated below:
This lambda function must be compiled and deployed to AWS lambda with the configuration shown below. This is done automatically through Travis CI when commits are merged to master. Master is a protected branch and cannot be pushed to directly.
