Hello and welcome to Camino Financial Data Engineering take home project!
The mission, should you choose to accept, is to sift through some data craziness and make some sense out of them.
Be comfortable with key concepts of data engineering.
This is a tangible way for you to show us:
- That you know the basics of working with data
- That you can think outside the box
- Given this json file, please design a sql table or tables that would be query-able.
- Deliverable A: The code that does the parsing, in any language of your choice (we prefer Python 3).
- Deliverable B: The sql schema diagram
- For the question below, you may use the SQL DB Fiddle https://www.db-fiddle.com/f/82DzJYdoLWuiPyxvxufERN/0
- Question 1: Write a query to counts the number of duplicates in car table for each company.
- Question 2: Write a query to list every entry in car table and its associated company owner. If an entry doesn't have an owner in the company table, then print 'not found' in the cell.
- If you prefer your own local database, you can use the sql to recreate the tables and data set.
- Now imagine for a moment, we are receiving 1000 records per second. Please design an ETL data pipeline that will be able to handle this and then eventually send everything to a data warehouse ...
- Deliverable: A diagram depicting your ETL data pipeline. Please elaborate also your tool of choice, processing frequency and cost.
Please provide your deliverable in a private repo and send to us for review.
If you have any questions, concerns or suggestions please feel free to ping us on the recruiting platform. Happy coding! Camino Financial is a pretty awesome place to work, We hope to see you soon!