What are some common Data Engineering projects and ideas for practices?

[moin-oss]

In order to kick-off the creation of a playbook for data engineers, let's start off by collecting some common project types that data engineers work on as a starting point. From there we can discuss possible sustainable practices that can be applied for such projects, or perhaps more generally.

• What are typical projects that data engineers develop?
• What practices can be applied to those projects to make them run cleaner?

[f-mellinghoff]

Typical project/activity: Data mangement & architecture

Ideas for best practices:

• Identification and classification of all existing data (e.g. also data in sandboxes, old archives, test data, ...) -> deletion of all identified ROT (redundant, outdated, trivial) and generally unnecessary data
• Plan housekeeping and deletion concepts from the beginning of the project
• Limit data duplication and archiving as much as possible; share data instead of duplication
• Move archived data to appropriate storage (maybe cold storage is enough for some archived data)
• Ensure possibility for long-term operations (e.g. consider possible data growth durch setup)
• In general: apply lean principles also to data
• only load data where changes occurred, maybe think about event based triggers (only load delta, but also only start followup ETL processes, if some source data changed)

Note: Data storage and transfer of 100GB in the cloud for one year causes approx. 200kg CO2eq

[moin-oss]

Some additional ideas for practices, including some partly adapted from Google Cloud Go Green Guide:

• Try to keep data storage co-located with the processing and retrieval (in other words, use data centers nearest to where it is evaluated)
• Minimize the frequency of batch jobs
• Run compute jobs when the electric grid is cleanest (for example, using Vertflow for Airflow)
• Implement appropriate sized auto-scaling