Use Entity/Component or Factory pattern(s)?

[7yl4r]

I have been meandering a bit, so I think I need to formalize a plan here.

---

Here is the argument for using some composition-based pattern:

We need to build DAGs for each region. DAGs will be quite similar, but not necessarily identical. Here is the plan I think makes most sense (using modis_aqua_sst "pipeline" as an example:

1. one DAG file for each "pipeline" (and all regions) eg dags/modis_aqua_sst.py
2. options for different regions are passed as parameters to a DAGBuilder function
   1. parameters like {"args":[], "kwargs":{}} within the "sst" DAG or nearby (/dags/modis_aqua_sst/regional_params.py)
   2. builder ModisAquaSSTDAGBuilder(place_name='gom'), adds tasks to create a modis_aqua_sst pipeline for the gom region.
   3. a list of parameters can be defined to build all ROI DAGs quickly: for region_opts in [{"args":[],"kwargs":{"place_name": "gom"}}, {...}]: ModisAquaSSTDAGBuilder(region_opts['args']*, region_opts['kwargs']**).build()

Some thoughts on how editing this pipeline works:

• to add an ROI to a pipeline: open pipeline DAG, add options for region
• to customize pipeline for an ROI: change the parameters for the builder
• no global "regions", although enum-like classes should probably be used for "magic" strings (eg place_names like "gom", "fknms", etc)

---

So that above looks like a simple factory pattern, but does it make sense to go all the way to an entity-component-like pattern? I think that is overkill because "entities" maintain state outside of this builder pattern anyway. That is... Assuming that in this case by "Entity" we mean "DAG", and by "Component" we mean "set of 1+ operators".

[7yl4r]

I did some work on this on the regionBuilders branch, but it is messy. I think I need to focus in and come back to this after I finish cleaning up the last few experiments.

[7yl4r]

🙄 "only use builders in airflow v1.9+"

[7yl4r]

Now that I have learned more about airflow, the correct solution is jus to use DAG and Operator subclasses. The correct approach is to extend the DAG and *Operator classes and use mixins for DAG subclass composition if you really want to get fancy.

Other patterns mentioned here are likely to cause more confusion than they are worth.