Feature/earthbeam dynamic task mapping

[rlittle08]

Introduces build_dynamic_tenant_year_taskgroup, for running dynamic task mapping in earthbeam. This allows you to run multiple instances of earthmover, etc. if you receive multiple files

Some things that definitely need review:

• dynamicPythonOperator. I added this to allow for some kwargs to be static, some dynamic. But there is likely a better way to accomplish this
• evaluate_kwargs and handling of kwargs in general. This probably overlaps with Jay's work to make a configurable factory.. we need to be flexible to multiple different structures of earthmover kwargs, depending on the configured template. I sort of have that here, but the context from here is hard coded to include one input_file, when we likely need to be flexible to multiple input parameters and other earthmover kwarg constructions??

I have been testing with this refactored DAG in texas - https://github.com/edanalytics/stadium_txexchange/blob/feature/refactor_dynamic_earthbeam/airflow/dags/refactor_dynamic_earthbeam_dags.py

[jayckaiser]

@rlittle08 Why not just update this line to the following:

raw_to_s3_kwargs = list_files_task.output.map(lambda raw_dir: {
    'local_filepath': raw_dir['local_filepath'],
    's3_conn_id': s3_conn_id,
    's3_destination_key': s3_raw_filepath,
    'remove_local_filepath': False,
})

And then remove the need for the static_kwargs argument?

[rlittle08]

This might work. I thought I had some issue with passing static values as dynamic args, but I can't remember the details of it, so I think this is worth a shot

[jayckaiser]

Why are we passing ds_nodash and ts_nodash strings across these methods? These date stamps are already added when a user uses EarthbeamDAG.build_local_raw_dir().

[rlittle08]

can you help me with this? I'm not sure how to correctly construct output_dir here