Scaling Marquette to reduce the amount of user input

[taddyb]

What is the issue:

• Currently, there is a decent amount of offline scripting and ArcGIS Pro manipulation that needs to be done for a map to be created.
• The maps that are created are localized to the end point of a basin, so depending on where you want to map you may have to run this program many times.
• The MERIT flow lines already separate CONUS, and other global basins, so it makes sense to have Marquette work for a whole MERIT area, rather than just a defined basin

TODOS:

• Scalability!
  • We need to ensure that we can run this code for a whole merit basin in a small amount of time. This code needs to scale to the global level!
  • Implement Zarr file storage since shp/csv files are too large to read/write
• No end points
  • Run this code for basins of all size
• triple check that the code works and integrates with the routing product.

[taddyb]

The scalable edge creation has finished and is working well.

Next is to create the Transition Matrix (TM) for HUC10 to MERIT basin, then MERIT basin to edges

[taddyb]

This is finished up and #5 should be closed today.

Currently, the only MERIT requirements for generating river connectivity and formatting streamflow are:

save_paths:
  attributes: ${data_path}/streamflow/attributes_dpl_v3.csv
  basins: ${data_path}/${name}/raw/basins/cat_pfaf_${continent}${area}_MERIT_Hydro_v07_Basins_v01_bugfix1.shp
  flow_lines: ${data_path}/${name}/raw/flowlines
  huc10: ${data_path}/HUC/huc_10_CONUS.shp
  streamflow_files: ${data_path}/streamflow/predictions/dpl_v3/

• attributes is a list of the basin IDS from your streamflow predictions
• basins is a list of the MERIT basins with no formatting required
• flow_lines is the MERIT flowlines with no formatting required
• huc10 is the shp file for all of the HUC 10 files to then run an intersection with the MERIT basins
• streamflow_files is the streamflow predictions. This can be in either numpy format or separated out.