Prevent iNaturalist from running alongside any other DAGs

[AetherUnbound]

Description

Similar to #1277 in reasoning, iNaturalist can be intensive and disruptive for the other DAGs running on the instance. If possible, we would like to prevent iNaturalist from running while other DAGs are in progress. One way to do this would be to leverage Airflow pools. Currently most of our tasks sit in the default_pool, which has 128 slots. (We could increase this to a higher number if desired).

Tasks which use pools can request multiple slots. One way to prevent iNaturalist concurrency would be to to set its pool_slots to the maximum size for default_pool, meaning it would require all pool slots to be available. This, paired with a reduced priority weight would mean that all other tasks would run prior to iNaturalist, and the latter could only run if all slots are available. This would apply per-task, so even if iNaturalist ran a task, as soon as it completed a task it would free up the pool slots for other DAGs.

Alternatives

Additional context

See #1277 for an additional/alternate option.

[rwidom]

I'm sure that this is much lower priority now, but I have some questions that I didn't want to lose for whenever we get back to this. Mainly, they're about the mapped task that takes the vast majority of the iNaturalist runtime. Are the mapped subtasks going to count as separate tasks where others could go in between? I can't tell if that would be good (just have iNaturalist upsert a few more records whenever there is downtime) or bad (keep the inaturalist schema around almost all the time). If the mapped task counts as one single task, then it might be like just stopping every other dag for two days once a month. Do we have two days a month without any dags running currently? What kinds of changes to batch size and/or timeouts might we want to make as a result of answers to the questions above?

[AetherUnbound]

The tasks themselves would count individually, rather than as a block. I think that should make it easy to run other tasks while iNaturalist is processing that batch of load tasks! I'm less concerned about the space the inaturalist schema takes up in that case, so I think we wouldn't need to make any other changes to other DAGs!