Limiting number of active workflow attempts per site

[frsyuki]

There're schedules and REST API (PUT /api/attempts) that issue new attempts. require> also issues a new attempt but it should be implemented as an alias of REST API.

[frsyuki]

Implementation ideas:

Schedule: if number of active attempts reached hard limit, starting a new schedule delays for several minutes (1 minute first, 10 minutes maximum, using exponential back-off is ideal). After the wait, scheduler tries to issue attempts again. I think this is good because users have chance to fix the schedule when scheduling is blocked.

REST API: if number of active attempts reached hard limit, sever returns 429 Too Many Requests or 400 Bad Request.

Soft limit: if number of active attempts reached soft limit, scheduler and REST API put a new attempt to a queue and don't put tasks. Workflow executor puts tasks of the attempts later. Soft limit could be used also for multi-tenant scheduling.