added difference between Deferrable and Non-Deferrable Operators

[rohan472000]

closes: #31819

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

Isn't that the same as described at the top? Maybe just adding the header of your above the two paragraphs is enough?

Standard :doc:Operators </core-concepts/operators> and :doc:Sensors <../core-concepts/sensors> take up a full worker slot for the entire time they are running, even if they are idle; for example, if you only have 100 worker slots available to run Tasks, and you have 100 DAGs waiting on a Sensor that's currently running but idle, then you cannot run anything else - even though your entire Airflow cluster is essentially idle. reschedule mode for Sensors solves some of this, allowing Sensors to only run at fixed intervals, but it is inflexible and only allows using time as the reason to resume, not anything else.

This is where Deferrable Operators come in. A deferrable operator is one that is written with the ability to suspend itself and free up the worker when it knows it has to wait, and hand off the job of resuming it to something called a Trigger. As a result, while it is suspended (deferred), it is not taking up a worker slot and your cluster will have a lot less resources wasted on idle Operators or Sensors.

[potiuk]

(and maybe cleaning/clarifiying those two chapters that already exist if you feel like they are not clear enough - but I have a feeling that adding yet another description of what defferrable vs. non-deferrable is not necessary. better to improve the existing one).

[utkarsharma2]

cc: @phanikumv

[rohan472000]

@uranusjr , plz check my latest push.

[potiuk]

LGTM.

[raphaelauv]

+--------------------------------------------------------+--------------------------------------------------------+
| Resource Usage is Higher (repeated execution)          |  Resource Usage is Lower (pauses when idle, frees      |
|                                                        |  up worker slots)                                      |

the reschedule mode also frees up worker slot will it is in UP_FOR_RESCHEDULE