Change FAQ documentation for max_threads

[shenoykarthikd]

FAQ Documentation for max_threads currently reads as follows:

max_threads: Scheduler will spawn multiple threads in parallel to schedule dags. This is controlled by max_threads with default value of 2. User should increase this value to a larger value (e.g numbers of cpus where scheduler runs - 1) in production.

The example above creates confusion in the minds of new developers as it is incorrectly understood as the maximum number of threads for the scheduler cannot exceed the number of cpus - 1. I have seen many Airflow installations where the value is setup as max number of cpus - 1, while the upper limit of threads should actually be determined by the size of the instance (CPU + Memory) onto which the scheduler is installed. Due to this misunderstanding, I've heard many new Airflow developers say that Airflow is very slow at scheduling DAGs. When I delve deeper into their config I see the max_threads configuration limited to the number of CPUs.

Kindly consider changing this to the below as follows -
max_threads: Scheduler will spawn multiple threads in parallel to schedule dags. This is controlled by max_threads with default value of 2. User should increase this value to a larger value that fits the size of the installed hardware in production.

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[teastburn]

User should increase this value to a larger value that fits the size of the installed hardware in production. doesn't seem to guide the user on what fit and size mean. CPU? RAM? It should be as specific as possible in my opinion.

[shenoykarthikd]

Good point, @teastburn! I have typically changed the max_threads value for my Airflow clusters so that CPU utilization doesn't increase beyond 80% over a reasonable time period. Rarely ever checked the RAM as memory usage is typically pretty low on the master node. It may make sense to change the FAQ as below unless someone feels memory should also be a consideration.

max_threads: Scheduler will spawn multiple threads in parallel to schedule dags. This is controlled by max_threads with default value of 2. User should increase it to a higher value in production. Note that an increase in this value causes a corresponding increase in CPU utilization.

[eladkal]

The quote you are referring to is from older airflow versions it was changed in 1.10.14 by #12605 when max_threads is changed to parsing_processes After that it was refactored again for Airflow 2.0
See updated doc

Closing as the docs are updated.