asyncio uses too many threads by default

[VojtchBoek]

BPO	35279
Nosy	@asvetlov, @methane, @1st1, @EpicWink
PRs	bpo-35279: reduce default max_workers of ThreadPoolExecutor #13618

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-05-28.12:03:20.311>
created_at = <Date 2018-11-19.14:26:55.537>
labels = ['expert-asyncio', '3.7', 'performance']
title = 'asyncio uses too many threads by default'
updated_at = <Date 2019-05-28.12:03:20.311>
user = 'https://bugs.python.org/VojtchBoek'

bugs.python.org fields:

activity = <Date 2019-05-28.12:03:20.311>
actor = 'methane'
assignee = 'none'
closed = True
closed_date = <Date 2019-05-28.12:03:20.311>
closer = 'methane'
components = ['asyncio']
creation = <Date 2018-11-19.14:26:55.537>
creator = 'Vojt\xc4\x9bch Bo\xc4\x8dek'
dependencies = []
files = []
hgrepos = []
issue_num = 35279
keywords = ['patch']
message_count = 9.0
messages = ['330101', '340013', '340015', '343688', '343733', '343737', '343740', '343743', '343766']
nosy_count = 5.0
nosy_names = ['asvetlov', 'methane', 'yselivanov', 'Vojt\xc4\x9bch Bo\xc4\x8dek', 'Epic_Wink']
pr_nums = ['13618']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue35279'
versions = ['Python 3.7']

[VojtchBoek]

By default, asyncio spawns as many as os.cpu_count() * 5 threads to run I/O on. When combined with beefy machines (e.g. kubernetes servers) with, says, 56 cores, it results in very high memory usage.

This is amplified by the fact that the concurrent.futures.ThreadPoolExecutor threads are never killed, and are not re-used until max_workers threads are spawned.

Workaround:

    loop.set_default_executor(concurrent.futures.ThreadPoolExecutor(max_workers=8))

This is still not ideal as the program might not need max_workers threads, but they are still spawned anyway.

I've hit this issue when running asyncio program in kubernetes. It created 260 idle threads and then ran out of memory.

I think the default max_workers should be limited to some max value and ThreadPoolExecutor should not spawn new threads unless necessary.

[EpicWink]

What about making it dependant on memory as well as logical processor count:

n_workers = min(RAM_GB / some_number, N_CORES * 5)

[methane]

node.js default threadpool size is 4 regardless number of cores.
https://nodejs.org/api/cli.html#cli_uv_threadpool_size_size

Since we has GIL, I think fixed-size pool is better idea.

[asvetlov]

asyncio uses bare concurrent.futures.ThreadPoolExecutor.
Tha question is: should asyncio reduce the number of threads in the pool or concurrent.futures should change the default value?

[methane]

Current default value is decided in here:
https://bugs.python.org/review/21527/#ps11902

It seems there were no strong reason for current cpu_count * 5.
I think cpu_count + 4 is better default value, because:

• When people are using threadpool for CPU heavy job which releases GIL, workers >= cpu_count is good.
• When people are using threadpool for multiplexing I/O, best workers number is vary on the workload. But I think 4~16 is good for typical case.

This is amplified by the fact that the concurrent.futures.ThreadPoolExecutor threads are never killed, and are not re-used until max_workers threads are spawned.

Note that this is fixed by bpo-24882.

[asvetlov]

I'm ok with changing the default threads number limit.
Not sure about numbers.
If you want to limit to 16-20 that may be ok but cpu_count + 4 doesn't work in this case. On cloud servers, I see 128 or even more cores very often. 160+4 is not that you want to propose, sure.

I insist on changing the default calculation schema in concurrent.futures, not in asyncio. There is no case for asyncio to be exceptional.

[methane]

If you want to limit to 16-20 that may be ok but cpu_count + 4 doesn't work in this case. On cloud servers, I see 128 or even more cores very often. 160+4 is not that you want to propose, sure.

I proposed cpu_count + 4 because bpo-24882 almost fixed the problem of large maxworks.
If you don't like it, how about min(32, cpu_count+4)?

I insist on changing the default calculation schema in concurrent.futures, not in asyncio. There is no case for asyncio to be exceptional.

Makes sense.

[asvetlov]

how about min(32, cpu_count+4)?

I think it produces reasonable good numbers for any CPU count.

Do you have time for preparing a pull request?

[methane]

New changeset 9a7e5b1 by Inada Naoki in branch 'master':
bpo-35279: reduce default max_workers of ThreadPoolExecutor (GH-13618)
9a7e5b1