Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove thread count limitation on workers #1962

Closed
Hazhzeng opened this issue Jul 13, 2021 · 2 comments
Closed

Remove thread count limitation on workers #1962

Hazhzeng opened this issue Jul 13, 2021 · 2 comments

Comments

@Hazhzeng
Copy link

Hazhzeng commented Jul 13, 2021

Motivation

What is the motivation for the change?
This will increase the throughput of I/O intensive Python and PowerShell function apps.

Impact

How many customers (roughly) would be impacted by this break? If not known, how can we figure it out? This may require host changes to gather metrics.
This is possible to break customer's with memory intensive Python function apps. May increase the number of out-of-memory errors (e.g. "Python exit code 137")

Compat-mode support

We'd like to enable compat-mode support for every breaking change. This may not be feasible in reality, but each proposal should include a plan to switch back to the previous behavior with a feature flag. This requirement will be evaluated on a case-by-case basis.
If the error is impacting large number of customers, changing the feature flag (e.g. PYTHON_THREADPOOL_THREAD_COUNT) to 1 should mitigate the issue.

Alternatives

Were any alternatives discussed? Is there any way to do this without a break?
This requires function host to expose host capability and let Python worker to consume. Control of this value lays in the worker logic. This feature requires collaboration with the Intelligent Language Worker Concurrency design, @alrod has more context on this.

Detection

Can we detect that a customer is using this when they upgrade from v3? Is there a specific error that can be thrown with a link to migration guidance?
Checking the FunctionsLogs table. During the worker initialize request handling, it will show that the thread count is unlimited. Correlating this information with Function Host version, can determine how many customers are using this feature.

Support

Will there be any incidents-per-day impact? Who will be the support contact? Does support need to be notified of this change? (SPOT)
As mentioned, "Python exit with 137" error count may increase. @vrdmr and @Hazhzeng are the ones to help in this scenario. Anything with thread scheduling, @alrod is the one to contact.

Documentation

Feature flag default value needs to be updated for v4 function apps.

Components impacted

What components does this change impact? Examples of areas (this list may not be exhaustive):
- Host
- Worker

Performance

Does the change have any performance impact? There may need to be some implementation complete before this can be measured.
Should increase I/O bound function app throughput. Introduce a risk or memory intensive function app.

/cc: @AnatoliB @vrdmr @stefanushinardi

@Azure Azure locked and limited conversation to collaborators Jul 13, 2021
@Hazhzeng Hazhzeng changed the title Remove PYTHON_THREADPOOL_THREAD_COUNT limitation Remove threading count limitation on workers Jul 13, 2021
@Hazhzeng Hazhzeng changed the title Remove threading count limitation on workers Remove thread count limitation on workers Jul 13, 2021
@Hazhzeng
Copy link
Author

Discussion: #1967

@vrdmr
Copy link
Member

vrdmr commented Sep 23, 2021

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants