Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Autotuning of the number of async I/O threads #151

Closed
michaelklishin opened this issue May 7, 2015 · 4 comments
Closed

Autotuning of the number of async I/O threads #151

michaelklishin opened this issue May 7, 2015 · 4 comments
Assignees
Milestone

Comments

@michaelklishin
Copy link
Member

See this mailing list thread.

I wonder if we can do autotuning that will work reasonably well on all supported platforms.

@binarin
Copy link
Contributor

binarin commented Mar 23, 2016

So is default value like 16 * erlang:system_info(logical_processors_available) good enough?

@michaelklishin
Copy link
Member Author

@binarin that sounds good but I'd add "but no fewer than 64".

@binarin
Copy link
Contributor

binarin commented Mar 28, 2016

Looks like detecting number of CPUs is not as simple as it seems on some OSes. Lev Walkin posted the following snippet in erlang-russian ML:

ncpus() ->
    case erlang:system_info(logical_processors_available) of
        unknown -> % Happens on Mac OS X.
            erlang:system_info(schedulers);
        N -> N
    end.

@michaelklishin
Copy link
Member Author

@binarin combined with the "no fewer than 64" condition above, it sounds good to me. It's unlikely that there will be significantly more schedulers configured than there are logical cores. We might want to cap the maximum number, say, at 512 or 1024, too.

binarin added a commit to binarin/rabbitmq-common that referenced this issue Mar 30, 2016
@michaelklishin michaelklishin added this to the 3.6.2 milestone Mar 30, 2016
openstack-gerrit pushed a commit to openstack-archive/fuel-library that referenced this issue Jun 22, 2016
Proper version of this calculation is implemented upstream -
rabbitmq/rabbitmq-server#151

And removed version of the code is actually harmful, as it's using
physicalprocessorcount (i.e. number of CPU sockets) for
calculations. So on a 2 CPU/48 thread system there it was 30 threads
instead of 768 calculated upstream.

I've decided that it's easier to completely remove this code instead
of duplicating current formula in upstream and providing a way to
override this value through hiera - just not worth a hassle.

Change-Id: I415d446af0a822d2a5ce3478fd9db1dd0f13e115
Closes-Bug: 1573696
openstack-gerrit pushed a commit to openstack-archive/fuel-library that referenced this issue Dec 1, 2016
Proper version of this calculation is implemented upstream -
rabbitmq/rabbitmq-server#151

And removed version of the code is actually harmful, as it's using
physicalprocessorcount (i.e. number of CPU sockets) for
calculations. So on a 2 CPU/48 thread system there it was 30 threads
instead of 768 calculated upstream.

I've decided that it's easier to completely remove this code instead
of duplicating current formula in upstream and providing a way to
override this value through hiera - just not worth a hassle.

Change-Id: I415d446af0a822d2a5ce3478fd9db1dd0f13e115
Closes-Bug: 1573696
(cherry picked from commit 0c16cbc)
openstack-gerrit pushed a commit to openstack-archive/fuel-library that referenced this issue Dec 1, 2016
Proper version of this calculation is implemented upstream -
rabbitmq/rabbitmq-server#151

And removed version of the code is actually harmful, as it's using
physicalprocessorcount (i.e. number of CPU sockets) for
calculations. So on a 2 CPU/48 thread system there it was 30 threads
instead of 768 calculated upstream.

I've decided that it's easier to completely remove this code instead
of duplicating current formula in upstream and providing a way to
override this value through hiera - just not worth a hassle.

Change-Id: I415d446af0a822d2a5ce3478fd9db1dd0f13e115
Closes-Bug: 1573696
openstack-gerrit pushed a commit to openstack-archive/fuel-library that referenced this issue May 2, 2017
Proper version of this calculation is implemented upstream -
rabbitmq/rabbitmq-server#151

And removed version of the code is actually harmful, as it's using
physicalprocessorcount (i.e. number of CPU sockets) for
calculations. So on a 2 CPU/48 thread system there it was 30 threads
instead of 768 calculated upstream.

I've decided that it's easier to completely remove this code instead
of duplicating current formula in upstream and providing a way to
override this value through hiera - just not worth a hassle.

Change-Id: I415d446af0a822d2a5ce3478fd9db1dd0f13e115
Closes-Bug: 1573696
dcorbacho pushed a commit that referenced this issue Nov 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants