cpu_affinitity(physical=True)

[orenbenkiki]

Summary

• OS: Any
• Type: new-api

Description

Add a physical parameter to psutil.Process().cpu_affinity().

The default value would be False, in which case the function behaves exactly as today.

If physical is True, then instead of accepting/returning a list of integers, the function would accept/return a list of lists of integers, where each sub-list contains the logical CPUs belonging to the same physical core.

Motivation

I would like to create a process pool with one process per physical core. Since I'm running memory-heavy compute-heavy code in each one, using logical processors is counter-productive. It gains nothing since the compute-heavy (numpy) code already uses close to 100% of the physical core's ALUs, and at the same time increases memory and cache pressure, which decreases performance.

Right now I am using cpu_count(logical=False) to get the number of processes to use. However, this is wrong in case affinity is specified for the application. However, cpu_affinity() returns logical CPUs, so I can't use that as a guide.

Having a physical=True parameter to cpu_affinity() would allow using len(psutil.Process.cpu_affinity(physical=True)) to provide the number of physical cores usable by the application, to complement the current len(psutil.Process.cpu_affinity()) idiom to provide the number of logical CPUs usable by the application.

[giampaolo]

If physical is True, then instead of accepting/returning a list of integers, the function would accept/return a list of lists of integers, where each sub-list contains the logical CPUs belonging to the same physical core.

Mmm... rather than expanding cpu_affinity() API, this sounds like something which would be better served by exposing a separate psutil.cpu_topology() API, returning a dict of some sort, which can be used in conjunction with cpu_affinity().
Also, it reminded me of this https://github.com/giampaolo/psutil/pull/643/files.

[SimonHeimberg]

psutil is a good place for this.

Currently I can get the cpu affinity with os.sched_getaffinity(0) for systems supporting this function (on some Unix platforms (1)). Or I can use psutil instead and run psutil.Process().cpu_affinity().

I surely need psutils to find out the number of real cores (cpu_count(logical=False)).
Then I do some assumptions and calculate the number of available cores (see (2)).

An issue on cPython has some details about processor affinity on windows. (And lack of support of os.sched_getaffinity() on many systems.)
python/cpython#105972

This would give more information than nproc (of coreutils), which has no option for logical=false. (I would suggest it there if I knew where to.)

1: https://docs.python.org/3.11/library/os.html#interface-to-the-scheduler

2: code to get the number of available physical cores (doing some assumptions):

import psutil

def cpu_count_sched(logical = True):
    """
    Returns the number of logical (or phsical) cores available to the current process.
    
    Not all cores might be available to a process. See os.sched_getaffinity().
    """
    aff = psutil.Process().cpu_affinity()
    if logical:
        return len(aff)
    n_log = psutil.cpu_count(logical=True)
    n_phys = psutil.cpu_count(logical=False)
    if n_log == len(aff) or n_log == n_phys:
        # all (logical) cores enabled or 
        # each physical core has only one logical core
        return n_phys
    # assume logical cores on one physical core are after each other
    # and that all physical cores are the same
    n_scheduled_cores = 0
    n_log_per_core = n_log // n_phys
    aff = set(aff)
    for log_core_id in range(0, n_log, n_log_per_core):
        if aff & set(range(log_core_id, log_core_id + n_log_per_core)):
            n_scheduled_cores += 1
    return n_scheduled_cores

[orenbenkiki]

Getting the number of the physical cores is just half the battle. One also needs to know which logical cores are contained in each of the physical cores (to manage affinity). I initially proposed enhancing cpu_affinity but as @giampaolo said, perhaps this is better served using a separate cpu_topology.