Potential memory leak while using shared memory

[seraphlivery]

BPO	41587
Nosy	@tim-one, @ericvsmith

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-08-19.18:53:55.357>
created_at = <Date 2020-08-19.09:49:29.135>
labels = ['3.8', 'library', 'invalid', 'performance']
title = 'Potential memory leak while using shared memory'
updated_at = <Date 2020-08-19.18:53:55.356>
user = 'https://bugs.python.org/seraphlivery'

bugs.python.org fields:

activity = <Date 2020-08-19.18:53:55.356>
actor = 'seraphlivery'
assignee = 'none'
closed = True
closed_date = <Date 2020-08-19.18:53:55.357>
closer = 'seraphlivery'
components = ['Library (Lib)']
creation = <Date 2020-08-19.09:49:29.135>
creator = 'seraphlivery'
dependencies = []
files = []
hgrepos = []
issue_num = 41587
keywords = []
message_count = 10.0
messages = ['375642', '375648', '375649', '375650', '375652', '375655', '375658', '375659', '375667', '375669']
nosy_count = 3.0
nosy_names = ['tim.peters', 'eric.smith', 'seraphlivery']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue41587'
versions = ['Python 3.8']

[seraphlivery]

We find an issue while using shared memory. When opening another process to overwrite the shared memory, the memory of this process will increase to about the size of this shared memory. So when several processes try to read or write the shared memory, there will be N times memory usage.

import os, psutil
import numpy as np
from multiprocessing import Process, shared_memory

SHM_SIZE = 100000 * 30 * 20


def f(name):
    print('[Sub] (Before) Used Memory of {}: {} MiB'.format(
        os.getpid(),
        psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024,
        ))
    shm = shared_memory.SharedMemory(name=name)
    b = np.ndarray(shape=(SHM_SIZE, 1), dtype=np.float64, buffer=shm.buf)
    for i in range(SHM_SIZE):
        b[i, 0] = 1
    print('[Sub] (After) Used Memory of {}: {} MiB'.format(
        os.getpid(),
        psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024,
        ))


def main():
    print('[Main] Used Memory of {}: {} MiB'.format(
        os.getpid(),
        psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024,
        ))
    shm = shared_memory.SharedMemory(create=True, size=8*SHM_SIZE, name='shm')
    p = Process(target=f, args=('shm',))
    p.start()
    p.join()
    print('[Main] Used Memory of {}: {} MiB'.format(
        os.getpid(),
        psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024,
        ))
    input()
    shm.close()
    shm.unlink()


if __name__ == '__main__':
    main()

[ericvsmith]

It's likely that the same memory is being counted by both processes, to the output is misleading. Shared memory is notoriously difficult to allocate per-process. For example, it's definitely true that the shared memory is consuming virtual address space in both processes. That's often the value that's reported. But since the actual memory is shared, it's not consuming more physical memory than expected.

[seraphlivery]

You mean if I have a machine that has 16GB RAM, and the maximum shared memory size is 8GB. I then create two processes to write to this shared memory, and the system won't run out of memory. Is it right?
I can try this experiment later on. But I can not understand this. I can see this memory increment in htop. I think the memory stat from htop is correct, otherwise, any system monitor software could deliver a fake warning message about the memory.

[ericvsmith]

You'll have to play with it. I'm just saying that it's a very complicated subject, and not as simple as asking how much memory an individual process is using. For example, see https://www.howtogeek.com/659529/how-to-check-memory-usage-from-the-linux-terminal/ for the many statistics that are involved.

[seraphlivery]

Okay. I know this is complicated. So how can I make sure this is not an issue? Can you provide some steps or a bash script to prove that the memory increment issue does not exist? I'm now not being persuaded because I don't know how to prove there is no issue about the memory. Will the experiment in the last message prove it? Or maybe you can provide an simple example showing there is no memory issue. Thank you.

[ericvsmith]

Sorry, I don't have any particular suggestion other than accounting for all virtual, shared, and physical memory of all types, and seeing how they're being used and allocated per-process by the various tools.

There are probably guides for this on the internet, but I haven't been able to find any with a quick search.

I'm not saying there isn't a problem, but I'm saying it's going to require more analysis before we can verify that a problem does exist.

[seraphlivery]

Sorry, I don't know what I can do about it. In my perspective, I think there is a memory leak because monitor software have provided proof for me to believe that.
I have provided a script to reproduce this issue. I think that is enough for developers to conduct other tests to verify whether this issue does exist. I don't think I can provide any more information since you can reproduce this problem.
As a user, I just need some advice about how to fix this issue or confirmation about this issue, or some explanations that make me believe that there is no issue.

[ericvsmith]

Okay. We'll see if someone else can provide more info.

[tim-one]

There's no evidence of a Python issue here, so I recommend closing this. It's not the Python bug tracker's job to try to make sense of platform-specific reporting tools, which, as already explained, can display exceedingly confusing numbers. We (the Python project) didn't write them, have no control over what they display, and have no special insight into them.

Here's a similar simple program that creates a 4 GiB block of shared memory, and passes it to 16 processes. I'm running this on Windows 10, on a machine with 4 physical cores and 16 GiB of RAM. If the memory weren't actually being shared, Windows would have killed the job (there's no "over allocation" allowed at all on Windows), because there's nowhere near 4 * 17 = 68 GiB of RAM available.

Instead, the program brings the machine to its knees, but because it created 16 processes each of which is 100% CPU- and memory-bound. The Windows task manager detailed view shows less than a MiB of non-shared memory in use by each of the worker processes, with the 4 GiB in the "memory in use that can be shared with other processes" column for each worker process. I'm running lots of other stuff too, and Windows still says there's over 5 GiB of my 16 GiB of RAM free for use.

Windows reporting has its own quirks. For example, the shared memory block doesn't show up at all in the workers at first. Instead the amount reported steadily increases, as the write loop in each worker process forces the OS to materialize shared pages in the worker's virtual address space (that is, Windows reports pages actually in use by a process, not virtual address space reservations).

from multiprocessing import Process, shared_memory

SHM_SIZE = 2 ** 32 # 4 GiB

def f(name):
    shm = shared_memory.SharedMemory(name=name)
    print(f"shm of size {shm.size:,}")
    buf = shm.buf
    for i in range(shm.size):
        buf[i] = 123
    shm.close()

def main():
    shm = shared_memory.SharedMemory(create=True, size=SHM_SIZE, name='shm')
    ps = []
    for i in range(16):
        p = Process(target=f, args=('shm',))
        p.start()
        ps.append(p)
    for p in ps:
        p.join()
    shm.close()
    shm.unlink()

if __name__ == '__main__':
    main()