np.memmap works differently on windows and linux/mac

[lstolcman]

Hi. I have a big array (shapes 40k x 40k and bigger), and I want to use memmap and multiprocessing to fill it. I coded simple example to test it, and it does not work on windows. Expected result is to have values saved by each process (see screenshots).

Any suggestions?

Valid result (tested on virtualboxed xubuntu and MacOS):

Invalid result - array empty (Windows 7):

Reproducing code example:

import os
import numpy as np
import time
import multiprocessing as mp


def child(v, lock, m, p):
    np.random.seed(int.from_bytes(os.urandom(4), byteorder='little'))
    while True:
        vval = 0
        with lock:
            if v.value > 0:
                v.value -= 1
                vval = v.value
            else:
                break
        sl = np.random.rand()
        print(f'p{p} sleeping for {sl}')
        time.sleep(sl)
        m[0, vval] = p
        print(f'proc {p}: memmap[{vval}]={p} memmap={m}')



if __name__ == '__main__':
    memmap = np.memmap(
        'test1.memmap',
        dtype='int',
        mode='w+', shape=(1,10)
        )

    value = mp.Value('i', 10)
    lock = mp.Lock()
    procs = [mp.Process(
        target=child,
        args=(value, lock, memmap, i)
        ) for i in range(1,3)]
    for proc in procs:
        proc.start()
    for proc in procs:
        proc.join()
    
    print(f'after join, result:')
    print(memmap)

Numpy/Python version information:

windows: 1.16.2 3.7.1 (default, Dec 10 2018, 22:54:23) [MSC v.1915 64 bit (AMD64)]
linux: 1.16.2 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0]

[eric-wieser]

A much simpler repro:

import numpy as np
import multiprocessing as mp


def child(m, p):
    m[p] = 10 + p
    print("In process {}: {}".format(p, m))


if __name__ == '__main__':
    memmap = np.memmap(
        'test1.memmap',
        dtype='int',
        mode='w+', shape=(2,)
        )

    procs = [
        mp.Process(
            target=child,
            args=(memmap, i)
        ) for i in range(2)
    ]
    for proc in procs:
        proc.start()
    for proc in procs:
        proc.join()

    print('after join, result:')
    print(memmap)

[eric-wieser]

It seems the underlying problem is that the memmap._mmap attribute is None in the child process, yet set in the parent.