array_tofile does not close file when PyArray_ToFile fails

[tim-mitchell]

When an open file is passed to numpy.save and the save fails, then file is then undeletable because numpy still has the file open.

Reproducing code example:

Firstly create a small partition F that has < 200Mb of space

import numpy as np
import os

a = np.zeros((100000000, 3), dtype=float)
filename = 'F:\\array.npy'
try:
    with open(filename, 'wb') as f:
        np.save(f, a)
except Exception as e:
    if os.path.exists(filename):
        os.remove(filename)  # This fails

Error message:

Traceback (most recent call last):
File "S:/scripts/numpy_bug.py", line 9, in
np.save(f, a)
File "<array_function internals>", line 6, in save
File "S:\Leapfrog\components\py3_win64\lib\site-packages\numpy\lib\npyio.py", line 553, in save
pickle_kwargs=pickle_kwargs)
File "S:\Leapfrog\components\py3_win64\lib\site-packages\numpy\lib\format.py", line 687, in write_array
array.tofile(fp)
OSError: 300000000 requested and 0 written # We expect this
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 1, in
File "C:\Program Files\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev_pydev_bundle\pydev_umd.py", line 197, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "C:\Program Files\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev_pydev_imps_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "S:/scripts/numpy_bug.py", line 12, in
os.remove(filename)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'F:\array.npy'

NumPy/Python version information:

1.18.1 3.6.6 (v3.6.6:4cf1f54eb7, Jun 27 2018, 03:37:03) [MSC v.1900 64 bit (AMD64)]
1.19.2 3.8.0 (tags/v3.8.0:fa919fd, Oct 14 2019, 19:37:50) [MSC v.1916 64 bit (AMD64)]

[nagesh-chowdaiah]

import numpy as np
import os

a = np.zeros((100000000, 3), dtype=float)
filename = 'test.py'
try:
with open(filename, 'wb') as f:
np.save(f, a)
except Exception as e:
if os.path.exists(filename):
os.remove(filename) # This fails

I was able to successfully run this code

[nagesh-chowdaiah]

what's the issue here? is it something related to os

[tim-mitchell]

@nagesh-chowdaiah Yes - it runs fine on any drive with enough space. But if you Firstly create a small partition F that has < 200Mb of space or create any situation where PyArray_ToFile returns False it reproduces every time

[mattip]

Thanks for the report. It seems we do not clean up on failure.

The tricky part for writing a PR will be to make a test that can reliably return -1 from PyArray_ToFile. I think modifying the code to fail writing an object array also exposes the issue

import numpy as np
import os

a = np.zeros((10), dtype=object)
filename = 'array.npy' # in a test use tmp_path / filename to create this in  a temporary directory
try:
    with open(filename, 'wb') as f:
        a.tofile(f, sep='')
except Exception as e:
    if os.path.exists(filename):
        os.remove(filename)  # This fails