Memory leak when using MemoryFile

[valpesendorfer]

Memory used with MemoryFile doesn't seem to be released after the reference is gone. When creating MemoryFiles iteratively, the memory consumption continues to increase with every iteration.

Seems to be an issue with version 1.1.5 - works as expected with version 1.1.4.

Expected behavior and actual behavior.

Expected:

Memory to be released after reference is gone, no memory increase in iteration.

Actual:

Memory is not released, consumed memory increases with every iteration.

Comparison between 1.1.4 and 1.1.5:

Steps to reproduce the problem.

import psutil

import numpy as np
import rasterio

def main():

    profile = dict(
        count=1,
        height=2000,
        width=7200,
        nodata=-9999,
        dtype='int16')

    for it in range(100):

        arr = np.random.randint(0, 100, size=(2000,7200), dtype='int16')

        with rasterio.MemoryFile() as memf:
            with memf.open(driver='GTiff', **profile) as dataset:
                dataset.write(arr, 1)

        del memf
    
        mem_pct = psutil.virtual_memory().percent
        print(f'Memory used: {mem_pct}%')


if __name__ == '__main__':
    main()

Operating system

NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"

Rasterio version and provenance

pip freeze

affine==2.3.0
attrs==19.3.0
click==7.1.2
click-plugins==1.1.1
cligj==0.5.0
numpy==1.19.0
psutil==5.7.0
pyparsing==2.4.7
rasterio==1.1.5
snuggs==1.4.7

rio wheel from pypi:

rasterio-1.1.5-cp37-cp37m-manylinux1_x86_64.whl

pip:

pip 19.2.3

python:

Python 3.7.7

[sgillies]

Thanks for the report @valpesendorfer . We did change the implementation in 1.1.5. I'll look into it soon.