Skip to content

Avoid deep copy of numpy buffers on unspill#74

Merged
crusaderky merged 4 commits into
dask:mainfrom
crusaderky:File_bytearray
Dec 28, 2022
Merged

Avoid deep copy of numpy buffers on unspill#74
crusaderky merged 4 commits into
dask:mainfrom
crusaderky:File_bytearray

Conversation

@crusaderky
Copy link
Copy Markdown
Collaborator

@crusaderky crusaderky commented Dec 27, 2022

Note that, with this PR, distributed 2022.01.1 will be backwards compatible with zict 2.2.0, but distributed 2022.12.1 won't work with zict 2.3.0. There is precedent for this; the consensus in the previous instance was that this should not be an issue for most users.

Benchmark

import gc
import numpy
import zict
from distributed.spill import SpillBuffer

print(zict.__version__)
a = numpy.random.random(512 * 2**20 // 8)
buf = SpillBuffer("/tmp", target=1)
buf["xxx"] = a
buf.slow.cache.clear()  # Clear weakref cache
buf["xxx"]
buf.get_metrics()

Output:

2.2.0
{...
 'unpickle_time_total': 0.07338492600138125}
2.3.0
{...
 'unpickle_time_total': 0.00016480999875057023}

@crusaderky crusaderky changed the title Avoid deep-copy of numpy arrays upon unspill Avoid deep copy of numpy arrays upon unspill Dec 27, 2022
@crusaderky crusaderky self-assigned this Dec 27, 2022
@crusaderky crusaderky changed the title Avoid deep copy of numpy arrays upon unspill Avoid deep copy of numpy buffers upon unspill Dec 27, 2022
@crusaderky crusaderky changed the title Avoid deep copy of numpy buffers upon unspill Avoid deep copy of numpy buffers on unspill Dec 27, 2022
@crusaderky crusaderky marked this pull request as ready for review December 27, 2022 21:41
@mrocklin
Copy link
Copy Markdown
Member

No objection to this or the related PR in distributed. Feel free to merge at will.

@crusaderky crusaderky merged commit c61ad64 into dask:main Dec 28, 2022
@crusaderky crusaderky deleted the File_bytearray branch December 28, 2022 12:09
@crusaderky crusaderky mentioned this pull request Mar 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deep copy causes memory flare on unspill

2 participants