Skip to content

WIP: memory checks, chunking, and new setting#66

Merged
TimMonko merged 10 commits intondev-kit:mainfrom
TimMonko:enh/memory-checks
Apr 5, 2026
Merged

WIP: memory checks, chunking, and new setting#66
TimMonko merged 10 commits intondev-kit:mainfrom
TimMonko:enh/memory-checks

Conversation

@TimMonko
Copy link
Copy Markdown
Member

@TimMonko TimMonko commented Apr 4, 2026

References and relevant issues

When trying to load an 18MB (on disk) int32 label image I noticed a few things:

  1. it's in-memory size is >8GB (data.nbytes)
  2. it was being read as an image, despite its name containing labels, which I want for it to be read as a Label
  3. it both loads in REALLY slowly (as in memory, because its uncompressing) and if it was lazy loaded...
  4. slice-by-slice navigation of a TIFF was excruciatingly slow. I realizing this was because it was chunking by ZYX so each call to the slice was having to uncompress the entire file each time.

Description

  1. Detect labels from file names, if channel names aren't enough
  2. Change memory check to be the would-be in memory size, not the on disk size
  3. use YX for chunking, not ZYX, so usual slice-by-slice in napari is much quicker (slicing by X or Y though will be then really slow, this is still better than before)
  4. A new setting allowing users to control the in-memory limit for RAM

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 4, 2026

Codecov Report

❌ Patch coverage is 97.93814% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/ndevio/nimage.py 97.70% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

@TimMonko TimMonko marked this pull request as ready for review April 4, 2026 21:34
@TimMonko TimMonko merged commit 0451832 into ndev-kit:main Apr 5, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant