Skip to content

Add deeplense_npy_loader.py with robust loader for inconsistent .npy formats#179

Open
abhiram123467 wants to merge 4 commits into
ML4SCI:mainfrom
abhiram123467:dataset-utils-fix
Open

Add deeplense_npy_loader.py with robust loader for inconsistent .npy formats#179
abhiram123467 wants to merge 4 commits into
ML4SCI:mainfrom
abhiram123467:dataset-utils-fix

Conversation

@abhiram123467
Copy link
Copy Markdown

Fixes #178

Problem

DeepLense .npy files exist in 3 different formats causing
ValueError for new contributors loading the dataset.

Solution

Added deeplense_npy_loader.py with:

  • load_npy_image() — handles all 3 .npy formats automatically
  • LensingDataset — clean PyTorch Dataset class with train/val split

Formats handled

  1. Plain (64,64) float arrays
  2. Object arrays of row vectors
  3. Flat (4096,) arrays with label row appended

Testing

Tested on all 3 classes (no_sub, cdm, axion)
with ~90,000 files on Google Colab T4 GPU ✅

Added a robust loader for DeepLense .npy files that supports multiple formats and includes a PyTorch Dataset class for easy data handling.
Added a robust loader for DeepLense .npy files that handles multiple formats, along with a PyTorch Dataset class for easy data management.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add robust dataset loader utility for inconsistent .npy formats

1 participant