scdl neighbor update

[camirr-nv]

Description

This PR introduces comprehensive neighbor sampling functionality to the Single Cell Data Loader (SCDL), enabling efficient retrieval of cell neighbors for graph-based machine learning workflows and spatial analysis tasks.

Core Functionality

• Neighbor data loading: Automatic detection and loading of neighbor information from .neighbors files
• Flexible neighbor sampling: Support for random sampling strategies with configurable neighbor counts
• Memory-efficient implementation: Lazy loading of neighbor data with proper memory management

Public API Methods

• get_row_with_neighbor(): Retrieve cell data with a single randomly sampled neighbor
• get_row_padded_with_neighbor(): Get cell data with multiple neighbors, padded to specified length
• get_neighbor_data(): Direct access to neighbor information for a given cell
• get_neighbor_stats(): Statistical analysis of neighbor connectivity

Internal Implementation Methods

• _init_neighbor_args(): Initialize neighbor-related configuration and validate parameters
• _extract_neighbor_data(): Extract and process neighbor data from memory-mapped files
• get_neighbor_indices_for_cell(): Retrieve all neighbor indices for a specific cell
• sample_neighbor_index(): Efficiently sample random neighbor indices with validation

File Structure

• Neighbor data stored in three memory-mapped .npy files:
  -- neighbor_indices.npy: Contains neighbor cell indices in CSR format
  -- neighbor_indptr.npy: Index pointers for CSR sparse matrix structure
  -- neighbor_values.npy: Weights/values associated with neighbor relationships
• Data extracted from AnnData's .obsp sparse matrices (e.g., 'next_cell_ids')
• Automatic file discovery and validation during dataset initialization

Type of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Refactor
• Documentation update
• Other (please describe):

CI Pipeline Configuration

Configure CI behavior by applying the relevant labels:

• SKIP_CI - Skip all continuous integration tests
• INCLUDE_NOTEBOOKS_TESTS - Execute notebook validation tests in pytest
• INCLUDE_SLOW_TESTS - Execute tests labelled as slow in pytest for extensive testing

Note

By default, the notebooks validation tests are skipped unless explicitly enabled.

Authorizing CI Runs

We use copy-pr-bot to manage authorization of CI
runs on NVIDIA's compute resources.

• If a pull request is opened by a trusted user and contains only trusted changes, the pull request's code will
  automatically be copied to a pull-request/ prefixed branch in the source repository (e.g. pull-request/123)
• If a pull request is opened by an untrusted user or contains untrusted changes, an NVIDIA org member must leave an
  /ok to test comment on the pull request to trigger CI. This will need to be done for each new commit.

Usage

TODO: Add code snippet

Pre-submit Checklist

• I have tested these changes locally
• I have updated the documentation accordingly
• I have added/updated tests as needed
• All existing tests pass successfully

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[polinabinder1]

On a high level - test cases!

[polinabinder1]

/ok to test c6fa91e

[skothenhill-nv]

Left mostly style and documentation related comments. One thing I noticed is from a top down reading, its not clear what the expected behavior is when neighbor data is requested but doesnt exist. I saw a few methods where an exception is raised, and others return a boolean or maybe throw a warning. If you feel like you have the control flow in a good place for incomplete neighbor data then no worries- but if not it might be worth taking another pass. Approved for now, I trust youll merge when comments are addressed.

[polinabinder1]

Remove the 3rdParty changes - they should be the same as in main.

[camirr-nv]

Remove the 3rdParty changes - they should be the same as in main.

@polinabinder1 I'm not sure why, but I realize at some point when i initialized the submodules they showed up as changes to be committed under 'git status' and I did. I just reverted that commit.

[camirr-nv]

/ok to test c562990

[camirr-nv]

/ok to test 6943412

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 86.74699% with 22 lines in your changes missing coverage. Please review.

Project coverage is 84.24%. Comparing base (2aac67e) to head (402bc66).

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
.../src/bionemo/scdl/io/single_cell_memmap_dataset.py	85.80%	22 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #843      +/-   ##
==========================================
+ Coverage   84.18%   84.24%   +0.06%     
==========================================
  Files         144      144              
  Lines        9229     9394     +165     
==========================================
+ Hits         7769     7914     +145     
- Misses       1460     1480      +20     

Files with missing lines	Coverage Δ
...dl/src/bionemo/scdl/util/torch_dataloader_utils.py	100.00% <100.00%> (ø)
.../src/bionemo/scdl/io/single_cell_memmap_dataset.py	90.02% <85.80%> (-2.04%)	⬇️

... and 1 file with indirect coverage changes

[camirr-nv]

/ok to test 402bc66