000728 - one file has truncated ophys data

[rly]

In dandiset 000728, the file
"sub-691657859/sub-691657859_ses-712919679-StimB_ophys.nwb" is 3 MB instead of the ~33-55 GB norm for all other files that do not contain "behavior+image" in the file name.

This file has only 10 frames in each of the acquisition objects. The partner "behavior+image" file contains processed ophys data with 113,861 frames. I suspect this was a stub test conversion that was upload by mistake. The file creation date is also earlier than the others by about a month (2024-03-16 vs 2024-04-12).

[bendichter]

Confirmed, and your read was exactly right — it was a stub test conversion uploaded by mistake. I tracked down why it happened and why it was never replaced. Full write-up, verification script, and fix: https://gist.github.com/bendichter/62fd0760ebcaa0c7711162bb0ea29fc3

It's exactly one file out of 1518. A scan of all 3036 assets found no other truncated file. The next smallest raw asset is 32.8 GB, so the gap is four orders of magnitude — no ambiguity and no second suspect. The processed behavior+image+ophys files are all fine; their 119 MB–1.05 GB spread is ROI count, not truncation (I checked the three smallest directly — each holds a full recording with only 1–2 ROIs).

The expected frame count is 113,861, confirmed by two independent sources that agree exactly: the partner processed file's DfOverF traces (113861, 10), and the Allen source movie s3://allen-brain-observatory/visual-coding-2p/ophys_movies/ophys_experiment_712919679.h5, which is (113861, 512, 512) uint16 / 59.7 GB. So 99.99% of the raw imaging data for this session is missing, and the file should be ~38 GB.

Root cause — three things in visual-coding-to-nwb-v2 lined up:

1. stub_test=True in _two_photon_series.py slices both acquisition objects to [:10], which reproduces the observed (10, 512, 512) and (10, 2) shapes exactly. The chunking corroborates it: the 10 MB chunk target yields 19 frames/chunk in a real file (and the healthy siblings do have chunks=(19, 512, 512)), but collapses to (10, 512, 512) with only 10 frames — which is what this file has.

2. safe_download_convert_and_upload_raw_session.py had session_id = "712919679" hardcoded as its __main__ default, and its last step is an unconditional automatic_dandi_upload with no validation in between. The raw conversion scripts landed 2024-03-15 (Reordering list of authors #35, Will we always need to use a terminal to upload data to DANDI? #36); this file's file_create_date is 2024-03-16T05:58Z, the day after. Bulk conversion didn't start until ~2024-04-12 — which is the date discrepancy you noticed.

3. This is the part that made it permanent: convert_raw_dataset.py decides what still needs converting purely from whether an asset exists at the expected path. Because the 3 MB stub already occupied that path, session 712919679 counted as completed. Replaying that exact logic against the current asset list:

sessions with a processed file : 1518
sessions counted as 'completed': 1518
sessions the driver would still convert: 0 []
Is 712919679 counted as completed? True
Would the driver ever revisit it?     False

The dataset looks 100% complete to its own tooling, which is why this survived the full conversion run, publication, and validation.

Fix. The gist has three things:

• verify_000728_integrity.py — scans all 3036 assets in ~6 s over HTTP range requests, no bulk download, exits nonzero if anything is truncated.
• repair_session_712919679.py — re-converts and re-uploads the session, validating the frame count against the Allen source movie before uploading. The upload replaces the asset at the same path, so no metadata or path changes are needed. Needs DANDI_API_KEY with write access to 000728, ~100 GB scratch disk, and a few hours; --dry-run converts and validates without uploading.
• prevent_recurrence.patch — upstream fix for both gaps: a validation guard on both upload paths, removal of the hardcoded test session, and a size floor so a partial upload gets re-converted instead of masking the real one forever. Applies cleanly to main and passes the repo's black config.

I haven't run the re-upload — it's a ~38 GB write to a published Allen Institute dandiset, so that should be someone with write access on 000728 deciding to pull the trigger. @CodyCBakerPhD, want me to open the patch as a PR against visual-coding-to-nwb-v2 and run the reconversion, or would you rather take it from here?

[bendichter]

Opened the patch as a PR: catalystneuro/visual-coding-to-nwb-v2#48.

It fixes the two code-level gaps — the missing validation between conversion and upload, and the completed-session check that let the stub mask the real conversion indefinitely. With both in place, re-running convert_raw_dataset.py picks up 712919679 on its own; I verified that against the live asset list (unpatched: 0 sessions to convert, patched: exactly ['712919679']).

Still outstanding is the actual reconversion and re-upload of the ~38 GB file, which needs someone with write access on 000728. @CodyCBakerPhD happy to run it if you'd like — just say the word.

[CodyCBakerPhD]

@bendichter You have always been an owner on 000728

[bendichter]

yeah I just wanted to get on the same page before making the correction since you were the lead on this

[rly]

Could one of you please convert and upload the fixed file, and re-publish the dandiset? It's blocking me on a Zarr conversion of that data and would also just be good to fix. Thanks!