[ENH]: Allow for dumping data object to disk for debugging purposes

[synchon]

Are you requiring a new dataset or marker?

• I understand this is not a marker or dataset request

Which feature do you want to include?

The possibility to set a set of configuration flags that dumps the data object into a directory (configurable) either at the end of preprocessing or minor step.

How do you imagine this integrated in junifer?

I see a few env vars/config flags (ConfigManager) that make junifer dump the data object:

• JUNIFER_PREPROCESSING_DUMP_LOCATION: which takes a path. Within the path, a directory for the element will be created and data dumped inside
• JUNIFER_PREPROCESSING_DUMP_GRANULARITY: "final" or "full". If "final", only after preprocessing. if "full", it will be dumped before preprocessing (output from data reader) and after each preprocessing step.

I see this implemented in a class DataObjectDumper which also allows to register functions that allows to save each class in a native format (instead of using pickle). Example: if its a nifti image, don't use pickle for that field, but save as .nii.gz. This should be extensible as it might be needed for EEG data too.

Do you have a sample code that implements this outside of junifer?

Anything else to say?

No response

[codecov]

Codecov Report

❌ Patch coverage is 91.11111% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.29%. Comparing base (ee99ec6) to head (8540d98).
⚠️ Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
junifer/pipeline/_data_object_dumper.py	90.00%	12 Missing and 4 partials ⚠️

❌ Your patch status has failed because the patch coverage (91.11%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #452      +/-   ##
==========================================
- Coverage   91.30%   91.29%   -0.02%     
==========================================
  Files         135      136       +1     
  Lines        5588     5764     +176     
  Branches      910      941      +31     
==========================================
+ Hits         5102     5262     +160     
- Misses        313      325      +12     
- Partials      173      177       +4     

Flag	Coverage Δ
docs	100.00% <ø> (ø)
junifer	91.28% <91.11%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
junifer/api/decorators.py	100.00% <100.00%> (ø)
junifer/pipeline/marker_collection.py	100.00% <100.00%> (ø)
junifer/typing/_typing.py	100.00% <100.00%> (ø)
junifer/pipeline/_data_object_dumper.py	90.00% <90.00%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

PR Preview Action v1.6.2
Preview removed because the pull request was closed.
2025-07-31 14:55 UTC

[fraimondo]

Excellent!