Repository navigation
Releases: HallquistLab/BrainGnomes
Release list
1.0
-
Preserve integer NIfTI storage types after interpolation, regression, and
filtering. Recalculate slope and intercept from the processed range and round
to the output grid, avoiding fractional-value truncation, constant-output
scaling failures, clipping, and narrowing integer intermediates. Reserve
RNifti's INT32 missing-value code and reject non-finite integer output before
writing a file. Regression tests cover all eight integer types, NIfTI-1/2,
input scaling, expanded ranges, zero backgrounds, and extreme values. -
Decode integer images into double-precision working buffers and preserve
FLOAT64 results without an intermediate FLOAT32 cast. This retains small
differences at large offsets and supports NIfTI-2 scaling outside FLOAT32's
range. Integer working buffers use twice the memory of the previous FLOAT32
buffers; conversion temporaries are now freed instead of leaked. -
Suppress RNifti pointer/reference-count diagnostics in all native image
helpers, including debug builds where pkgbuild overrides-DNDEBUG.
Preserve normal BrainGnomes logging and other debug settings. Cleanly rebuild
the package and create new runtime bundles to update future workers; existing
sealed bundles retain their installed binary. -
Correct spline interpolation when retained timepoints are constant:
censored spikes and edge samples now receive the retained value. Reject
non-finite interpolation points explicitly. -
Apply fitted regression models to censored volumes even when fitted rows are
constant. Respect partial regression that retains the intercept, including
constant input series. Recenter residuals over fitted rows before applying
a requested nonzero mean in both image and matrix regression. -
Support scalar-gain filters without invalid memory access, including
steady-state initialization. Validate empty/non-finite coefficients and
padding before processing, including all-zero images. Filter and demean
nonzero constant signals when requested. -
Reject images with extra dimensions instead of silently processing their
first block, and use size-safe voxel indices in native image operations. -
Reject missing regression selections, non-finite designs, and non-finite
retained quantile voxels with clear errors. Quantiles continue to permit
non-finite values outside a supplied mask. -
Detect configuration changes inside unnamed, mixed-name, and duplicate-name
lists so confirmed saves retain those edits. -
Allow SQLite reads for job status, project inspection, submission history,
and table existence to wait up to ten seconds for concurrent writers rather
than failing immediately while a worker publishes its state. -
Correct menu titles and single-choice selection indices. Reject partial
numeric choices and exit menu loops on EOF. Escape, Ctrl+D, and EOF cancel
typed prompts cleanly and restore terminal settings. Cancelling guided
stream selection stops before submission instead of selecting every stream. -
Preserve negative CLI values, empty assignments, and trailing equals signs.
Keep leading zeros in subject/session identifiers and reject invalid boolean
options before acting. -
Preserve literal shell-quoted paths containing spaces or apostrophes across
CLI plans/setup/validation and worker entry points, while retaining vector
option parsing. -
Parse complete BIDS filename entities instead of incidental substrings, and
preserve alphanumeric resolution labels when constructing filenames. -
Validate motion-filter flags and require finite motion-QC thresholds before
summarization. -
Document CLI activation after installation, correct the quickstart's Bash
activation command, and clarify guided and prompt-free project creation.
Update native function documentation for validation, mean handling, storage
precision, and quantization behavior. -
Fix the missing quickstart pipeline diagram on the pkgdown website by
referencing a vignette-local PNG that is copied into the rendered article.
Installed vignette sources fall back to the existing package image. -
Correct reversed temporal-filter examples, zero-padded CompCor and motion
column selections, AROMA mode support and its fMRIPrep removal version, and
intensity-normalization variance scaling. Clarify ROI masking, censoring,
minimum-voxel settings, local extraction, output-directory requirements, and
connectivity limits. Refresh container setup, offline TemplateFlow guidance,
debug-versus-dry-run behavior, scheduler polling defaults, article links,
and contribution instructions; finish the intermediate-file documentation
and make shell run-ID examples safe to copy after substituting the ID. -
Snapshot tracked worker scripts, helpers, and the BrainGnomes runtime into
sealed, reusable project/run-owned bundles. Package reinstalls no longer
change queued workers' startup code. Independent bootstrap handlers publish
failure receipts and record non-array startup failures without loading
BrainGnomes; external R dependencies and containers remain shared inputs. -
Register UUID submission attempts in SQLite before invoking Slurm, TORQUE,
or local execution, including dynamically submitted postprocessing arrays.
Workers can bind their own allocation IDs before submission returns, and
late acknowledgements preserve started or completed states. Retain uncertain
submissions, block duplicate launches of the same work unit and role, and
expose these attempts throughinspect_project()$submissions. Inspection
and scheduler refresh remain read-only and SQLite-first. -
Aggregate Slurm array-task accounting under the tracked array identifier,
preserving active and uncertain states until all observed tasks terminate.
Distinguish scheduler query failures from missing records. Do not infer success
from expired TORQUE records; verify retained completed jobs using exit status.
Retain confirmed terminal observations while waiting for other jobs to finish.
Scheduler refresh remains read-only and project summaries remain SQLite-first. -
Require tracked worker status updates to persist in SQLite or exit with an error.
Store completion status, timestamp, and output manifest atomically so failed
manifest writes cannot leave a falsely completed job in the tracking database.
Close SQLite connections after rejected queries as well as successful ones. -
Extend spatial smoothing checks to isotropic and anisotropic voxel spacings
of 1.8--4 mm and kernels 1.5--3 times geometric mean voxel spacing. Use smooth
directional FWHM response functions only where calibration and independent
validation meet the half-voxel error limit. Use operator replay for conditions
whose FWHM response cannot meet that limit, preserving full-run SUSAN
parameters and checking Gaussian output with the matching AFNI operator.
Replay compares 96 distributed volumes in bounded chunks, rejects incorrect
kernels and unchanged outputs, and records only compact QA metadata.
Automatic checks also replay in-domain outputs whose FWHM comparison fails,
retaining the failed comparison and the replay result separately in QA.
The current continuous coefficient model passes for template-masked-input
SUSAN; the other eight production contexts require replay. -
Calibrate SUSAN and Gaussian spatial validation for isotropic 2 mm images
with 3--8 mm kernels using separate resolution-specific fits. Verify those
fits on held-out subjects, including genuine 2 mm fMRIPrep outputs. Establish
2 mm error limits from internal cohort-transfer checks, capped at 1 mm, while
preserving coarser-resolution coefficients and limits. Reject unchanged
images even when their predicted change is smaller than the error limit. -
Resample template masks and atlases using reference header geometry without
loading the reference BOLD voxel data. -
Reject image objects, spatial arrays, and binary payloads before writing
provenance JSON, configuration snapshots, validation records, or QC inventories.
Keep image files out of provenance companion directories. -
Keep intensity-reference NIfTI voxel arrays out of derivative JSON; record
published companion paths and compact header geometry instead. Validate
intensity normalization across all volumes in bounded chunks to avoid
memory spikes on long runs. -
Match QC validation audits to missing outputs through filesystem aliases,
retaining their failed status instead of incorrectly marking them invalid.
Canonicalize missing-configuration diagnostics consistently across platforms.
Keep setup menu text portable and bound headless CLI tests so inherited
Windows consoles cannot stall the test suite. -
Resolve relative configuration paths from a colocated project YAML: the YAML
establishes the project root, all runtime and scheduler paths become
absolute, and project-contained paths are saved portably relative to that
root. Reject detached or mismatched project configurations before execution,
and remove unsafeproject_directorychanges from the ordinary field editor. -
Ask about motion filtering after optionally adding framewise displacement
during postprocessing setup, ensuring the saved stream records an explicit
motion-filter decision and can be passed directly torun_project(). -
Present temporal-filtering and intensity-normalization methods as descriptive
numbered menus during guided setup instead of requiring users to type internal
configuration values. Interactive prompts no longer insert an extra blank line
between their final instruction and the input marker. -
Allow
BrainGnomes setup_project(orinit) with no project name to open the
guided setup and prompt for the project name, project directory, and workflow
settings. Supplying a project name retains prompt-free creation and...
0.9-2
- Make
run_project()explain its pre-submission work in plain language, including subject discovery, run-record creation, and the first full read of a large container file. Large cohort submissions now report bounded progress through the subject list, and explicit subject filters are applied before session directories are inspected so small requested subsets do not scan the full project tree. - Extend the established
setup_project()->run_project()workflow across the complete R and command-line lifecycle without adding required setup steps. Optional inspection tooling now provides non-mutating configuration validation,doctor()preflight checks, and serializable execution plans; direct runs resolve the same execution model exposed by plans. Run handles, tracked-run views, log discovery, non-interactive diagnosis, failed-job retry planning, and guarded scheduler cancellation support observation and recovery. Flywheel controller snapshots are now run-specific so concurrent submissions cannot overwrite one another. - Record complete provenance before each run begins submission. Every run now retains its resolved request and subject scope, exact configuration, planned resources and dependencies, software/R/host/scheduler identity, and content fingerprints for selected containers and other execution-driving files.
get_run_provenance()and theBrainGnomes provenancecommand expose the record together with current job-tracking rows; plan and retry origins are retained explicitly. - Expand applied-user recovery documentation around the established workflow. The Quickstart and diagnosis guide now show how to select one run, inspect its failed jobs, logs, and provenance, preview a retry, submit it as a separate new run, decide whether to include downstream blocked work, and safely preview cancellation. Function and CLI help use the same plain-language behavior and safety guidance.
- Make the R recovery APIs preview-first as well:
retry_project_run()andcancel_project_run()now default todry_run = TRUE, and require an explicitdry_run = FALSEto submit or cancel work. Retry planning preserves postprocessing and extraction stream names containing underscores. - Add an executable motion-QC vignette for
calculate_motion_outliers(). It demonstrates raw and filtered FD summaries, strict threshold interpretation, unavailable-filter handling, run-level QC/exclusion exports, and the boundary between interactive QC summaries and postprocessing scrubbing. - Make guided
setup_project()andedit_project()saves use the same atomic configuration writer as non-interactive lifecycle tooling after the user confirms replacement of an existing file. - Recalibrate masked-SUSAN validation on real fMRIPrep BOLD data for the distinct no-input-mask, fMRIPrep-mask, and TemplateFlow-mask conditions. Validation now enforces the selected detrending-plus-MAD estimator, uses up to 96 timepoints deterministically distributed over the complete run (or all timepoints in shorter runs), reads only those volumes with RNifti, and cannot pass by extrapolating across input-mask, kernel-size, voxel-size, or sampling support. Calibration retains the full-run SUSAN threshold, temporal mean, and extents while estimating smoothness from the selected timepoints.
- Validate the promoted 3--8 mm masked-SUSAN models on independent fMRIPrep 25.2.5 derivatives and a 96-volume postprocessing E2E fixture drawn across the complete run. The 10 mm stress kernel remains outside the supported calibration range.
- Allow Slurm and PBS fsaverage setup to copy with GNU
cpwhen newer fMRIPrep containers do not providersync, while retaining the existingrsyncpath when present. - Strengthen postprocessing validation so masking is replayed exactly on 32 deterministic volumes distributed across the complete run (or every volume in shorter runs), interpolation preserves retained volumes and matches sampled natural-spline values, removed volumes match the censor vector in order, and AROMA/confound regression samples are deterministic, pre-selected, and spatially balanced across image resolutions.
- Make temporal-filter validation deterministic and pre-selected, require finite per-voxel stopband and passband evidence, verify that no-noise-IC AROMA output is actually unchanged, reject wholly invalid AROMA component requests, and fail every image validator when spatial NIfTI grid metadata change unexpectedly.
- Install
multitaperas a required runtime dependency so strict temporal-filter validation is available in ordinary package installations. Preserve target qform/sform matrices and coordinate-system codes when resampling TemplateFlow masks, and automatically rebuild incompatible cached masks rather than failing later validation. - Reduce peak memory use in
image_quantile()by avoiding a second full image copy and whole-image inclusion bitmap; masked quantile values and numerical behavior are unchanged. - Route postprocessing checks through a common validation runner: validator errors now obey the configured continue/stop policy, reused intermediates are checked, structured results are retained in a JSON audit beside the subject log, and final images remain staged until last-step validation completes.
0.9
- Document stage-specific runtime requirements in package metadata, the README, and Quickstart. Add an installed miniature project configuration and an executable local-onboarding vignette that clearly separates no-cluster helpers, submission-free dry runs, and scheduler/container-dependent project execution.
- Correct ROI-connectivity provenance and execution: estimator-specific filenames are now unique, scheduled extraction honors nested correlation settings,
cor.shrinkhas a stable BIDS entity, and time-series-only extraction supportscor_method = "none". - Preserve the one-to-one association between multiple postprocessing input streams and their BIDS descriptions during ROI extraction. Ambiguous vector lengths now fail explicitly instead of selecting or combining unintended inputs.
- Apply interactively configured ROI masks, preserve every atlas label when an ROI is fully masked, and return schema- and dimension-stable time-series, connectivity, and diagnostic outputs for empty ROIs and entirely masked atlases.
- Report per-stream ROI-extraction state in project and subject status. Scheduled extraction now writes an explicit manifest of its actual time-series, connectivity, and diagnostic outputs so completion checks do not depend on a directory-wide snapshot.
- Honor
save_ts = FALSEin scheduled extraction and reject contradictory extraction configurations before output is created. - Align the CLI,
run_project()help, examples, and vignettes with the seven supported submitted stages. BIDS validation remains project-configured but is submitted separately withrun_bids_validation(); stream selection and dry-run output now expose resolved settings. - Harden public and native interfaces with complete help signatures, working examples, stable empty data-frame schemas, finite image-quantile validation, zero-length and dimension checks, and clearer argument errors.
- Keep development-only calibration resources, local Codex files, audit reports, prior build products, and Python bytecode caches out of source packages and installed-package tests. Remove the obsolete
ROI_TempCorr.Rentry point after migrating its useful voxel-retention diagnostics into supported ROI extraction. - Stabilize empty-result contracts:
extract_bids_info(character())now returns its complete typed BIDS schema, andget_project_status()returns configured status columns even before any subject jobs exist.image_quantile()now rejects empty, missing, NaN, and infinite probability vectors before reading image data. - Improve onboarding and release hygiene: generate Quickstart CLI help from the installed command, document the standalone BIDS-validation boundary consistently, show resolved postprocessing and extraction stream settings during dry runs, and remove the obsolete
ROI_TempCorr.Rinstalled entry point after migrating its useful diagnostics intoextract_rois(). - Add optional per-ROI voxel-retention diagnostics to ROI extraction. Reports
separately track atlas size, optional-mask survival, BOLD-valid voxels,
minimum-voxel requirements, retention status, and exclusion reasons; the
scheduled extraction workflow includes requested diagnostics in its explicit
output manifest. - Preserve source FD in notch-filtered calculated confounds and write the
recomputed series immediately beside it asframewise_displacement_filtered.
The output respects the configured header setting and logs column order when
headers are disabled; filtering-based scrubbing and confound regression use
the filtered FD. Clearly reversed notch bounds are repaired automatically;
skipped filters are logged and never produce a misleading filtered-FD label.
When FD is selected as a processed confound, both the source-derived and
notch-derived FD columns retain all configured BOLD-matched confound
processing. - Add a
voxel_pscintensity-normalization mode that uses the existing robust reference-core and eligible-frame policy and applies denominator-guarded baseline-to-100 scaling after spatial processing. Reliable local baselines use ordinary PSC scaling, very low positive baselines use a lower denominator bound, and invalid baselines or those with too few eligible frames use a conservative run-level fallback. The guards do not clip observations or mask voxels; the user'sapply_maskdecision is preserved, and the multiplier map and guard counts are saved for provenance. Guard counts and percentages within the conservative automask are logged at info level, with complete-grid counts at debug level. - Add a user-oriented intensity-normalization vignette documenting the
targetconvention, robust reference-core policy, provenance outputs, QA, and limitations. - Replace
automask()'s background-sensitive positive-voxel quantile interpolation with an iterative AFNI-style clip estimator and a smoothly varying local threshold field. - Match AFNI's
automask()peeling more closely with a 17-of-18 NN2 survival rule, layer-aware restoration, and post-peel face-connected reclustering. - Replace postprocessing's late 4D-median intensity estimate with an automask-based robust reference core selected from the original positive-scale BOLD image; measure and apply the run factor after masking/smoothing but before AROMA, temporal filtering, confound regression, or timepoint removal, and save the core mask and JSON provenance.
- Accept
postprocess/intensity_normalize/targetas the simplified normalization setting while retainingglobal_medianas a backward-compatible alias. - Preserve each voxel's pre-AROMA temporal mean during both aggressive and non-aggressive AROMA denoising, retaining the positive baseline intensity used for cross-run scaling.
- Refactor postprocessing to use job arrays and sentinels for cleanup
- Add additional templates to prefetch needed by MRIQC
- Preserve user-specified
metadata/sqlite_dbvalues and exposesqlite_dbinedit_project(). - Clean postprocessing scratch workspaces and temporary automask files on errors as well as successful exits.
- Use exit-time cleanup for temporary FSL postprocessing files generated during temporal filtering, smoothing, confound regression, and brain-mask computation.
- Add regression tests for editable SQLite database configuration and postprocessing temp-file cleanup after failures.
- Prefetch resolution-1 T1w and brain-mask assets used by fMRIPrep anatomical reports, including when an output space explicitly requests another resolution.
- Recalibrate spatial-smoothing validation on held-out BOLD runs from three datasets and resolutions; expected post-smoothing FWHM now conditions on baseline smoothness, program/mask mode, and the voxel-to-kernel ratio.
0.8-2
- Add a
voxel_pscintensity-normalization mode that uses the existing robust reference-core and eligible-frame policy and applies denominator-guarded baseline-to-100 scaling after spatial processing. Reliable local baselines use ordinary PSC scaling, very low positive baselines use a lower denominator bound, and invalid baselines or those with too few eligible frames use a conservative run-level fallback. The guards do not clip observations or mask voxels; the user'sapply_maskdecision is preserved, and the multiplier map and guard counts are saved for provenance. Guard counts and percentages within the conservative automask are logged at info level, with complete-grid counts at debug level. - Add a user-oriented intensity-normalization vignette documenting the
targetconvention, robust reference-core policy, provenance outputs, QA, and limitations. - Replace
automask()'s background-sensitive positive-voxel quantile interpolation with an iterative AFNI-style clip estimator and a smoothly varying local threshold field. - Match AFNI's
automask()peeling more closely with a 17-of-18 NN2 survival rule, layer-aware restoration, and post-peel face-connected reclustering. - Replace postprocessing's late 4D-median intensity estimate with an automask-based robust reference core selected from the original positive-scale BOLD image; measure and apply the run factor after masking/smoothing but before AROMA, temporal filtering, confound regression, or timepoint removal, and save the core mask and JSON provenance.
- Accept
postprocess/intensity_normalize/targetas the simplified normalization setting while retainingglobal_medianas a backward-compatible alias. - Preserve each voxel's pre-AROMA temporal mean during both aggressive and non-aggressive AROMA denoising, retaining the positive baseline intensity used for cross-run scaling.
- Refactor postprocessing to use job arrays and sentinels for cleanup
- Add additional templates to prefetch needed by MRIQC
- Preserve user-specified
metadata/sqlite_dbvalues and exposesqlite_dbinedit_project(). - Clean postprocessing scratch workspaces and temporary automask files on errors as well as successful exits.
- Use exit-time cleanup for temporary FSL postprocessing files generated during temporal filtering, smoothing, confound regression, and brain-mask computation.
- Add regression tests for editable SQLite database configuration and postprocessing temp-file cleanup after failures.
- Prefetch resolution-1 T1w and brain-mask assets used by fMRIPrep anatomical reports, including when an output space explicitly requests another resolution.
0.8-1
- Improve CLI interface to support --help or BrainGnomes help
- Add CLI status command to get project status from command line
- Add dry_run option to run_project to see what would be run without executing it
- Refactor prefetch to accept cohort specifications and extend them to T2w fetch.
- Refactor prefetch to fall back to no desc field if desc:brain fails
- Expand fMRIPrep TemplateFlow defaults to include MNI152NLin2009cAsym boldref, res-2 brain mask, brain probseg, and carpet dseg assets observed during workflow construction
- Add conditional CIFTI TemplateFlow defaults so prefetch only stages MNI152NLin6Asym and fsLR sphere assets when fMRIPrep CLI options request
--cifti-output - Harden prefetch caching and validation checks so that later failures invalidate skip logic
- Make prefetch state query-specific so that an exact snapshot of templateflow files is retained
- Move prefetch state files out of
templateflow_homeand into hashed project log paths; legacy state files intemplateflow_homeare now migrated and removed to avoid poisoning TemplateFlow standard-space discovery. - Expand TemplateFlow default to desc=None for T1w to mirror some versions of fmriprep.
- Harden check on flywheel location to accommodate missing fw command.
- Update RSQLite connections to default to
synchronous=NULLto prevent spurious warnings - Included OASIS30 as a default template space for prefetch because it is used by fmriprep
- bugfix: preserve
cohort-<n>in BIDS parsing/reconstruction so postprocessing can resolve cohort-qualified fMRIPrep outputs such asspace-MNIPediatricAsym_cohort-2
0.8
- Add optional low-pass filtering of motion parameters before FD recomputation; rename notch config fields to
bandstop_min_bpm/bandstop_max_bpm(deprecated:band_stop_min/band_stop_max). - All HPC jobs are now tracked in detail by an SQLite database
- Job failures and other errors can now be investigated using
diagnose_pipeline - Added a new vignette, "Diagnosing Pipeline Runs", that walks through
get_project_status(),get_subject_status(),
and interactive use ofdiagnose_pipeline() - Improved error logging in HPC scripts so that success and failure are indicated more clearly
- Stale .fail files are removed when a newer .complete file exists, clarifying status of processing steps
- Jobs now write a manifest of files and times to the job tracking database for more thorough completeness tests
- Added optional low-pass filtering of motion parameters, matching Gratton
- Gracefully adjust motion filtering parameters if they fall above Nyquist at this TR
- Modify extract ROIs config to avoid input_regex and always generate it internally from postproc stream
- Add optional header row for postprocessed confounds TSVs, configurable via postprocess YAML and validated during setup
- Added extensive checks on write/permission issues with directories and files
- bugfix: Get CSF probseg image for MRIQC during prefetch
run_project()now skips TemplateFlow prefetch only when a prior successful prefetch covers requested spaces and the TemplateFlow manifest in job tracking still verifies; missing/deleted template files trigger re-prefetch.- bugfix: preserve user-specified
metadata/log_directory(including external paths) instead of always resetting to<project_directory>/logs. - During postprocess setup,
confound_calculatenow offers guided prompts to addframewise_displacementwhen omitted, including whether to use motion-filtered FD and whether FD should be processed vs kept asnoprocfor QC/exclusion workflows. - Increase consistency of instructions and formatting in
setup_project() - bugfix: avoid spurious "Already disconnected" warnings on exit from
diagnose_pipeline() - bugfix:
diagnose_pipeline()now respects configuredmetadata/log_directoryinstead of assuming<project_directory>/logs - bugfix:
diagnose_pipeline()now matches subjects by exactsub-<id>tokens to avoid accidental partial matches - bugfix:
run_bg_and_wait()now suppresses and restoresERRtrap handling aroundwait, so non-zero container exits can be reconciled against success tokens before jobs are marked failed. - bugfix: shell trap handlers now attempt a best-effort SQLite status update to
FAILEDbefore exit, reducing_fail/DB mismatch after abrupt failures. - bugfix:
update_tracked_job_status()now warns when no tracking rows are updated for a job_id (instead of failing silently). - bugfix:
check_status_reconciliation()now checks.failmarkers against DB status and reports mismatch details.
0.7-5
0.7-4
0.7-3
0.7-2
Support multisession directories for AROMA checks