[FIX] Fix PSC normalization when a high-pass filter is applied

[ymzayek]

• Closes [BUG] NiftiSpheresMasker - Combination of percent signal change normalization and high-pass filtering only outputs signals in incorrect range. #4087

[github-actions]

👋 @ymzayek Thanks for creating a PR!

Until this PR is ready for review, you can include the [WIP] tag in its title, or leave it as a github draft.

Please make sure it is compliant with our contributing guidelines. In particular, be sure it checks the boxes listed below.

• PR has an interpretable title.
• PR links to Github issue with mention Closes #XXXX (see our documentation on PR structure)
• Code is PEP8-compliant (see our documentation on coding style)
• Changelog or what's new entry in doc/changes/latest.rst (see our documentation on PR structure)

For new features:

• There is at least one unit test per new function / class (see our documentation on testing)
• The new feature is demoed in at least one relevant example.

For bug fixes:

• There is at least one test that would fail under the original bug conditions.

We will review it as quick as possible, feel free to ping us with questions if needed.

[ymzayek]

Is it valid to just add back the original mean to the signal for calculating PSC?

[ymzayek]

Is it valid to just add back the original mean to the signal for calculating PSC?

Or alternatively pass the original mean signal and subtract that from the filtered signal like was suggested in the issue? But that doesn't center around 0

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (3d80bac) 92.10% compared to head (fd532e5) 92.07%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4116      +/-   ##
==========================================
- Coverage   92.10%   92.07%   -0.03%     
==========================================
  Files         144      144              
  Lines       16368    16368              
  Branches     3427     3427              
==========================================
- Hits        15075    15071       -4     
- Misses        754      756       +2     
- Partials      539      541       +2     

Flag	Coverage Δ
macos-latest_3.10_test_plotting	91.86% <100.00%> (-0.03%)	⬇️
macos-latest_3.11_test_plotting	91.86% <100.00%> (-0.03%)	⬇️
macos-latest_3.12_test_plotting	91.86% <100.00%> (-0.03%)	⬇️
macos-latest_3.8_test_plotting	?
macos-latest_3.9_test_plotting	?
ubuntu-latest_3.10_test_plotting	91.86% <100.00%> (-0.03%)	⬇️
ubuntu-latest_3.11_test_plotting	91.86% <100.00%> (-0.03%)	⬇️
ubuntu-latest_3.12_test_plotting	91.86% <100.00%> (-0.03%)	⬇️
ubuntu-latest_3.12_test_pre	91.86% <100.00%> (-0.03%)	⬇️
ubuntu-latest_3.8_test_min	68.95% <100.00%> (-0.03%)	⬇️
ubuntu-latest_3.8_test_plot_min	91.57% <100.00%> (-0.03%)	⬇️
ubuntu-latest_3.8_test_plotting	91.82% <100.00%> (-0.03%)	⬇️
ubuntu-latest_3.9_test_plotting	91.83% <100.00%> (-0.03%)	⬇️
windows-latest_3.10_test_plotting	91.84% <100.00%> (-0.03%)	⬇️
windows-latest_3.11_test_plotting	91.84% <100.00%> (-0.03%)	⬇️
windows-latest_3.12_test_plotting	91.84% <100.00%> (-0.03%)	⬇️
windows-latest_3.8_test_plotting	91.80% <100.00%> (-0.03%)	⬇️
windows-latest_3.9_test_plotting	91.81% <100.00%> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[htwangtw]

I will need to do some experiment around this to see what's the most sensible way to move forward.
Indeed the original implementation of psc doesn't have enough safe guards. Even without filtering, user can generate things that are unusable if their inputs have mean signals close to 0.

[htwangtw]

The default of detrend is True so the current changes are

1. not really correctly tested in test_clean_psc
2. doesn't address the bug of the original issue

I tried to add a test with butterworth filter and we ended up with very wonky means (number around -100 or 100).

I have also discovered the psc output will have really wild numbers if we have means less than 1. This standardization method might need a new review @bthirion

[bthirion]

The change looks good, bit we should probably test all relevant configurations. Best,

[htwangtw]

It took me a long time to wrap my head around before and after holiday but I have found a satisfying solution. The correct tests broke lots of things but luckily the solution turned out to be quite simple after I figured thins out.

For tests:

• make sure the signal is not detrended in test so psc is tested in isolation
• add butterworth filter as a special case to the test

For code:

• streamline the logic of standardisation so we make sure mean signals are added back only when using psc and when butterworth filter or detrend was done.
• remove the mean_signals parameter from standardize_signal but calculate it inside the function. This is due to signal butterworth filter will have a non-zero mean that is really close to zero.
• make sure filter_type returns the correct filter that's applied. Before, we relied on the check in the butterworth function to apply things correctly. I have triple-checked and the filters were applied correctly despite filter_type was not correctly reflecting the strategy applied in case of butterworth filter... phew

@ymzayek Have a look and see if this is good to go

[bthirion]

Thx for digging into this I did not spot any issue.

[ymzayek]

@htwangtw thanks a lot! The changes look good and I checked the plots and they are as expected! I just added a whatsnew and will resolve the conflicts, then merge when everything is green