Preprocess updates #1314

garrettwrong · 2025-08-04T14:13:01Z

I put this together hastily so I'll want to self-review and consider/discuss the changes more carefully before going farther...

codecov · 2025-08-04T14:34:03Z

Codecov Report

❌ Patch coverage is 93.75000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.60%. Comparing base (15effd7) to head (7444617).
⚠️ Report is 23 commits behind head on develop.

Files with missing lines	Patch %	Lines
src/aspire/source/image.py	82.35%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1314      +/-   ##
===========================================
- Coverage    90.61%   90.60%   -0.01%     
===========================================
  Files          133      133              
  Lines        14373    14394      +21     
===========================================
+ Hits         13024    13042      +18     
- Misses        1349     1352       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

garrettwrong · 2025-08-11T12:03:21Z

Currently testing the JSB gallery updates for this merged with the mctf work. If it seems okay I'll move this pr onto review.

garrettwrong · 2025-08-21T15:41:26Z

For 10028 80S dataset loaded as single precision and processed to 129 pixels in ASPIRE-Python these changes achieve max nrmse 7.6313734e-07 and avg nrmse 3.7122902e-07 across all images as compared to the MATLAB cryo_workflow_preprocess utilizing the expected sequence (as I understand it).

Phase Flip (360x360)  - phase_flip()
Crop to 359  - crop(359)
Downsample 129. - legacy_downsample(129)
Normalize Background - legacy_normalize_background()
Whiten - legacy_whiten()
Global Phase Flip - invert_contrast()

Note, that comparison included merging in the tentative CTF changes proposed in #1315 .

Besides the different defaults I also had to alter our code to upcast some (most) of the internal processing computations to double precision. This is essentially what the MATLAB code does in general, casting to singles at the end via the mrc save. Some computations test well for small/single image cases but differences show on larger images/stacks.

I expect there will be discussion about how to take the changes, but we can at least replicate the preprocessed dataset to my satisfaction now.

garrettwrong · 2025-08-22T15:08:49Z

Alright @j-c-c , this should be ready to review. Then I can get to the CTF stuff that is holding up your other PRs.

j-c-c · 2025-08-22T15:24:15Z

Alright @j-c-c , this should be ready to review. Then I can get to the CTF stuff that is holding up your other PRs.

Sounds good. I'll start reviewing!

j-c-c

Looks good! Just one small docstring fix and two typos while you're in there.

j-c-c · 2025-08-22T15:58:39Z

src/aspire/image/xform.py

+        """
+        Initialize Xform to crop Image to a specific size.
+
+        :param L: int - new size, should be <= the current size


Is the should be <= the current size in there by mistake? If we're padding wouldn't L > current size?

yea I forgot to delete this after I extended from just crop to crop_pad. removed. thanks

j-c-c · 2025-08-22T18:18:25Z

tests/test_preprocess_pipeline.py

+        np.testing.assert_allclose(pad_odd_to_even_many.images[:], ref)
+
+        # Padding odd to odd by many should pad the first and last cols equally.
+        # This test will also excecise `fill_value`


Not important, but "exercise".

fixed comment typo

j-c-c · 2025-08-22T18:19:21Z

tests/test_preprocess_pipeline.py

+        np.testing.assert_allclose(pad_even_to_even_many.images[:], ref)
+
+        # Padding even to odd by many should pad the first and last+1 cols equally.
+        # This test will also excecise `fill_value`


fixed comment typo

j-c-c

LGTM!

janden

Thank you for going through all this stuff. Just some minor formulation stuff.

src/aspire/image/image.py

src/aspire/source/image.py

…ethods

janden

LGTM

garrettwrong self-assigned this Aug 4, 2025

garrettwrong added documentation Improvements or additions to documentation enhancement New feature or request cleanup labels Aug 4, 2025

garrettwrong force-pushed the ppdbg branch 3 times, most recently from daf3608 to e0052ec Compare August 11, 2025 12:02

garrettwrong requested a review from j-c-c August 22, 2025 15:08

j-c-c reviewed Aug 22, 2025

View reviewed changes

garrettwrong requested a review from j-c-c August 25, 2025 18:31

garrettwrong marked this pull request as ready for review August 25, 2025 18:31

garrettwrong requested a review from janden as a code owner August 25, 2025 18:31

j-c-c previously approved these changes Aug 26, 2025

View reviewed changes

janden requested changes Sep 3, 2025

View reviewed changes

src/aspire/image/image.py Outdated Show resolved Hide resolved

src/aspire/image/image.py Show resolved Hide resolved

src/aspire/source/image.py Outdated Show resolved Hide resolved

garrettwrong dismissed j-c-c’s stale review via 1e2c573 September 3, 2025 13:32

garrettwrong force-pushed the ppdbg branch from 35dd68e to 1e2c573 Compare September 3, 2025 13:32

garrettwrong requested review from j-c-c and janden September 3, 2025 13:33

garrettwrong added 7 commits September 3, 2025 15:29

Add crop preprocess method

648c30c

Create legacy_normalize_background with workflow default radius

629ac2d

Create legacy_downsample and lightly refactor downsample/Downsample m…

65a5fe4

…ethods

Adjust/cleanup Xforms

f3117e2

update unit tests

19970ae

update gallery jsb examples

69be080

Add legacy crop to JSB examples

e577713

garrettwrong added 16 commits September 3, 2025 15:29

Use DiracBasis3D for jsb examples

fcf52ec

forgot basis dtypes

e5a0f56

use our existing crop_pad_2d

da27022

Normalize bg should be computed in double precision

7ea8234

Whitening should also be in double precision

663c141

Might as well extend crop to crop_pad

cdb9066

revert/postpose DiracBasis

77ef944

cleanup docstrings

d017fb3

cleanup docstrings

f27f7a9

add crop_pad unit tests

c4bf0b1

update crop comment

7caf164

words

3639474

add test for legacy_normalize_background

76d2c90

remove CropPad docstring oversight

a6a74ac

fixed comment typo

3b23296

Review string Updates

7444617

garrettwrong force-pushed the ppdbg branch from 1e2c573 to 7444617 Compare September 3, 2025 19:30

janden approved these changes Sep 4, 2025

View reviewed changes

garrettwrong merged commit 87c5976 into develop Sep 4, 2025
35 checks passed

garrettwrong deleted the ppdbg branch September 4, 2025 12:46

Preprocess updates #1314

Preprocess updates #1314

Uh oh!

Conversation

garrettwrong commented Aug 4, 2025

Uh oh!

codecov bot commented Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

garrettwrong commented Aug 11, 2025

Uh oh!

garrettwrong commented Aug 21, 2025

Uh oh!

garrettwrong commented Aug 22, 2025

Uh oh!

j-c-c commented Aug 22, 2025

Uh oh!

j-c-c left a comment

Choose a reason for hiding this comment

Uh oh!

j-c-c Aug 22, 2025

Choose a reason for hiding this comment

Uh oh!

garrettwrong Aug 25, 2025

Choose a reason for hiding this comment

Uh oh!

j-c-c Aug 22, 2025

Choose a reason for hiding this comment

Uh oh!

garrettwrong Aug 25, 2025

Choose a reason for hiding this comment

Uh oh!

j-c-c Aug 22, 2025

Choose a reason for hiding this comment

Uh oh!

garrettwrong Aug 25, 2025

Choose a reason for hiding this comment

Uh oh!

j-c-c left a comment

Choose a reason for hiding this comment

Uh oh!

janden left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

janden left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Aug 4, 2025 •

edited

Loading