`estimate-shifts` appear to have X-Y flipped

[garrettwrong]

Did a quick look over estimate shifts code in commonline base and ran some sanity tests.

If I take the current unit test for estimating shifts and replace the non-zero shift case (currently skipped, 😆 ) with a set of shifts that are all offsets of [1,2], the code returns values roughly near [2,1]. Interesting. They are not anywhere near close enough for allclose, but using np.round() indicated that we're in the right area already after flipping XY. Flipping XY can be accomplished by changing the these lines. (or by [:,::-1] the resulting shift array directly). Perhaps there is a deeper reason, but I didn't go any farther at this time.

Using these XY flipped estimated shifts with a small example pipeline (having small shifts), I think I can see some improvement.

I found the indexing arithmetic quite burdensome to read. I'm thinking it should be cleaned up and in the process, compared with the MATLAB code by taking an set of Rij rotation matrices from one of these tests (or pipeline demo).

Discussed with @j-c-c and main tasks below. These may make sense to come as separate PRs.

• Figure out this flipping issue
• Refactor the array indexing so it is clearer
• Compare with MATLAB (ideally reproduce, or minimally understand where and why we diverge)
• Create a functional unit test for the base class that actually runs.
• Discuss where/how to call estimate_shifts (automatically in oriented source?)
• Check this "memory factor" code (ideally, have way we can just disable it, we have hi-mem....)
• Update docs/demos to utilize the shifts

Relates to #995 , #676

[garrettwrong]

Updates. I've had some luck (got a recon) taking the MATLAB shifts and negating them along with flipping XY.

I've also been able to get a recon following merging similar patches in #1180 .

Josh has found that our Simulation default offsets generate poor estimated shifts in MATLAB, but offsets based on more realistic values appear to work in both packages (although it seems the python code with 1180 patches measures better, nice).

Im getting the feeling our default Simulation offsets are too aggressive for our purposes, and should probably be reduced to better reflect the experimental datasets we intend to reconstruct. Based on the datasets I got from Yoel, offsets uniform in xy <10% would be easy to implement and a better initial default configuration (for testing purposes).

• Revise default offsets and update unit tests using offset

[garrettwrong]

It looks like we can also address #966 along with these.

[janden]

Makes sense to tighten the default distribution of the shifts. What do we have currently?

[garrettwrong]

The default is Normal(0,1) * L/16 in both X and Y. With this we tend to be getting extremal shifts. Could clip it, change var, or just use a uniform...

[janden]

I'd rather reduce the variance than use uniform, if we can get that to work.

[garrettwrong]

I'd rather reduce the variance than use uniform, if we can get that to work.

Can certainly try it. I think the extremal ones are sort of trouble because they not only are across the boundary, but they wrap around.

[garrettwrong]

Shifts are added to the Orient3D gallery example in #1201 . Other than reviewing the need for the memory factor feature, and potentially reducing default shift var in future, the shift issues should be resolved. Closing.