CTF metadata parameters

[GenevieveBuckley]

Closes #86

Converting the CTF parameter metadata from CryoSPARC back to Relion star files.
This work is in progress, I'm opening the PR to make collaboration & code review easier. I've tried to implement the suggestions from #86 (comment).

[GenevieveBuckley]

Does this also need a conversion factor, or is that already taken care of in BEAMTILTX and BEAMTILTY?

[asarnow]

I think the _3 coefficients are 2 * π * λ**2 times the tilt vector (tilt_A) like in Eqn 5 from the older "new tools for automated high-resolution..." paper.

In Zernike terminology these are the horizontal and vertical coma components. Then there are the "vertical and oblique trefoil" Z(-3,3) and Z(3,3), I think they have the same prefactor as the tilt and are called 'trefoil_A' in cryoSPARC.

[GenevieveBuckley]

I think the _3 coefficients are 2 * π * λ**2 times the tilt vector (tilt_A) like in Eqn 5 from the older "new tools for automated high-resolution..." paper.

Did you mean to write 2 * π * Cs * λ**2 here instead? Equation 5 includes the Cs (spherical abberation) term, and we also used 2 * π * Cs * λ**2 as the prefactor for the trefoil terms in the code here too.

I'm going to proceed on the assumption that you did mean to say 2 * π * Cs * λ**2, so if that's not the case please clarify and I can edit/fix things.

[GenevieveBuckley]

... although I see here that the cs['ctf/cs_mm'] term was divided out from ctf/tilt_A, so perhaps i should not be re-introducing it.

[GenevieveBuckley]

1. Ok this is ready for a second review, paying special attention for mistakes in things like the conversion factors.

2. There are three parameters we haven't made use of yet: ctf/scale, ctf/scale_const, and ctf/bfactor:

   • ctf/bfactor is included in the .star file metadata in a different part of the code, so I don't think I need to worry about this.
   • ctf/scale and ctf/scale_const are not used anywhere, as far as I can tell. I'm going to assume they're in the CryoSPARC metadata because they're important, should we be doing something with these?

3. After we are satisfied with it, this will need a test case.
   I've been given one file I could get the cs data structure out of, but I'm not sure how to confirm correctness after converting to .star, besides asking my colleagues to check for any obvious errors in magnitude of the output results. We could maybe run the CTF estimation in Relion separately, then check for differences between the two?

[GenevieveBuckley]

In case you're curious, I just ran it on a test file I had lying around, and got this output for the data_optics metadata in the output .star file:

data_optics

loop_
_rlnVoltage #1 
_rlnImagePixelSize #2 
_rlnSphericalAberration #3 
_rlnAmplitudeContrast #4 
_rlnBeamTiltX #5 
_rlnBeamTiltY #6 
_rlnOpticsGroup #7 
_rlnImageSize #8 
_rlnMagMat00 #9 
_rlnMagMat01 #10 
_rlnMagMat10 #11 
_rlnMagMat11 #12 
_rlnOddZernike #13 
_rlnEvenZernike #14 
_rlnImageDimensionality #15 
300.000000 0.650000 2.686780 0.100000 0.076775 0.032975 3 360 -0.004738 -0.005020 -0.005219 0.005668 "[6.3394876, 6.3043222, -5375.929, 0.64963967, 0.27901965, 2234.6147]" "[-1.5372919, 279.67224, 16605.41, 30.46631, 15336.822, -49.591393, -1.4976486, 6214.921, -5361.158]" 2
300.000000 0.650000 2.688615 0.100000 0.080809 0.029494 4 360 -0.004517 -0.004959 -0.004960 0.005799 "[6.345222, 6.3053083, -4898.355, 0.68424, 0.24973576, 2455.8643]" "[-1.5350487, 210.5198, 15697.895, -63.045483, 14601.965, -2822.9922, -1.4986714, 6445.5967, -3120.5176]" 2

data_particles
...

I don't know if the quotation marks around our Zernike coefficient strings will be a problem for Relion to parse, that's another thing to check.

[asarnow]

@GenevieveBuckley I ended up merging this into a new branch, since it was easier for me to review and make changes that way. I did some refactoring to match the code I had already put in the master branch for the star.Relion and star.UCSF constants, and for writing the Zernike coefs field out to a star file in the right format and order for Relion.

I also pulled out all of the π/wavelength/etc prefactors into some pre-computed variables. Apologies if it's a little disorienting! I think this way is less error prone, but it also might be less readable / easy to connect to the equations. We could change it back if you think that is best.

Finally, I corrected the conversions according to my current understanding. I think we should test it with some data where CTF refinement in both Relion and cryoSPARC yields a similar improvement to the resolution, and then compare the coefficients we obtain. I have a feeling they might only be correct to some linear operation.

When we're sure the branch is ready we can squash it into one commit and merge with the master branch. I'll probably ask you to do that in another PR at that point so that the credit is reflected clearly on GitHub.

[GenevieveBuckley]

Ah, I was a bit confused about why this was marked merged, makes much more sense if you've put it into a separate branch to test things with.

Thank you for refactoring some things. I'll head over to your convert_zernike branch and have a look at the changes.

Sure, squash merging a second PR sounds like the cleanest way to do it.