Issue with pattern centre definition in detector class for Oxford/HKL data

[IMBalENce]

Hi Håkon,

There seems to have some issues with PC (pattern centre) definition in detector class for Oxford/HKL data. I was testing a dataset acquired from our Oxford Nordlys detector with 640x480 CCD, pixel size ~59.2um using binning 4.

• The first issue I found is that the L value after PC conversion from Oxford to EMsoft is not consistent with the equations (3) in JPDeGraef19 doi:10.1007/s40192-019-00137-4.

For example, if the detector is defined as below (PC values were get from one of the raw pattern tiffs' metadata):

detector = kp.detectors.EBSDDetector(
    shape=s.axes_manager.signal_shape[::-1],
    pc=[ 0.48935617839432832, 0.61856923343313686, 0.49397253582464035],
    convention="oxford",
    px_size=236.8,  # microns
    binning=1,
    tilt=0,
    sample_tilt=70.004152426213111
)
detector.pc_emsoft()

Outputs:
array([[1.7030115e+00, 1.4228308e+01, 1.4036724e+04]], dtype=float32)

The third value in the output 14036.724 is different from 18716.010 calculated using [#of pixel in X] * [pixel size] * [PCz]. In fact, I find the output value matches [#of pixel in Y] * [pixel size] * [PCz] perfectly. Is it possible that the #of pixel in X variable has been misused?

• Secondly, I searched a few different versions of Oxford and EDAX manuals and can confirm that the two vendors are using exactly the same way in defining their pattern centres. Both are positioned with respect to the (0,0) being at the bottom left of the screen; and both are measure as a fraction of the EBSP width. Therefore, within the pattern, x ranges from 0 to 1, y from 0 to H/W (0.75 in our case). And in EMsoft, the PC for Oxford and EDAX should be calculated using exactly the same equations (2) in JPDeGraef19 with Xpc flipped. This calculation seems has already been applied in the latest EMsoft release, as the DI results using corrected PC calculations are extremely good. But I have not read anything about the corrections yet.

• The third one is about the display of PC in Kikuchipy. I find if I define the detector using the code above and display one of the pattern
  detector.plot(pattern=s.inav[35, 35].data)
  Outputs
  
  The position is different from the display in Oxford Aztec
  

I guess this is also because the PCy is measured as a fraction of pattern width rather than height.

After correction seems about the same

Please let me know your thoughts on these. Thank you.

BR,
ZX

[hakonanes]

Hi Zhou!

The images tell me there's a bug in kikuchipy, as you say. Upon creation of an EBSDDetector, the input PC is converted to Bruker's definition. We use the same convertion from TSL/Oxford to Bruker. When converting to the EMsoft convention, the Bruker->EMsoft route is the only one implemented; to get L we multiply with number of rows (Y) as per Jackson et al. (2019) as you say.

The bug must be in Oxford->Bruker. When I wrote EBSDDetector, I didn't consider the fact that y could range from anything other than 0 to 1. I believe we could change this

kikuchipy/kikuchipy/detectors/ebsd_detector.py

Lines 605 to 608 in 4a3170e

	def _pc_tsl2bruker(self) -> np.ndarray:
	new_pc = deepcopy(self.pc)
	new_pc[..., 1] = 1 - self.pcy
	return new_pc

to this

    def _pc_tsl2bruker(self) -> np.ndarray:
        new_pc = deepcopy(self.pc)
        new_pc[..., 1] = (1 - self.pcy) * self.aspect_ratio  # n rows / n columns
        return new_pc

This should at least fix your PCy value so that its position in the plot is correct.

---

Edit: See next comment, as I misunderstood the signal shape here!

When it comes to conversion from Oxford to EMsoft coordinates:

1. What's the unbinned pixels size of your Oxford Nordlys detector, (1920, 2560)?
2. I cannot reproduce your detector.pc_emsoft() numbers with your input, which version of kikuchipy do you use? I got

import kikuchipy as kp
detector = kp.detectors.EBSDDetector(
    shape=(480, 640),
    pc=(0.48935617839432832, 0.61856923343313686, 0.49397253582464035),
    convention="oxford",
    px_size=236.8,  # microns
    binning=1,
    tilt=0,
    sample_tilt=70.004152426213111
)
print(detector)
print(detector.pc_emsoft())

EBSDDetector (480, 640), px_size 236.8 um, binning 1, tilt 0, azimuthal 0, pc (0.489, 0.381, 0.494)
[[6.8120461e+00 5.6913231e+01 5.6146895e+04]]

Note that the documentation states that the unbinned pixel size should be given accompanied by the binning factor. I think your input should be

import kikuchipy as kp
detector = kp.detectors.EBSDDetector(
    shape=(480, 640),
    pc=(0.48935617839432832, 0.61856923343313686, 0.49397253582464035),
    convention="oxford",
    px_size=59.2,  # microns
    binning=4,
    tilt=0,
    sample_tilt=70.004152426213111
)
print(detector)
print(detector.pc_emsoft())

EBSDDetector (480, 640), px_size 59.2 um, binning 4, tilt 0, azimuthal 0, pc (0.489, 0.286, 0.494)
[[2.7248184e+01 4.1073969e+02 5.6146895e+04]]

56 mm seems like a long detector distance... Could it be that you're binning factor is only 2, and that the unbinned detector shape is (1280, 960)? If so, a detector distance of 28 mm seems more plausible, but still long...

[hakonanes]

Scratch the previous comment, as I misunderstood what the signal shape was.

The following change to EBSDDetector._pc_tsl2bruker() should fix your issues:

    def _pc_tsl2bruker(self) -> np.ndarray:
        new_pc = deepcopy(self.pc)
        new_pc[..., 1] = (1 - self.pcy) * self.aspect_ratio  # n rows / n columns
        new_pc[..., 2] /= self.aspect_ratio
        return new_pc

right? Because Bruker defines detector distance in fractions of detector height, while TSL and Oxford defines it in terms of detector width.

[hakonanes]

With the above change, I get:

>>> import kikuchipy as kp
>>> detector = kp.detectors.EBSDDetector(
...     shape=(120, 160),
...     pc=(0.48935617839432832, 0.61856923343313686, 0.49397253582464035),
...     convention="oxford",
...     px_size=59.2,  # microns
...     binning=4,
...     tilt=0,
...     sample_tilt=70.004152426213111
... )
>>> print(detector)
EBSDDetector (120, 160), px_size 59.2 um, binning 4, tilt 0, azimuthal 0, pc (0.489, 0.286, 0.659)
>>> print(detector.pc_emsoft())
[[6.8120461e+00 1.0268492e+02 1.8715631e+04]]

Would it be possible for you to make the fix? This is a bug, so should be fixed in a v0.5.3 patch release. Thus, you need to base your bug fix branch on main, not develop, as explained in the contributing guide. Just set up a PR, and we can take it from there!

[IMBalENce]

Hi Håkon,

Thank you for the advise. and sorry for my initial confusing introduction.

The native resolution for the detector is 640x480 with ~59.2um pixel size. The map was acquired in binning 4 so I was taking the shortcut to define the detector as native 160x120 with 236.8um pixel size. And this can be reproduced in Kikuchipy v0.52.

detector = kp.detectors.EBSDDetector(
    shape=(120,160),
    pc=[ 0.48935617839432832, 0.61856923343313686, 0.49397253582464035],
    convention="oxford",
    px_size=236.8,  # microns
    binning=1,
    tilt=0,
    sample_tilt=70.004152426213111
)
print(detector)
detector.pc_emsoft()

EBSDDetector (120, 160), px_size 236.8 um, binning 1, tilt 0, azimuthal 0, pc (0.489, 0.381, 0.494) array([[1.7030115e+00, 1.4228308e+01, 1.4036724e+04]], dtype=float32)

If I directly calculate PC using the equations (3) (with -PCx) in Jackson et al. (2019) gives
[1.704846, 14.22831, 18715.63]

which is different in L from the pc_emsoft calculation in Kikuchipy.

However, if I use the directly calculated PC in EMsoft for dictionary indexing, the output map looks terrible even though ISR can be 91%+.

Instead, if I use the TSL equations to calculate PC ( gives [1.704846, 38.97107, 18715.63]) for EMsoft DI, the output map is a lot more reasonable with much better indexing rate.

Just did one more test to use the tsl route in Kikuchipy. And things look a bit more peculiar...

detector = kp.detectors.EBSDDetector(
    shape=(120,160),
    pc=[ 0.48935617839432832, 0.61856923343313686, 0.49397253582464035],
    convention="tsl",
    px_size=236.8,  # microns
    binning=1,
    tilt=0,
    sample_tilt=70.004152426213111
)
print(detector)
detector.pc_emsoft()

EBSDDetector (120, 160), px_size 236.8 um, binning 1, tilt 0, azimuthal 0, pc (0.489, 0.381, 0.494) array([[1.7030115e+00, 1.4228308e+01, 1.4036724e+04]], dtype=float32)

Did I misunderstand anything here?

Sure, once I have confirmed I'll send a PR if it is necessary.

[hakonanes]

The indexing success rate is calculated from the minimum disorientation angle between the best match and the remaining n best matches in EMsoft. If the best match is wrong, it is not unlikely that the next best match is similar to the best match, giving a low disorientation angle and high indexing success rate. I never judge the success of an indexing run based on that parameter. Instead, I visually compare some experimental and best matching simulated patterns.

As mentioned above and described in the user guide, the conversion from TSL/Oxford to Bruker is done in the same manner in kikuchipy... You've discovered some inconsistencies.

It is worth noting that to do DI and orientation refinement in kikuchipy, the pixel size in unnecessary unless the EMsoft PC is given upon detector initialization. After conversion to Bruker PC internally, this PC is used throughout.

I'll go over the equations again next week when I have time, and also compare patterns simulated with EMsoft and kikuchipy from an Oxford PC to see that everything is as expected.

[hakonanes]

It would be great if we could then fix this bug together, and release a v0.5.3!

[IMBalENce]

Sure, I'll also try to edit the routes you have pointed out and see if I can get consistent results.

[hakonanes]

@mikesmic, you wrote the Oxford .ebsp reader in EMsoft, could you tell us which conversion formula from Oxford's PC to EMsoft's PC you use for dictionary indexing in EMsoft (the relevant equations in the DI tutorial paper)?

Referencing @IMBalENce's relevant comment from above, where he's talking about using the Oxford -> EMsoft PC equation:

However, if I use the directly calculated PC in EMsoft for dictionary indexing, the output map looks terrible even though ISR can be 91%+. Instead, if I use the TSL equations to calculate PC ( gives [1.704846, 38.97107, 18715.63]) for EMsoft DI, the output map is a lot more reasonable with much better indexing rate.

I don't use Oxford myself, only TSL, and that equation is correct.

[hakonanes]

Have you had time to look more into this @IMBalENce?

[hakonanes]

EDIT: There was still an error in my use of binning, this is now fixed.

Based on your very useful comments on Oxford/TSL PC conventions, and the Bruker convention given in the excellent paper by Britton et al. (2016) in the (a-c) list just below the figure in the link, I think the following conversions should give us consistent PC (x, y, z) results between all three (Oxford/TSL, Bruker, EMsoft) conventions:

TSL/Oxford (T) -> Bruker (B):
xB = xT
yB = 1 - (Nx / Ny)yT
zB = (Nx / Ny)zT

EMsoft (E) -> Bruker:
xB = 0.5 - (xpc / Nx), version >= 5
xB = 0.5 + (xpc / Nx), version < 5
yB = 0.5 - (ypc / Ny)
zB = L / (pixel_size * Ny * binning)

Using these equations, I get the following results with your detector above:

>>> import kikuchipy as kp
# TSL/Oxford to Bruker to EMsoft, taking the "shortcut" and "ignoring" binning
>>> detector = kp.detectors.EBSDDetector(
...     shape=(120, 160),
...     pc=(0.48935617839432832, 0.61856923343313686, 0.49397253582464035),
...     convention="oxford",
...     px_size=236.8,
...     binning=1,
...     sample_tilt=70.004152426213111
... )
>>> print(detector)
EBSDDetector (120, 160), px_size 236.8 um, binning 1, tilt 0, azimuthal 0, pc (0.489, 0.175, 0.659)
>>> print(detector.pc_emsoft())
[[1.70301146e+00 3.89710773e+01 1.87156314e+04]]
# EMsoft to Bruker to EMsoft, taking the "shortcut" and "ignoring" binning
>>> detector = kp.detectors.EBSDDetector(
...     shape=(120, 160),
...     pc=(1.70, 38.97, 18715.63),
...     convention="emsoft",
...     px_size=236.8,
...     binning=1,
...     sample_tilt=70.004152426213111
... )
>>> print(detector)
EBSDDetector (120, 160), px_size 236.8 um, binning 1, tilt 0, azimuthal 0, pc (0.489, 0.175, 0.659)
>>> print(detector.pc_emsoft())
[[1.700000e+00 3.897000e+01 1.871563e+04]]
# EMsoft to bruker to EMsoft, providing exact pixel size and binning factor
>>> detector = kp.detectors.EBSDDetector(
...     shape=(120, 160),
...     pc=(1.70, 38.97, 18715.63),
...     convention="emsoft",
...     px_size=59.2,
...     binning=4,
...     sample_tilt=70.004152426213111
... )
>>> print(detector)
EBSDDetector (120, 160), px_size 236.8 um, binning 1, tilt 0, azimuthal 0, pc (0.489, 0.175, 0.659)
>>> print(detector.pc_emsoft())
[[1.700000e+00 3.897000e+01 1.871563e+04]]
# TSL/Oxford to Bruker to EMsoft, providing exact pixel size and binning factor
>>> detector = kp.detectors.EBSDDetector(
...     shape=(120, 160),
...     pc=(0.48935617839432832, 0.61856923343313686, 0.49397253582464035),
...     convention="oxford",
...     px_size=59.2,
...     binning=4,
...     sample_tilt=70.004152426213111
... )
>>> print(detector)
EBSDDetector (480, 640), px_size 59.2 um, binning 4, tilt 0, azimuthal 0, pc (0.489, 0.175, 0.659)
>>> print(detector.pc_emsoft())
[[1.70301146e+00 3.89710773e+01 1.87156314e+04]]

Would you be willing to fix the private conversion methods in ebsd_detector.py?

kikuchipy/kikuchipy/detectors/ebsd_detector.py

Lines 595 to 622 in 3ca855d

	def _pc_emsoft2bruker(self, version: int = 5) -> np.ndarray:
	new_pc = np.zeros_like(self.pc, dtype=np.float32)
	if version == 5:
	new_pc[..., 0] = 0.5 + (-self.pcx / (self.ncols * self.binning))
	else:
	new_pc[..., 0] = 0.5 + (self.pcx / (self.ncols * self.binning))
	new_pc[..., 1] = 0.5 - (self.pcy / (self.nrows * self.binning))
	new_pc[..., 2] = self.pcz / (self.nrows * self.px_size * self.binning)
	return new_pc
	def _pc_tsl2bruker(self) -> np.ndarray:
	new_pc = deepcopy(self.pc)
	new_pc[..., 1] = 1 - self.pcy
	return new_pc
	def _pc_bruker2emsoft(self, version: int = 5) -> np.ndarray:
	new_pc = np.zeros_like(self.pc, dtype=np.float32)
	new_pc[..., 0] = self.ncols * (self.pcx - 0.5)
	if version == 5:
	new_pc[..., 0] = -new_pc[..., 0]
	new_pc[..., 1] = self.nrows * (0.5 - self.pcy)
	new_pc[..., 2] = self.nrows * self.px_size * self.pcz
	return new_pc * self.binning
	def _pc_bruker2tsl(self) -> np.ndarray:
	new_pc = deepcopy(self.pc)
	new_pc[..., 1] = 1 - self.pcy
	return new_pc

If you set up the PR from your fork, I could contribute with making the tests pass, documentation etc.

[hakonanes]

@IMBalENce, alternatively, I can set up the PR, but ask you to contribute to it. The main thing is that you discovered this bug, and you should get credit for it! I think the most appropriate credit would be commits to the repo and your name in the package credits and the contributor grid in the README.

[hakonanes]

I set a tentative release date of v0.5.3 to the coming Friday. I'll set up a PR on Thursday unless you want to do it @IMBalENce.

[hakonanes]

The fix for this is now merged onto the main branch, and will be released over the weekend as part of a v0.5.3 patch release. Thanks for reporting this bug and contributing to fixing it, @IMBalENce!