Signal type of EDS dataset

[ericpre]

We should improve the situation with setting the signal_type of EDS dataset, make it consistent across EDS data file reader and document its behaviour in rosettasciio and in hyperspy.

The approach used in the bruker reader seems to be the most sensible:

rosettasciio/rsciio/bruker/api.py

Lines 1496 to 1511 in 57fccc4

	def guess_mode(hv):
	"""there is no way to determine what kind of instrument
	was used from metadata: TEM or SEM.
	However simple guess can be made using the acceleration
	voltage, assuming that SEM is <= 30kV or TEM is >30kV"""
	if hv > 30.0:
	mode = "TEM"
	else:
	mode = "SEM"
	_logger.info(
	"Guessing that the acquisition instrument is %s " % mode
	+ "because the beam energy is %i keV. If this is wrong, " % hv
	+ "please provide the right instrument using the 'instrument' "
	+ "keyword."
	)
	return mode

In any case, data acquired on a SEM doesn't guaranty that the signal_type should be EDS_SEM, as the specimen can be thin enough to make EDS_TEM more suitable!

[sem-geologist]

SEM_EDS and TEM_EDS should go away from rosettasciio, why? 1) because this is feature creep to io library, 2) brakes compatibility with i.e. emsa formats (format saved with hyperspy is not compatible with DTSA-II or NeXLSpectrum.jl), 3) complicates (rather blocks) the life for other methods which wanted for long time already to extend on this library: i.e. μXRF, but I can see other methods like PIXE and Xray- inducing possibilities are coming - it is insane to have separate signal type, while it is Xrays gathered by same type of detector. SEM, TEM, XRF, PIXE - those are interpretations of data gathered by spectrometer which does not care. Make this non-mandatory mode as attribute - not the root of metadata structure!

[jlaehne]

To my understanding, the metadata->Signal->signal_type is actually an attribute and not the root of the metadata structure.

Users of HyperSpy, of course still will want that the signal_type is set correctly when reading the files. For the moment, I don't see a good way to keep that out of RosettaSciIO completely, otherwise the HyperSpy-IO functions will need again a format-specific code that decides on what signal-type to choose (and for the moment all the format-specific stuff has been moved to this repository).

brakes compatibility with i.e. emsa formats

If a file_writer does not adhere to format definitions, we should of course correct that. EMSA is a clearly defined format specification, where we should not add anything that breaks compatibility. If so, you should open a separate issue on that.

[ericpre]

I suspect that there isn't a perfect solution to this problem (not setting a signal type at all is not convenient for a vast majority of users) and we need to be pragmatic with what is the most appropriate approach. I opened the issue so that we can discuss about it! :)

2. brakes compatibility with i.e. emsa formats (format saved with hyperspy is not compatible with DTSA-II or NeXLSpectrum.jl)

@sem-geologist, can you please elaborate on what is the issue?

[sem-geologist]

My point is that signal_type should be EDS. Is it XRF or TEM or SEM or PIXE - that is interpretative stuff and has nothing to do with IO. I understand that hyperspy uses some mixins to hide or show some appropriate methods depending if EDS_TEM or EDS_SEM. That actually not good idea at all. That does not depend from instrument, acceleration voltage or detector position but from the sample. If You insert very thin (i.e.) 20nm thin foil in SEM and will analyse that with EDS - then TEM-like quantification should be used. Probably 30keV beam would be used for best resolution, however You could still get EDS signal with 20 or 15kV even in "STEM" mode. Then there are other means to excite atoms for X-ray emission with followed detection by EDS of that. XRF and PIXE (proton bombardment) comes on my mind as such alternative means. µXRF is real instruments and there were already few attempts to get those EDS working and this signal_type was standing in the way.

The point is that if it is EDS type of file - the attribute signal_type should be EDS, the hyperspy then could use some guessing functions to adjust that to EDS_TEM, EDS_SEM or other implemented type of signal for internal use, but should never save it externally to other formats than its own. To begin with actually having two separate type of signals instead of one unified in hyperspy is to blame for this whole situation.

problem with EMSA is that signal_type which hyperspy puts under #SIGNALTYPE should be EDS, not EDS_SEM or EDS_TEM which are not understood by i.e. DTSA-II or any EMSA specification complying software.
Look there:
https://the-mas.org/wp-content/uploads/2018/11/emmff_ascii.txt
under allowed attributes under #SIGNALTYPE there is EDS, WDS, ELS, AES... there is no EDS_SEM or EDS_TEM; why it would be? it logically should not be there as EDS_TEM or EDS_SEM is fused terms of signal type with signal interpretation. IO should not do anything interpretative.

[ericpre]

My point is that signal_type should be EDS. Is it XRF or TEM or SEM or PIXE - that is interpretative stuff and has nothing to do with IO. I understand that hyperspy uses some mixins to hide or show some appropriate methods depending if EDS_TEM or EDS_SEM. That actually not good idea at all. That does not depend from instrument, acceleration voltage or detector position but from the sample. If You insert very thin (i.e.) 20nm thin foil in SEM and will analyse that with EDS - then TEM-like quantification should be used. Probably 30keV beam would be used for best resolution, however You could still get EDS signal with 20 or 15kV even in "STEM" mode. Then there are other means to excite atoms for X-ray emission with followed detection by EDS of that. XRF and PIXE (proton bombardment) comes on my mind as such alternative means. µXRF is real instruments and there were already few attempts to get those EDS working and this signal_type was standing in the way.

The point is that if it is EDS type of file - the attribute signal_type should be EDS, the hyperspy then could use some guessing functions to adjust that to EDS_TEM, EDS_SEM or other implemented type of signal for internal use, but should never save it externally to other formats than its own. To begin with actually having two separate type of signals instead of one unified in hyperspy is to blame for this whole situation.

Yes, I fully agree with this summary, My primary intention when opening this issue was to make things more consistent and not to change significantly the current situation. However, it may be worth considering improvement too! What's about doing something along the lines of:

• set signal_type to "EDS" in rosettasciio
• in EDSSpectrum.__init__, check the value of the energy beam change the signal_type accordingly?

problem with EMSA is that signal_type which hyperspy puts under #SIGNALTYPE should be EDS, not EDS_SEM or EDS_TEM which are not understood by i.e. DTSA-II or any EMSA specification complying software. Look there: https://the-mas.org/wp-content/uploads/2018/11/emmff_ascii.txt under allowed attributes under #SIGNALTYPE there is EDS, WDS, ELS, AES... there is no EDS_SEM or EDS_TEM; why it would be? it logically should not be there as EDS_TEM or EDS_SEM is fused terms of signal type with signal interpretation. IO should not do anything interpretative.

That should be easy to fix!

[jlaehne]

Yes, I think that would be a good approach for RosettaSciIO.

Anything concerning the naming within HyperSpy would need to be discussed in the context of splitting off the EDS/EELS signal types to an extension.

[jlaehne]

* set `signal_type` to `"EDS"` in rosettasciio

problem with EMSA is that signal_type which hyperspy puts under #SIGNALTYPE should be EDS, not EDS_SEM or EDS_TEM which are not understood by i.e. DTSA-II or any EMSA specification complying software. Look there: https://the-mas.org/wp-content/uploads/2018/11/emmff_ascii.txt under allowed attributes under #SIGNALTYPE there is EDS, WDS, ELS, AES... there is no EDS_SEM or EDS_TEM; why it would be? it logically should not be there as EDS_TEM or EDS_SEM is fused terms of signal type with signal interpretation. IO should not do anything interpretative.

That should be easy to fix!

The rsciio part is done in #39