Further features

[jo-moeller]

Hi, as previously discusssed with @dallanto and James, we would need a few additions for online analysis with extra-foam at MID.

• Find the beamcenter (called Cx and Cy coordinates in the azimuthal integration tab of the image tool) from a refrence sample. Similar as in the example notebook provided, minimizing the difference in I(q) calculated from different azimuthal cones on the detector.
• Circular ROIs. Regions-of-Interest based on similar q-values.
• Display a running average of the data in the overview tab, so that masking and finding of the beamcenter can be done on less noisy data.
• Option of thresholding the incoming data, in order to convert from ADU to photon integer. Called photon-binning in the karabo speckle analysis device.

Some good to haves:

1. log-colorscale on the detector images
2. Instead of a running average in the overview tab, display a running std/var/min/max as well.
3. Saving AGIPD masks generated in extra-foam in a (16,512,128) format, without applying the geometry to it. (Already discussed in Loading/saving masks in pixel coordinates #116, but not solved yet)

[JamesWrigley]

Finding the beam center, displaying a running average, and thresholding the incoming data are implemented. Support for circular ROI's is nearly done, all that's left to do is actually compute and apply the mask (which should be easy, but it's late 😛 ). I also want to clean up the GUI controls for it, but I think that would take a bit of refactoring, which probably belongs in another PR.

WRT point 2., do you mean displaying the running std/var/min/max over an ROI, i.e. scalar values? Or computing them for each pixel and displaying them as images?

Demo:

The code is in the mid_special_suite branch if you want to have a look. It may be completely broken on systems other than my own though.

[ebadkamil]

Thanks @JamesWrigley for the nice work. I have few questions.

• Will the possibility of Circular ROI, be available only in this special suite or do you plan to provide it in core ImageTool too? It will be nice to have both Circular and RectROI.
• You are using AzimuthalIntegrator from pyFAI and setFit2D to evaluate beam center. Did you test the performance with the functionality of beam center available in EXtra-foam core.
• what is the rate of processing for SpeckleContrastProcessor since it also assembles the image using extra-geom AGIPD assembler.

[dallanto]

Impressive work, James!

[jo-moeller]

Cool, looks very nice. I'll try to give it a closer look before the weekend.

[JamesWrigley]

• Will the possibility of Circular ROI, be available only in this special suite or do you plan to provide it in core ImageTool too? It will be nice to have both Circular and RectROI.

Just in this special suite for now, but I could certainly extend it to the ImageTool as the next step. I think the nicest way would be to add an extra widget to the ROI controls to allow the user to select the shape of the ROI they want. That would need some modification to the _SingleRoiCtrlWidget class, perhaps to make it 'own' the ROI object instead of letting it be passed in.

• You are using AzimuthalIntegrator from pyFAI and setFit2D to evaluate beam center. Did you test the performance with the functionality of beam center available in EXtra-foam core.

Ah, no. I actually wasn't aware that EXtra-foam was capable of finding the beam center, where can I find the code for that? I tried grepping the codebase for 'beam' and 'center' but couldn't find anything.

• what is the rate of processing for SpeckleContrastProcessor since it also assembles the image using extra-geom AGIPD assembler.

Absolutely terrible 😛 Like, 0.5 - 1 Hz on my decently powerful i7. Is there another way to assemble the image that would be faster?

And finding the beam center is also really slow, on my machine it takes ~25s. On the plus side it should be easy to parallelize parts of it and run it in a separate process so it doesn't block the GUI. Another option would be to implement it in the C++ core or use Cython (not sure how complicated either of those would be).

[jo-moeller]

And finding the beam center is also really slow, on my machine it takes ~25s. On the plus side it should be easy to parallelize parts of it and run it in a separate process so it doesn't block the GUI. Another option would be to implement it in the C++ core or use Cython (not sure how complicated either of those would be).

Finding the beamcenter is something you would do once at the beginning of each shift. No problem if that is not super fast.

[ebadkamil]

Absolutely terrible stuck_out_tongue Like, 0.5 - 1 Hz on my decently powerful i7. Is there another way to assemble the image that would be faster?

Something like this where the assembler is implemented in C++. Take a look at foamalgo

EXtra-foam/extra_foam/geometries/__init__.py

Line 336 in 976bce5

def load_geometry(detector, *,

Ah, no. I actually wasn't aware that EXtra-foam was capable of finding the beam center, where can I find the code for that? I tried grepping the codebase for 'beam' and 'center' but couldn't find anything.

I think foamalgo implements ConcentricRingFinder

[zhujun98]

Detecting beam center is already implemented https://extra-foam.readthedocs.io/en/latest/image_tool.html#concentric-rings, which takes a few hundreds of milliseconds, depending on the image size. I don't see the point of reinventing the wheel here.

[zhujun98]

Adding the circular ROI is a good progress 👍 . But I would suggest to think about corner cases (e.g. on the edge of an image) and performance.

[JamesWrigley]

Something like this where the assembler is implemented in C++. Take a look at foamalgo

Ah thanks, will look into that.

WRT finding the beam center, I believe the reason for this different method is to handle images that don't have well-defined concentric rings. I played around a little with the concentric ring feature on the dataset I'm testing with, and the calculated beam center isn't that stable if you play around with the settings for prominence/min. count etc (though this could also be because of my inexperience with the tool).

[zhujun98]

The reason for the hyperparameters prominence and min count is because the behavior of detectors and experiments are very different. The user should roughly know their data. Of course, this is the first implementation and I had no chance to collect user feedback. But I tested with all the detectors and different datasets that I have. They all worked.

[zhujun98]

I believe the reason for this different method is to handle images that don't have well-defined concentric rings

If there are no rings, one could simply find the center of mass of the spot, which is how people do in experiments.

[JamesWrigley]

Ah, I see. @jo-moeller, has that already been tried? I could test it if not.

[jo-moeller]

Hey,
concerning finding the beam center there are basically two different types of data which have radial symmetry. Crystalline material can have well defined Bragg-rings, that's where the "concentric ring" feature would work well. And obviously it does. I can give it a try as well.

The other type of samples are unordered or non-crystalline. They might have some oscillating features, like the example data set we used. Or only very broad rings. They might also just have a I(q) ~ q^-4 power law all over the range of the detector. Here the beam center finding that I suggested normally works quite well. Center of mass could also work, but I think the one that James now implemented works better if there are some features like oscillations.

Some example plots are in our beamline paper, figure 10:
https://journals.iucr.org/s/issues/2021/02/00/ay5570/ay5570.pdf

[jo-moeller]

Maybe it would be worth converging both approaches into one feature?

[JamesWrigley]

Yep, can look into that :)