how to save/restore UB matrix?

[prjemian]

Document the steps to save and to restore the UB matrix so that a session can be resumed.

[prjemian]

Possible save locations could include:

• previous scan from databroker
• file on disk

Consider that using the UB matrix after loading it is feasible (and will not crash anything).

• motor limits
• constraints

[prjemian]

Mining scans from the databroker for UB matrices would be aided by standardizing the way in which they are stored (and named) into the databroker. This is good baseline info (saved twice per scan) or good for a separate stream or descriptor. Investigate and decide. Tending to favor kind="config" so it gets into the descriptor with the name as provided here.

[ambarb]

I am leaning towards datastore to access via databroker.

I was thinking about that it would be saved as a special kind of scan. so the plan name would be something that isn't a scan, but to signify that the data is "bookmarked. SO the data specifically used to calculate the UB matrix (so to save the reflections and lattice params used to calculate UB). I want to do the same thing for "saved" sample positions when the number of samples are small, all on one holder, and is not worth the trouble of amonstra. So bookmark_hkl and bookmark_positions would be a new kind of plan that doesn't exist yet, but I think it it has a lot of functionality to lower transcription errors.

lookup can be restricted by things in the start document like proposal id or group or sample and a date range when looking for reflections.

If just wanting the UB used for previous scan, then maybe less should be required. I worry about making mistakes myself and typing a number wrong. so it would be nice to have more information on where the UB comes from - e.g., what is in spec's .or file?

[ambarb]

Tending to favor kind="config" so it gets into the descriptor with the name as provided here

UB at CSX is stored in descriptors and utilized by skbeam.recip. i think it is in the tardis (diffractometer) object

i don't think the lattice reflections are saved. They exists as variables name arbitrarily in the namespace. We do some by hand book-keeping in a file incase bluesky is accidentally closed or crashes.

[ambarb]

also, it appears the configuration_attrs.kind is not correctly configured by default.

[prjemian]

Issue #68 should be part of the actions to resolve this issue.

[prjemian]

UB is saved (now) in a stream's metadata:

import databroker
cat = databroker.catalog["mongodb_config"]
run = cat[-1]
run.primary.metadata["descriptors"][0]["configuration"]["fourc"]["data"]
{'fourc_energy': 8.0,
 'fourc_UB': [[0.2391361194030003, -1.63596736327886, 0.00650159852620056],
  [0.03138404667579824, 0.024468169018191734, 0.5462056759830612],
  [-1.6357846423004514, -0.23869338693437303, 0.011429934593844969]]}

The stream metadata provides no information about the sample or orientation reflections. With the UB matrix, the session can be resumed but the sample and reflection information is useful for a complete set of scientific data. It is simple to add these by setting kwarg kind="config" in the hkl.diffract.Diffractometer() base class.

Demonstration from run with these set:

run.primary.metadata["descriptors"][0]["configuration"]["fourc"]["data"]
{'fourc_energy': 8.0,
 'fourc_energy_units': 'eV',
 'fourc_sample_name': 'PrYBCO',
 'fourc_lattice': [3.8, 3.8, 11.5, 90.0, 90.0, 90.0],
 'fourc_UB': [[0.2391361194030003, -1.63596736327886, 0.00650159852620056],
  [0.03138404667579824, 0.024468169018191734, 0.5462056759830612],
  [-1.6357846423004514, -0.23869338693437303, 0.011429934593844969]],
 'fourc_reflections': [[0.0, 0.0, 1.0], [-1.0, 0.0, 1.0]]}

[prjemian]

Previous example reveals an important synchronization error in the reported energy data (the energy value and units as reported above in the run metadata are incorrect). Here are the true values:

In [89]: fourc.pa()
===================== ==========================================================================
term                  value                                                                     
===================== ==========================================================================
diffractometer        fourc                                                                     
geometry              E4CV                                                                      
class                 FourCircleDiffractometer                                                  
energy (keV)          2.80000                                                                   
wavelength (angstrom) 4.42801                                                                   
calc engine           hkl                                                                       
mode                  bissector                                                                 
positions             ===== ========                                                            
                      name  value                                                               
                      ===== ========                                                            
                      omega 11.10000                                                            
                      chi   88.62089                                                            
                      phi   29.63217                                                            
                      tth   22.20001                                                            
                      ===== ========                                                            
constraints           ===== ========= ========== ================== ====                        
                      axis  low_limit high_limit value              fit                         
                      ===== ========= ========== ================== ====                        
                      omega -180.0    180.0      11.100004855758383 True                        
                      chi   -180.0    180.0      88.62089333948505  True                        
                      phi   -180.0    180.0      29.632172350719483 True                        
                      tth   -180.0    180.0      22.200009711516767 True                        
                      ===== ========= ========== ================== ====                        
sample: PrYBCO        ================= ========================================================
                      term              value                                                   
                      ================= ========================================================
                      unit cell edges   a=3.8, b=3.8, c=11.5                                    
                      unit cell angles  alpha=90.0, beta=90.0, gamma=90.0                       
                      ref 1 (hkl)       h=0.0, k=0.0, l=1.0                                     
                      ref 1 positioners omega=37.08000, chi=89.10000, phi=78.90000, tth=76.25000
                      ref 2 (hkl)       h=-1.0, k=0.0, l=1.0                                    
                      ref 2 positioners omega=3.76000, chi=80.65600, phi=78.90000, tth=150.90000
                      [U]               [[ 0.14462684 -0.98941471  0.01189976]                  
                                         [ 0.01898072  0.01479807  0.99971033]                  
                                         [-0.9893042  -0.14435908  0.02092   ]]                 
                      [UB]              [[ 0.23913612 -1.63596736  0.0065016 ]                  
                                         [ 0.03138405  0.02446817  0.54620568]                  
                                         [-1.63578464 -0.23869339  0.01142993]]                 
                      ================= ========================================================
===================== ==========================================================================

Out[89]: <pyRestTable.rest_table.Table at 0x7f7f38401160>

Here's the problem:

In [90]: fourc.energy.get()
Out[90]: 8.0

In [91]: fourc.calc.energy
Out[91]: 2.8000000000000003

[prjemian]

With the UB, reflection, energy, and sample-related information available from any previous run, access methods are needed to restore that information so that a session can be resumed.

[prjemian]

Also, must include the geometry description as part of the configuration information. Use SignalRO. Absolutely must.

[prjemian]

So, what about the case where more than one sample (and UB) is part of the data? Consider a bicrystal or a sample and substrate as common examples. For now, need to scan each one separately (at least once) to get a run that has recoverable information for each.

[prjemian]

With details of the orientation (UB matrix, sample lattice, and possibly the orientation reflections) available in the descriptors of a run, then restoration can be provided by a method (of the calc) module. The method takes a databroker run as input, checks it for matching diffractometer type and axes names, then creates the sample (if not already defined) and updates the UB and orientation reflections.

When setting each reflection:

1. original_wavelength = calc.wavelength
2. calc.wavelength = reflection["wavelength"]
3. calc.sample.add_reflection(...)
4. calc.wavelength = original_wavelength

[ambarb]

We don't have to use wavelength, right? We can still use energy instead, as we do now, right?

[prjemian]

Right. You can talk energy. In your own units. Behind the scenes, your energy will be converted into wavelength, in angstroms.

And, if you use neutrons or electrons, you must do additional work since the code assumes $E * \lambda= hc$ (X-rays).

[ambarb]

@prjemian your functions like fourc.pa() and fourc.wh() are really useful. It might be nice to also port these things into Bluesky magics. I guess one way is aliases configured on a profile basis, but some of the magics refer to items defined in the aphid device (e.g., %wa optics)

Just going to (AT) all the contributors to bluesky.magics @danielballan @DMagV @mrakitin @tacaswell to see if they have a preference

[prjemian]

@ambarb: @rodolakis suggested that, since we often work with only one diffractometer at a time, a set of functions can be used to shortcut access to the wh() and pa() reports. Keep your eye on #89 (and #63). Magics are a good addition to that, which will simplify further the user experience.

• code: https://github.com/bluesky/hklpy/blob/63-user-interface/hkl/user.py
• tests: https://github.com/bluesky/hklpy/blob/63-user-interface/hkl/tests/test_user.py

Lots of TODO items.

[ambarb]

@prjemian awesome progress. Would like to help review when you are ready.

[ambarb]

@prjemian demo with the primary stream descriptors, can we make these items default for configuration attributes as well as the calc engine and mode (as per the .pa() function output? I don't remember the object names off the top of my head. This will help people that switch modes a lot so they can back track any mistake that they may make.

If too much trouble, then we can cover it in the documentation. But if we agree now what should be default, it is going to help users dig out what is important in the header as it is in the same place every time and it has the same key name.

[prjemian]

This is planned.

…

On Mon, Jan 4, 2021, 5:16 PM Andi Barbour ***@***.***> wrote: @prjemian <https://github.com/prjemian> demo with the primary stream descriptors, can we make these items default for configuration attributes as well as the calc engine and mode (as per the .pa() function output? I don't remember the object names off the top of my head. This will help people that switch modes a lot so they can back track any mistake that they may make. If too much trouble, then we can cover it in the documentation. But if we agree now what should be default, it is going to help users dig out what is important in the header as it is in the same place every time and it has the same key name. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#50 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARMUMFMIF3ESYPKYTTJCDTSYJD5NANCNFSM4S74EPKA> .

[prjemian]

sample for the reflections stream was theoretical

Good. There are some fiddly parts about extracting the actual numbers from the data structures returned from databroker. I believe there will always be some reassembly code necessary when recalling a reflection recovered from databroker before it can be use with tardis.calc.sample.add_reflection(...). (Edited by replacing diffractometer with tardis.)

[ambarb]

I am not super familiar with diffractometer.calc.sample.add_reflection(...). Is this new? Do we have to use this to get diffractometer.calc.sample.UB_compute() to work or is it the other way around or can we by-pass the add_reflection by going straight to UB_compute?

[prjemian]

I was using the generic diffractometer to refer to your tardis instance. Otherwise, this matches with the examples above. I'll change my text above to clarify.

[ambarb]

I'm good with this as long as we can reference this issue in the new issue. I think there is a lot of conversation points that we should loose track of.

From comments above:

However, vote to leave the issue open regardless of the currently associated PR.

The current plans were to close this issue with the PR (which has yet to be created from branch 50-save-the-UB-matrix). Further comments would go into new issues.

Obviously this issue will not be resolved this week as there are two distinct algorithms to be explored in greater depth (we must explore how easy it is for the user to write the orientation and then later recover it):

1. save as stream(s)
2. save in descriptor

The 50-save-the-UB-matrix) branch has an implementation of algorithm 2.

In the interest of getting release 0.3.16 this week, I'm moving this issue to the next milestone (the 1.0.0 release).

[prjemian]

Still thinking of keeping this with the 1.0.0 release. Rather than try to rush this for a deadline this week, let's work it through so the design is clear.

[ambarb]

But for your deadline this week, you are going to include the descriptor stuff, right?

[prjemian]

I am not super familiar with diffractometer.calc.sample.add_reflection(...). Is this new? Do we have to use this to get diffractometer.calc.sample.UB_compute() to work or is it the other way around or can we by-pass the add_reflection by going straight to UB_compute?

The call to add_reflection() is not new. There are (at least) two ways of computing UB with two reflections.

1. define a reflection, define another, call compute_UB(ref1, ref2)
2. define a reflection, define another and add the kwarg compute_ub=True in the call to add_reflection()
3. define some reflections where the last one has the kwarg described above -- UB will be computed using the last two reflections.

        if compute_ub:
            r1 = self._sample.reflections_get()[-1]

        with TemporaryGeometry(calc):
            if position is not None:
                calc.physical_positions = position
            r2 = self._sample.add_reflection(calc._geometry, detector, h, k, l)

        if compute_ub:
            self.compute_UB(r1, r2)

The kwarg compute_ub=True is a shortcut to the process and only eliminates the explicit call to compute_UB(r1, r2)

[prjemian]

Decision

• Will implement UB and orientation reflections save/restore as descriptor.
• Will advise how to implement other cases as additional stream in custom plan.

[prjemian]

items to resolve in test notebook:

(bluesky_2021_1) mintadmin@mint-vm:~/.../epics-bluesky-vm/hklpy$ grep TODO examples/tst_UB_in_descriptor_document.ipynb
    "    # TODO: options for what to restore?\n",
    "        # TODO: How to modify existing lattice?\n",
    "orange = Fourc(\"\", name=\"orange\")  # TODO: remove this line before publishing\n",
    "TODO: How to update the lattice of existing sample?  (namedtuple has a [`_replace()`](https://docs.python.org/3/library/collections.html#collections.somenamedtuple._replace) method)"

no items tagged as FIXME