Initial implementation of Scan component

[JamesWrigley]

Leaving this as a draft for now because I still need to write tests, but y'all are welcome to try it on real data to make sure I'm not missing something obvious 😅

Known issues (that may or may not be fixed in the future):

• Automatic resolution detection on non-linear scans is dodgy.
• The first and last detected steps may be part of the motor backlash rather than part of the scan. Haven't thought much about how, but it would be nice to automatically detect and remove them.

[philsmt]

xarray is really slow to import and can take up to 10s of seconds especially from GPFS environments. Unlike in most other places it does look like this component will always use it, but since it's imported alongside any other component, I would prefer a lazy import.

[JamesWrigley]

Sure, fixed in 7f69671.

[philsmt]

When an xarray.DataArray is passed, it may be nice to set this argument manually via an optional argument or try to take it from the DataArray's name (KeyData.xarray() will set this to {source}.{key} by default).

[JamesWrigley]

Ah nice, didn't know about DataArray.name. Added in 7f69671.

[JamesWrigley]

I rebased the branch, added some tests, and cleaned up the code a bit. Still not quite happy with it so I'm gonna leave it as a draft for now.

[philsmt]

I gave it a try for a demo in the FXE workshop, and it worked beautifully. Doing so I found a few minor usability improvements, but this component is frankly awesome.

[JamesWrigley]

Rebased and fixed a few things, but there's one more issue with noisy data (noticed in the FXE demo) I want to fix before this is ready for review.

[philsmt]

I was also wondering whether the fixtures could directly return the DataCollection 🤔
@takluyver Could there be any state implications by doing so?

[JamesWrigley]

Ok, I think this is ready for review now.

Recent changes:

• Added an intra_step_filtering parameter for deciding how aggressively to filter trains within a step. Tweaking this fixes the issue I saw with FXE's noisy mono scan.
• Moved the parameter docs into their individual docstrings.
• Made __repr__() do something more conventional and recommend .info() instead to look at the scan info. Added .format() in case someone wants to save the string somewhere.

[JamesWrigley]

(if there's no objections I'll probably merge this at the end of next week)

[philsmt]

LGTM!

One additional idea I got while remaking the XAS demo on top of AdqChannel and PumpProbePulses:

• A property yielding tuples of position and extra_data.by_id slices.

[JamesWrigley]

The problem with using slices is that they'll include any trains that are filtered out within the steps. My inclination is to stick with lists of train IDs, but then again I don't think I've seen any scans that have jumps within the steps 🤔 Lemme think about that.

[philsmt]

Oh certainly, but I don't think it's that much of a deal, and slices would be there for convenience anyway. We could have an optional strict-style argument that throws an exception if no contiguous slices can be made. I'm curious though, what are the reasons right now a train in the middle can be filtered out?

[JamesWrigley]

There's actually none that I can think of, the original reason I did it that way is to be on the safe side in case there was some badly behaving motor (or other device). But I've never seen it happen so it's probably safe to assume that we always want all trains within a step, or make that the default behaviour at least.

[JamesWrigley]

I tried to switch to by_id by default, but that made the API a bit awkward because it's difficult to convert a by_id[] into a list of train IDs to easily find e.g. the length of a step (I know you could check the start/stop values directly but that's less convenient than len()) or pass it to DataArray.select(trainId=...) ☹️ So I stuck with a list of train IDs, but made it include all trains within a step in 90d0603 so we at least get the behaviour of by_id[], which I agree with you should be the default.