Changes for murko found from beamline testing

[DominicOram]

Fixes DiamondLightSource/mx-bluesky#530 and DiamondLightSource/mx-bluesky#147

Built on top of #808 so review/merge that first

Additionally this PR:

• Changes the format of data we store in redis to be just the raw JPEG bytes. This was done after discussion with @aragaod to reduce the memory usage of the redis database.
• Adds a timeout of 30s and so will not stream data for longer than this, FYI @aragaod. We can update this timeout later if needed

Instructions to reviewer on how to test:

1. Confirm tests still pass
2. Confirm device still connects with dodal connect i04

Checks for reviewer

• Would the PR title make sense to a scientist on a set of release notes
• If a new device has been added does it follow the standards
• If changing the API for a pre-existing device, ensure that any beamlines using this device have updated their Bluesky plans accordingly
• Have the connection tests for the relevant beamline(s) been run via dodal connect ${BEAMLINE}

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.59%. Comparing base (e673dbf) to head (0b2a05c).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #809   +/-   ##
=======================================
  Coverage   95.58%   95.59%           
=======================================
  Files         125      125           
  Lines        5419     5422    +3     
=======================================
+ Hits         5180     5183    +3     
  Misses        239      239           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[rtuck99]

this counter signal, is it really a counter? It looks like it's actually uuids? Or are they not really uuids and are just sequence numbers?

[rtuck99]

Q: How frequently does this signal update, is it guaranteed that we will see every frame id that we need to collect?

[DominicOram]

This is the frame counter on the device. We can use it as a UUID as it increments with every frame and we're only using the one camera so will be unique. I suspect there is a roll-over at something around the max int value but given we don't keep data more than a week I think it's unlikely to affect us.

It updates at the rate of the camera (~15Hz). You're right that we may end up missing some, though experimentally we haven't yet, but even if we do it's not critical we catch every frame.

Legitimate questions though, thanks for raising them, I will add some comments in the code to this affect.

[aragaod]

Re: uniqueness of the UUID

Unsure if it is overkill but can envisage during some operational issue a few camera IOC restarts could be needed. It is possible the counter always starts at one from IOC restart. In a combination of IOC restarts plus same sampleID being loaded and data collected on the same frame number (very unlikely combinations) then a UUID collision could technically happen. A possible solution would be to add a prefix of some kind randomly generated.

But unsure if it is worth the effort.

[DominicOram]

Good point, thanks. I will add a proper UUID section in too as a precaution.

[rtuck99]

Looks good apart from maybe change the name of the counter and also query about whether we might get all the frames.

[aragaod]

Before we move forward and forget would like 5 minutes look and maybe A comment left in the code why we make this decision now (allo5wing to change in the future if necessary).

First we decided to not store numpy arrays of the raw images (jpg decompressed) after realising it would occupy enormous amounts of DB space with no real benefit on speed.

Question on current implementation: there 3 ways to store binary jpg into redis

I) directly as an bytesIO object using data.getvalues()
II) serialising it (as we make here with pickle)
III) using a redis optimize structure called redis bitmap for such objects

Happy we stay with (II) as I don't fully know disadvantages of the other ones Vs this one but maybe leave a comment this could be relooked if necessary in the future?

[DominicOram]

Agreed, will add a comment and make an issue to look at a follow-up if we think useful

[DominicOram]

Issue at DiamondLightSource/mx-bluesky#592 to investigate

[DominicOram]

Ah, I actually think this PR already removes the pickling and just stores the raw bytes. I will add a comment though.