Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use a temporary array to read detector data with pulse selection #220

Merged
merged 3 commits into from Oct 6, 2021

Conversation

takluyver
Copy link
Member

This aims to speed up reading only selected frames from XTDF detector data, using the LPD/AGIPD/DSSC .get_array() method with pulses=. This was prompted by @daviddoji's use case reading LPD data.

I believe this is working around a performance issue in HDF5, which appears to have been around for years. It looks like HDF5 doesn't realise that it can copy large blocks of data, and falls back to copying point by point. By using a temporary array, we can persuade it that the source & destination are sufficiently similar so it works more efficiently. Then we copy the selected data to the output array with numpy.compress().

This involves some extra temporary memory use for the intermediate array. It should never be more than the size of a single file, and it should be possible for the operating system to do virtual memory tricks and only allocate memory for the frames we're actually reading, not the gaps between them.

The timings below for reading LPD parallel gain data with different numbers of pulses selected (e.g. .get_array('image.data', pulses=np.s[:1])). There are 100 pulses per train, but parallel gain mode records all 3 gain stages as separate frames, so we're actually reading 3n of 300 frames in each case. Times are per train; I used 10 or 25 trains to get a better average. I ran each one a couple of times to ensure data was cached.

Pulses selected Parent branch This branch
1 0.56 s 0.026 s
3 1.7 s 0.030 s
10 5.6 s 0.057 s
50 27.8 s 0.20 s
100 (all) 0.201 s 0.204 s

Without this change, you can see that reading even a single frame is slow, and the time scales linearly with the number of frames to read (except when we read all of them). With this change, reading a subset of frames per train is much faster.

@takluyver takluyver added the enhancement New feature or request label Sep 23, 2021
@daviddoji
Copy link

LGTM.
Great improvement @takluyver.

@takluyver
Copy link
Member Author

Thanks David! I'll hold off merging for now, because this goes into the branch for #218, and I don't want to complicate that PR.

@takluyver takluyver added this to the 1.8 milestone Oct 1, 2021
Base automatically changed from refactor-multimod-array to master October 6, 2021 09:26
@takluyver takluyver merged commit e603df9 into master Oct 6, 2021
@takluyver takluyver deleted the pulse-sel-tmp-array branch October 6, 2021 09:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants