Make get_fastccd_images() work again

[leofang]

With the current version it is not possible for get_fastccd_images() to work as expected:

Traceback (most recent call last):
  File "test_csxtools.py", line 37, in <module>
    image = get_single_image(csx_db, 0, 122630)
  File "test_csxtools.py", line 29, in get_single_image
    itr = load_scan_image_itr(db, scan_num, dark8ID, dark2ID, dark1ID)
  File "test_csxtools.py", line 20, in load_scan_image_itr
    silcerator = get_fastccd_images(db[scan_num], dark_headers=dark_headers)
  File "/opt/conda_envs/ptycho_production/lib/python3.7/site-packages/csxtools/utils.py", line 129, in get_fastccd_images
    return _correct_fccd_images(events, bgnd, flat, gain)
  File "/opt/conda_envs/ptycho_production/lib/python3.7/site-packages/slicerator.py", line 599, in process
    return func(obj, *args, **kwargs)
  File "/opt/conda_envs/ptycho_production/lib/python3.7/site-packages/csxtools/utils.py", line 196, in _correct_fccd_images
    image = correct_images(image, bgnd, flat, gain)
  File "/opt/conda_envs/ptycho_production/lib/python3.7/site-packages/csxtools/fastccd/images.py", line 40, in correct_images
    logger.info("Correcting image stack of shape %s", images.shape)
AttributeError: 'generator' object has no attribute 'shape'

This happens if running on CSX servers with packages installed from the defaults-2019C3.0 channel, but not on JupyerHub srv1 & srv2 (current).

This PR is needed to get get_fastccd_images() working. Also, the ptychography GUI currently relies on this PR (see NSLS-II/ptycho_gui#65).

As discussed below, all csxtools codes relying on slicerator's @pipeline decorator need to be re-worked to be compatible with Databroker's new API.

Closes #64.

[coveralls]

Coverage decreased (-0.4%) to 28.322% when pulling 23d6e6a on leofang:fix_get_images into eb3ab7f on NSLS-II-CSX:master.

[codecov-io]

Codecov Report

Merging #67 into master will not change coverage.
The diff coverage is 0%.

@@           Coverage Diff           @@
##           master      #67   +/-   ##
=======================================
  Coverage   28.72%   28.72%           
=======================================
  Files          10       10           
  Lines         282      282           
=======================================
  Hits           81       81           
  Misses        201      201

Impacted Files	Coverage Δ
csxtools/utils.py	20% <0%> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eb3ab7f...23d6e6a. Read the comment docs.

[leofang]

According to the discussion for this PR in Slack (starting around here: https://lightsource2.slack.com/archives/GBJDHCSTX/p1571145405158100; it's a bit scattered...), a two-step strategy was determined: First, we merge this PR to revert the previous mistaken change so that the tool works. Second, we overhaul csxtools to use new DataBroker APIs. Most likely the usage of the @pipeline decorator and the dependencies on pims and slicerators will be removed altogether.

[leofang]

Just FYI, this issue was brought up to @wen-hu yesterday.

[danielballan]

Note that this partially reverts #61, which seems to have introduced a breaking change that was unnoticed until now because the version of the csxtools deployed for users on JupyterHub is old.

I endorse this approach until we have the capacity to more comprehensively update csxtools.

[mrakitin]

Looks fine for now, as a short-term workaround, as @danielballan mentioned.

The PR description still says that it should not be merged, which is misleading. @leofang, do you mind to correct it?

[mrakitin]

Sadly, I didn't catch the issue in my review of #61 last year. #61 (comment) mentioned the issue, but it didn't bring enough attention. Hopefully we can resolve it soon.

[leofang]

Sorry @danielballan @mrakitin, I was being sloppy and didn't test the case of multiple headers. The bug is now fixed in 23d6e6a so that this works as expected:

from databroker import Broker
from csxtools import get_fastccd_images 
                                                                                                                            
db = Broker.named('csx')                                                                                                                                       
stack = get_fastccd_images((db[122630], db[122631]), (db[122627], db[122628], db[122629]))  

[leofang]

The PR description still says that it should not be merged, which is misleading. @leofang, do you mind to correct it?

@mrakitin Done.

[mrakitin]

Thanks for the fix, @leofang! Looks good to me.