Enh lazy signal

[to266]

[resubmit of #1102 ]
Adds a new class of "Lazy" signals that only actually operate on data (and even access it) when required or explicitly told to do so.

The intended workflow is as follows:

1. Load the data lazily.
2. Perform calculations (will most likely be fast / instantaneous) or operations
3. Save the result signal to a file. Then the operations are performed by dask, and results written to file.

One of the drawbacks is that the dask.array.Array (which all this PR relies upon) does not support slicing assignment (i.e. a[3] += 2 does not work, have to do something like a = da.concatenate(a[:3], a[3]+2, a[3:])), so not everything works as seamlessly.

Also, lazy signals mean plotting is slow (since it's calculated on requests), but any-size data should be feasible

NEW things

• Lazy*Signal classes: mirror the normal signal classes, but perform the operations lazily.
• nan*() methods: just like numpy.nanmax and others. Previously were missing...
• "ONMF" decomposition: Online NMF implementation as per this paper by Zhao et.al. In particular it's the OPGD implementation in the paper. Not yet thoroughly tested
• as_lazy() method: All signals now have as_lazy() method, which does the obvious. Will probably be used to convert any other signal into its lazy implementation.
• compute() method: Lazy signals can be converted to conventional ones using this method. The better (and more realistic) workflow saves the end lazy signal to a file (since it presumably does not fit in memory all at once).

Behaviour Changes

• CircleROI for LazySignal sets the elements outside the ROI to np.nan instead of using a masked array, since dask does not support masked arrays. Essentially the motivating factor for nan* methods.
• Lazy loading: deprecated all memmap, mmap, load_to_memory and similar kwargs, and instead left only the new lazy kwarg. This means user does not have to pay attention to the format of the file, and still be able to load it lazily, as it's handled by the format reader.
• Stacking : Majorly rewritten, can now perform the stacking lazily. Also supports stacking numbers (floats, integers, complex numbers) that get broadcasted as appropriate.
• get_histogram: lazy implementations do not support knuth' and 'blocks' bins.
• ragged argument in map: the ragged is necessary for lazy signals, but optional (i.e. can be determined automatically while running) for normal ones. If ragged, the results of the map are not assumed to be of similar shape or even numpy.arrays, and can be any python object.

Notable code changes (relevant for developers)

• Rewrote a bunch of iterating algorithms to use signal.map, since it's generalized and works for both conventional and lazy signals.
• misc.signal_tools.broadcast_signals added. Used in *nary functions, so no need for new tests.
• lazifyTestClass decorator for test classes. Creates new test_lazy_* methods from all existing test_* methods, where any signal in the class is casted as lazy. Allows reusing most of the regular tests for lazy signals. Can overwrite other class attributes as well.

TODO:

• stacking
• most existing readers can load lazily
• lazy/iterating decomposition tests (mainly O(R)NMF)
• bfc lazy loading
• developer guide update
• "Big Data" chapter in the user guide
• Holography signals

[francisco-dlp]

Also some words in the User Guide about what happens when mixing lazy and std signals (e.g. lazy + std) would be useful.

[to266]

Forgot that holography types were added, so now I have to make sure they work (and currently they don't for some reason). WIP

[thomasaarholt]

roaring applause

[sem-geologist]

yes, yes, yes :)

congratulations @to266 !

[magnunor]

Very nice! :)

[vidartf]

👏 🎉

[to266]

🥇 :DD