New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enh lazy signal #1219
Merged
francisco-dlp
merged 155 commits into
hyperspy:RELEASE_next_minor
from
to266:ENH_lazy_signal
Feb 12, 2017
Merged
Enh lazy signal #1219
francisco-dlp
merged 155 commits into
hyperspy:RELEASE_next_minor
from
to266:ENH_lazy_signal
Feb 12, 2017
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…thing new tested, everything old passes
…iterable with signal axes indices
Also some words in the User Guide about what happens when mixing lazy and std signals (e.g. |
Forgot that holography types were added, so now I have to make sure they work (and currently they don't for some reason). WIP |
* Fixes hyperspy#1411 * Automatic style corrections courtesy of autopep8
roaring applause |
yes, yes, yes :) congratulations @to266 ! |
Very nice! :) |
👏 🎉 |
🥇 :DD |
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[resubmit of #1102 ]
Adds a new class of "Lazy" signals that only actually operate on data (and even access it) when required or explicitly told to do so.
The intended workflow is as follows:
dask
, and results written to file.One of the drawbacks is that the
dask.array.Array
(which all this PR relies upon) does not support slicing assignment (i.e.a[3] += 2
does not work, have to do something likea = da.concatenate(a[:3], a[3]+2, a[3:])
), so not everything works as seamlessly.Also, lazy signals mean plotting is slow (since it's calculated on requests), but any-size data should be feasible
NEW things
Lazy*Signal
classes: mirror the normal signal classes, but perform the operations lazily.nan*()
methods: just likenumpy.nanmax
and others. Previously were missing...as_lazy()
method: All signals now haveas_lazy()
method, which does the obvious. Will probably be used to convert any other signal into its lazy implementation.compute()
method: Lazy signals can be converted to conventional ones using this method. The better (and more realistic) workflow saves the end lazy signal to a file (since it presumably does not fit in memory all at once).Behaviour Changes
LazySignal
sets the elements outside the ROI tonp.nan
instead of using a masked array, sincedask
does not support masked arrays. Essentially the motivating factor fornan*
methods.memmap
,mmap
,load_to_memory
and similar kwargs, and instead left only the newlazy
kwarg. This means user does not have to pay attention to the format of the file, and still be able to load it lazily, as it's handled by the format reader.get_histogram
: lazy implementations do not support knuth' and 'blocks' bins.ragged
argument inmap
: theragged
is necessary for lazy signals, but optional (i.e. can be determined automatically while running) for normal ones. Ifragged
, the results of themap
are not assumed to be of similar shape or even numpy.arrays, and can be any python object.Notable code changes (relevant for developers)
signal.map
, since it's generalized and works for both conventional and lazy signals.misc.signal_tools.broadcast_signals
added. Used in *nary functions, so no need for new tests.lazifyTestClass
decorator for test classes. Creates newtest_lazy_*
methods from all existingtest_*
methods, where any signal in the class is casted as lazy. Allows reusing most of the regular tests for lazy signals. Can overwrite other class attributes as well.TODO:
bfc
lazy loading