Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Protect OverlapWindowPlugin against empty chunks #212

Merged

Conversation

JelleAalbers
Copy link
Member

@JelleAalbers JelleAalbers commented Oct 10, 2019

As discovered by @jorana, strax's OverlapWindowPlugin crashes with an IndexError if an empty chunk is found in the data stream. In general strax is not very robust against this case (see #162), but we are improving.

You can read how OverlapWindowPlugin works here. For an empty chunk, this PR changes strax to simply not send or discard any data; the input/output caches remain the same. The case of the last chunk (after which you must send everything even it if is empty) is already handled by the iter method.

This escaped the property-based testing due to a spurious condition in the test (I guess I just didn't consider empty chunks would be possible) that is now removed.

(This also removes some old but totally unrelated commented code)

@JoranAngevaare
Copy link
Member

Thanks Jelle! It does solve the problem but causes problems elsewhere as it will keep waiting after which it crashes due to an strax.plugin.InputTimeoutExceeded error.

Could this be that the problem lies in that from line 52335ec#diff-db9a6da7a746e72e441939269037c9e6R126 there will never be an ‘end’ so it will not know that that chuck is finished (but just empty)?

Multiprocessing gives strange errors: apparently the classattribute overwriting for .provides does not propagate well... I could fix the multi_output check but who knows what else goes wrong.
@JelleAalbers
Copy link
Member Author

InputTimeoutExceeded should come from the DAQReader, this is an exception raised as part of the online input support. I would guess this is an unrelated problem, e.g. maybe something is wrong with how that particular run ended?

@JelleAalbers JelleAalbers merged commit 3b249da into AxFoundation:master Nov 12, 2019
@JelleAalbers JelleAalbers deleted the empty_chunks_overlapwindow branch November 12, 2019 13:41
JelleAalbers added a commit to JelleAalbers/strax that referenced this pull request Jan 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants