Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjust timeout automatically to amount of returned time / processing speed #255

Open
JelleAalbers opened this issue Apr 13, 2020 · 0 comments
Labels
enhancement New feature or request

Comments

@JelleAalbers
Copy link
Member

JelleAalbers commented Apr 13, 2020

Each plugin's mailbox times out processing when it has received no new data for too long, suspecting strax might be in deadlock.

The amount of time per chunk can very greatly across datatypes, especially when they are loaded from disk. Some small high-level ones might have only one chunk per run, while big low-level ones have one every few seconds.

Thus, if processing uses a one-chunk datatype, the timeout must currently be as long as the entire expected duration of the processing, or it causes unnecessary crashes. These will show up as e.g.

strax.mailbox.MailboxReadTimeout: <Mailbox: peaklet_classification_mailbox> could not progress beyond 1, no driving subscriber requested it

For now, I'm going to set the timeout to some silly high number, e.g. 24 hours. Effectively this means we don't have protection from hangs anymore.

It would be good to adjust the timeout per datatype to the amount of time per chunk and the processing speed somehow. Or replace the mailbox system altogether, see #81.

... or, we could add a 'global' timeout, which checks how long we see nothing new in any mailbox at all.

JelleAalbers added a commit to JelleAalbers/strax that referenced this issue Apr 13, 2020
@JelleAalbers JelleAalbers added the enhancement New feature or request label May 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant