Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Send data to detection in chunks #489

Closed
rozetko opened this Issue Mar 29, 2019 · 2 comments

Comments

Projects
None yet
3 participants
@rozetko
Copy link
Member

rozetko commented Mar 29, 2019

Detection works synchronously and blocks event loop but we run detection on the whole dataset which may take a while
It would be better to split dataset by chunks and run detect on them separately

@jonyrock

This comment has been minimized.

Copy link
Contributor

jonyrock commented Mar 30, 2019

Maybe we can use async generators for this

amper43 added a commit that referenced this issue Apr 3, 2019

amper43 added a commit that referenced this issue Apr 3, 2019

Merge branch 'send-data-to-detection-in-chunks-#489' of github.com:ha…
…stic/hastic-server into send-data-to-detection-in-chunks-#489

rozetko added a commit that referenced this issue Apr 3, 2019

@rozetko

This comment has been minimized.

Copy link
Member Author

rozetko commented Apr 3, 2019

Closed via #496

@rozetko rozetko closed this Apr 3, 2019

amper43 added a commit that referenced this issue Apr 4, 2019

amper43 added a commit that referenced this issue Apr 4, 2019

Merge branch 'send-data-to-detection-in-chunks-#489-bugfix' of github…
….com:hastic/hastic-server into send-data-to-detection-in-chunks-#489-bugfix

amper43 added a commit that referenced this issue Apr 4, 2019

Merge branch 'send-data-to-detection-in-chunks-#489-bugfix' of github…
….com:hastic/hastic-server into send-data-to-detection-in-chunks-#489-bugfix

rozetko added a commit that referenced this issue Apr 4, 2019

Send data to detection in chunks #489 (#503)
* Add `get_data_chunks` generator to `utils/dataframe.py`
* Add chunks generator usage to `analytic_worker.py`
* Add tests to `tests/test_detector_chunks.py`
* Minor fixes (constants, etc)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.