Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not-ending learning #264

Closed
jonyrock opened this Issue Nov 25, 2018 · 11 comments

Comments

Projects
None yet
3 participants
@jonyrock
Copy link
Contributor

jonyrock commented Nov 25, 2018

I think I am not the only how experienced when you start a "General" learning and it never ends

image

@jonyrock

This comment has been minimized.

Copy link
Contributor Author

jonyrock commented Nov 25, 2018

@ashwin42 mentioned:
I set up a few more analytics. While this time the server did not crash the analytics has gone in to pending and the server process is running at 100% continuously.
image

Do you think it is not able to get in to multiprocessing?

@ashwin42 please let me know if I should delete this info

@jonyrock jonyrock added the bug label Dec 6, 2018

@jonyrock

This comment has been minimized.

Copy link
Contributor Author

jonyrock commented Dec 7, 2018

It might be because analytic fails and restarts without restarting learning

@jonyrock

This comment has been minimized.

Copy link
Contributor Author

jonyrock commented Dec 11, 2018

It is not because of restarting. General pattern indeed could hang just with two steps:

  • label a big segment
  • save
  • delete something predicted
  • save
@rozetko

This comment has been minimized.

Copy link
Member

rozetko commented Dec 11, 2018

2018-12-11 12:41:23,076 [Analytics] [ERROR]  handle_analytic_task exception: 'Traceback (most recent call last):
  File "bin/../analytics/analytic_unit_manager.py", line 77, in handle_analytic_task
    result_payload = await self.__handle_analytic_task(task)
  File "bin/../analytics/analytic_unit_manager.py", line 71, in __handle_analytic_task
    return await worker.do_detect(data, payload['cache'])
  File "bin/../analytics/analytic_unit_worker.py", line 34, in do_detect
    return self._detector.detect(data, cache)
  File "bin/../analytics/detectors/pattern_detector.py", line 50, in detect
    detected = self.model.detect(dataframe, cache)
  File "bin/../analytics/models/model.py", line 47, in detect
    ) for x in result]
  File "bin/../analytics/models/model.py", line 47, in <listcomp>
    ) for x in result]
  File "/usr/local/lib/python3.6/site-packages/pandas/core/series.py", line 601, in __getitem__
    result = self.index.get_value(self, key)
  File "/usr/local/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2477, in get_value
    tz=getattr(series.dtype, 'tz', None))
  File "pandas/_libs/index.pyx", line 98, in pandas._libs.index.IndexEngine.get_value (pandas/_libs/index.c:4404)
  File "pandas/_libs/index.pyx", line 106, in pandas._libs.index.IndexEngine.get_value (pandas/_libs/index.c:4087)
  File "pandas/_libs/index.pyx", line 154, in pandas._libs.index.IndexEngine.get_loc (pandas/_libs/index.c:5126)
  File "pandas/_libs/hashtable_class_helper.pxi", line 759, in pandas._libs.hashtable.Int64HashTable.get_item (pandas/_libs/hashtable.c:14031)
  File "pandas/_libs/hashtable_class_helper.pxi", line 765, in pandas._libs.hashtable.Int64HashTable.get_item (pandas/_libs/hashtable.c:13975)
KeyError: 2794
'
@jonyrock

This comment has been minimized.

Copy link
Contributor Author

jonyrock commented Dec 11, 2018

Maybe it is important that General can`t detect anything after labeling

@rozetko

This comment has been minimized.

Copy link
Member

rozetko commented Dec 11, 2018

For some reason this error doesn't reach panel (analytic unit status is not updated)

@jonyrock

This comment has been minimized.

Copy link
Contributor Author

jonyrock commented Dec 11, 2018

@rozetko error doesn't reach panel cuz python process itself is down

@jonyrock

This comment has been minimized.

Copy link
Contributor Author

jonyrock commented Dec 11, 2018

This issue reveals following sub-issues:

  • Analytics should not be down and send message if #264 (comment) occurs
  • Error #264 (comment) must not happen
  • Status of analyticUnits should be updated after analytics restart

@rozetko we will create these sub-issues later

@rozetko

This comment has been minimized.

Copy link
Member

rozetko commented Dec 11, 2018

@jonyrock the 3rd sub-issue is already created btw: #251

@rozetko

This comment has been minimized.

Copy link
Member

rozetko commented Jan 21, 2019

I've managed to reproduce the "infinite" learning behavior without analytics failing
Reproduces only in "General" model
Steps to reproduce:

  • label something
  • save
  • label something else
  • save
  • learning will not end

@VargBurz do you know possible reasons maybe?

@rozetko

This comment has been minimized.

Copy link
Member

rozetko commented Feb 12, 2019

Not all analytics errors are still logged: #407

@rozetko rozetko reopened this Feb 12, 2019

rozetko added a commit that referenced this issue Feb 25, 2019

amper43 added a commit that referenced this issue Mar 14, 2019

rozetko added a commit that referenced this issue Mar 14, 2019

rozetko added a commit that referenced this issue Mar 29, 2019

Not-ending learning #264 (#484)
- optimize general model
- add logs

This was referenced Apr 8, 2019

@rozetko rozetko closed this in #535 Apr 8, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.