Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

matlab.engine.EngineError: MATLAB function cannot be evaluated #5

Open
learntolearn11 opened this issue Jul 26, 2016 · 9 comments
Open

Comments

@learntolearn11
Copy link

[2016-07-26 20:49:44,869 24261 greedy_tracking_from_raw_dets.py:58 INFO] Tracking lion...
[2016-07-26 20:49:45,243 24261 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-07-26 20:49:45,244 24261 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class lion
[2016-07-26 20:49:45,244 24261 greedy_tracking_from_raw_dets.py:58 INFO] Tracking motorcycle...
[2016-07-26 20:49:45,617 24261 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-07-26 20:49:45,618 24261 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class motorcycle
[2016-07-26 20:49:45,618 24261 greedy_tracking_from_raw_dets.py:58 INFO] Tracking cattle...
[2016-07-26 20:49:45,990 24261 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-07-26 20:49:45,991 24261 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class cattle
[2016-07-26 20:49:45,991 24261 greedy_tracking_from_raw_dets.py:58 INFO] Tracking airplane...
[2016-07-26 20:49:46,336 24261 track.py:222 INFO] tracking top No.0 in ILSVRC2015_val_00007011
[2016-07-26 20:49:46,336 24261 common.py:341 DEBUG] Use opened Matlab session: <matlab.engine.matlabengine.MatlabEngine object at 0x7f1e308fbcd0>
[2016-07-26 20:50:08,061 24261 common.py:341 DEBUG] Use opened Matlab session: <matlab.engine.matlabengine.MatlabEngine object at 0x7f1e308fbcd0>
[2016-07-26 20:50:09,544 24261 common.py:348 ERROR] Existing engine no response: <matlab.engine.matlabengine.MatlabEngine object at 0x7f1e308fbcd0>
[2016-07-26 20:50:13,866 24261 common.py:341 DEBUG] Use opened Matlab session: <matlab.engine.matlabengine.MatlabEngine object at 0x7f1e711e4710>
[2016-07-26 20:50:29,874 24261 common.py:348 ERROR] Existing engine no response: <matlab.engine.matlabengine.MatlabEngine object at 0x7f1e711e4710>
Traceback (most recent call last):
File "tools/tracking/greedy_tracking_from_raw_dets.py", line 61, in
cls_idx, opts)
File "./vdetlib/vdet/track.py", line 234, in greedily_track_from_raw_dets
new_tracks = track_method(vid_proto, anchor_frame_id, anchor_bbox, opts)
File "./vdetlib/vdet/track.py", line 82, in fcn_tracker
[matlab.double(bbox),] + fw_frames + [opts.gpu,], opts.engine)
File "./vdetlib/utils/common.py", line 349, in matlab_engine
raise e
matlab.engine.EngineError: MATLAB function cannot be evaluated
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(_targs, *_kargs)
File "/home/tju/.local/lib/python2.7/site-packages/matlab/engine/init.py", line 186, in exit_engines
eng().exit()
File "/home/tju/.local/lib/python2.7/site-packages/matlab/engine/matlabengine.py", line 246, in exit
pythonengine.closeMATLAB(self.__dict
["_matlab"])
SystemError: MATLAB process cannot be terminated.
Error in sys.exitfunc:
Traceback (most recent call last):
File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(_targs, *_kargs)
File "/home/tju/.local/lib/python2.7/site-packages/matlab/engine/init.py", line 186, in exit_engines
eng().exit()
File "/home/tju/.local/lib/python2.7/site-packages/matlab/engine/matlabengine.py", line 246, in exit
pythonengine.closeMATLAB(self.__dict
["_matlab"])
SystemError: MATLAB process cannot be terminated.
Exception SystemError: 'MATLAB process cannot be terminated.' in <bound method MatlabEngine.del of <matlab.engine.matlabengine.MatlabEngine object at 0x7f1e711e4710>> ignored

doses anyone happen to this? please let me know your solution, Thanks!

@zawlin
Copy link

zawlin commented Aug 26, 2016

try adding caffe.reset_all(); to fcn_tracker.m just before caffe.set_mode_gpu();. at least for me the memory keep running out because each call of the script recreate the entire network and the matlab engine keep crashing, that's why you see this error.

@myfavouritekk
Copy link
Owner

@zawlin Right. reset_all() will clear stand-alone nets in the GPU. I think adding reset_all() for each fcn_tracker call may solve this problem, but it requires loading VGG for each tracking process and may takes a lot of time compared to tracking itself. Probably should call rest_all only when restarting a matlab engine. What do you think?

@zong209
Copy link

zong209 commented Oct 12, 2016

I got the same error. I try to add caffe.reset_all; in fcn_tracker.m, but the same crash happened. I try to add it in track.py by matlab_energy(), but I am failed. Now I don‘t know how to solve it.

[2016-10-12 11:26:06,145 10341 greedy_tracking_from_raw_dets.py:42 INFO] Video: ILSVRC2015_val_00007011
[2016-10-12 11:26:09,939 10341 greedy_tracking_from_raw_dets.py:58 INFO] Tracking turtle...
[2016-10-12 11:26:10,188 10341 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-10-12 11:26:10,190 10341 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class turtle
[2016-10-12 11:26:10,190 10341 greedy_tracking_from_raw_dets.py:58 INFO] Tracking antelope...
[2016-10-12 11:26:10,421 10341 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-10-12 11:26:10,422 10341 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class antelope
[2016-10-12 11:26:10,422 10341 greedy_tracking_from_raw_dets.py:58 INFO] Tracking bicycle...
[2016-10-12 11:26:10,659 10341 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-10-12 11:26:10,660 10341 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class bicycle
[2016-10-12 11:26:10,660 10341 greedy_tracking_from_raw_dets.py:58 INFO] Tracking lion...
[2016-10-12 11:26:10,894 10341 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-10-12 11:26:10,895 10341 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class lion
[2016-10-12 11:26:10,895 10341 greedy_tracking_from_raw_dets.py:58 INFO] Tracking motorcycle...
[2016-10-12 11:26:11,127 10341 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-10-12 11:26:11,129 10341 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class motorcycle
[2016-10-12 11:26:11,129 10341 greedy_tracking_from_raw_dets.py:58 INFO] Tracking cattle...
[2016-10-12 11:26:11,363 10341 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-10-12 11:26:11,365 10341 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class cattle
[2016-10-12 11:26:11,365 10341 greedy_tracking_from_raw_dets.py:58 INFO] Tracking airplane...
[2016-10-12 11:26:11,560 10341 track.py:222 INFO] tracking top No.0 in ILSVRC2015_val_00007011
[2016-10-12 11:26:11,561 10341 common.py:341 DEBUG] Use opened Matlab session: <matlab.engine.matlabengine.MatlabEngine object at 0x7f5fe9562d10>
[2016-10-12 11:26:14,244 10341 common.py:348 ERROR] Existing engine no response: <matlab.engine.matlabengine.MatlabEngine object at 0x7f5fe9562d10>
[2016-10-12 11:26:17,436 10341 common.py:341 DEBUG] Use opened Matlab session: <matlab.engine.matlabengine.MatlabEngine object at 0x7f5fe95a8550>
[2016-10-12 11:26:20,180 10341 common.py:348 ERROR] Existing engine no response: <matlab.engine.matlabengine.MatlabEngine object at 0x7f5fe95a8550>
Traceback (most recent call last):
  File "tools/tracking/greedy_tracking_from_raw_dets.py", line 61, in <module>
    cls_idx, opts)
  File "./vdetlib/vdet/track.py", line 234, in greedily_track_from_raw_dets
    new_tracks = track_method(vid_proto, anchor_frame_id, anchor_bbox, opts)
  File "./vdetlib/vdet/track.py", line 82, in fcn_tracker
    [matlab.double(bbox),] + fw_frames + [opts.gpu,], opts.engine)
  File "./vdetlib/utils/common.py", line 349, in matlab_engine
    raise e
matlab.engine.EngineError: MATLAB function cannot be evaluated
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/local/lib/python2.7/dist-packages/matlab/engine/__init__.py", line 186, in __exit_engines
    eng().exit()
  File "/usr/local/lib/python2.7/dist-packages/matlab/engine/matlabengine.py", line 246, in exit
    pythonengine.closeMATLAB(self.__dict__["_matlab"])
SystemError: MATLAB process cannot be terminated.
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/local/lib/python2.7/dist-packages/matlab/engine/__init__.py", line 186, in __exit_engines
    eng().exit()
  File "/usr/local/lib/python2.7/dist-packages/matlab/engine/matlabengine.py", line 246, in exit
    pythonengine.closeMATLAB(self.__dict__["_matlab"])
SystemError: MATLAB process cannot be terminated.
Exception SystemError: 'MATLAB process cannot be terminated.' in <bound method MatlabEngine.__del__ of <matlab.engine.matlabengine.MatlabEngine object at 0x7f5fe95a8550>> ignored

My GPU is GTX960M with 2GB, is it the reason? Does it mean that I should use an another caffemodel ?

@myfavouritekk
Copy link
Owner

@zong209 what is matlab_energy?

@zong209
Copy link

zong209 commented Nov 29, 2016

I'm sorry, I didn't express it clearly @myfavouritekk . I actually try to add reset_all(); before matlab.engine.start.
For above problem, I changed my GPU into GTX1080, then added caffe.reset_all(); before caffe.set_mode_gpu(); in fcn_tracker.m. The crash didn't happen again.

@Jensen-Su
Copy link

Jensen-Su commented Dec 29, 2016

I've tried adding caffe.reset_all(); before caffe.set_mode_gpu(); in fcn_tracker.m right as what @zawlin @myfavouritekk and @zong209 suggested. And my GPU is TITAN X with 12GB. But nothing changed, why?

Cleared 0 solvers and 0 stand-alone nets
[2016-12-29 19:08:59,996 8369 greedy_tracking_from_raw_dets.py:42 INFO] Video: ILSVRC2015_val_00007011
[2016-12-29 19:09:03,220 8369 greedy_tracking_from_raw_dets.py:58 INFO] Tracking turtle...
[2016-12-29 19:09:03,597 8369 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-12-29 19:09:03,599 8369 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class turtle
[2016-12-29 19:09:03,599 8369 greedy_tracking_from_raw_dets.py:58 INFO] Tracking antelope...
[2016-12-29 19:09:03,978 8369 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-12-29 19:09:03,980 8369 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class antelope
[2016-12-29 19:09:03,980 8369 greedy_tracking_from_raw_dets.py:58 INFO] Tracking bicycle...
[2016-12-29 19:09:04,359 8369 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-12-29 19:09:04,361 8369 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class bicycle
[2016-12-29 19:09:04,361 8369 greedy_tracking_from_raw_dets.py:58 INFO] Tracking lion...
[2016-12-29 19:09:04,748 8369 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-12-29 19:09:04,750 8369 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class lion
[2016-12-29 19:09:04,750 8369 greedy_tracking_from_raw_dets.py:58 INFO] Tracking motorcycle...
[2016-12-29 19:09:05,135 8369 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-12-29 19:09:05,136 8369 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class motorcycle
[2016-12-29 19:09:05,136 8369 greedy_tracking_from_raw_dets.py:58 INFO] Tracking cattle...
[2016-12-29 19:09:05,520 8369 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-12-29 19:09:05,522 8369 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class cattle
[2016-12-29 19:09:05,522 8369 greedy_tracking_from_raw_dets.py:58 INFO] Tracking airplane...
[2016-12-29 19:09:05,867 8369 track.py:222 INFO] tracking top No.0 in ILSVRC2015_val_00007011
[2016-12-29 19:09:05,868 8369 common.py:341 DEBUG] Use opened Matlab session: <matlab.engine.matlabengine.MatlabEngine object at 0x7f560e086190>
[2016-12-29 19:09:22,661 8369 common.py:341 DEBUG] Use opened Matlab session: <matlab.engine.matlabengine.MatlabEngine object at 0x7f5632c66ed0>
Traceback (most recent call last):
  File "tools/tracking/greedy_tracking_from_raw_dets.py", line 61, in <module>
    cls_idx, opts)
  File "./vdetlib/vdet/track.py", line 234, in greedily_track_from_raw_dets
    new_tracks = track_method(vid_proto, anchor_frame_id, anchor_bbox, opts)
  File "./vdetlib/vdet/track.py", line 82, in fcn_tracker
    [matlab.double(bbox),] + fw_frames + [opts.gpu,], opts.engine)
  File "./vdetlib/utils/common.py", line 354, in matlab_engine
    eng.clear(nargout=0)
  File "/usr/local/lib/python2.7/dist-packages/matlab/engine/matlabengine.py", line 84, in __call__
    _stderr).result()
  File "/usr/local/lib/python2.7/dist-packages/matlab/engine/futureresult.py", line 106, in result
    self._future,self._nargout, None, out=self._out, err=self._err)
matlab.engine.EngineError: MATLAB function cannot be evaluated
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/local/lib/python2.7/dist-packages/matlab/engine/__init__.py", line 158, in __exit_engines
    eng().exit()
  File "/usr/local/lib/python2.7/dist-packages/matlab/engine/matlabengine.py", line 258, in exit
    pythonengine.closeMATLAB(self.__dict__["_matlab"])
SystemError: MATLAB process cannot be terminated.
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/local/lib/python2.7/dist-packages/matlab/engine/__init__.py", line 158, in __exit_engines
    eng().exit()
  File "/usr/local/lib/python2.7/dist-packages/matlab/engine/matlabengine.py", line 258, in exit
    pythonengine.closeMATLAB(self.__dict__["_matlab"])
SystemError: MATLAB process cannot be terminated.
Exception SystemError: 'MATLAB process cannot be terminated.' in <bound method MatlabEngine.__del__ of <matlab.engine.matlabengine.MatlabEngine object at 0x7f5632c66ed0>> ignored

@mingde510
Copy link

I added caffe.reset_all(); before caffe.set_mode_gpu(); in fcn_tracker.m, the problem had not solve,
Can you help me?@myfavouritekk

@EswarSaiKrish
Copy link

EswarSaiKrish commented Apr 4, 2018

@mingde510 or anyone else, were you able to solve this issue finally ?
I am using a GTX 1060 6 GB Variant and I get the same issue as this thread, MATLAB process cannot be terminated.

I have tried adding caffe.reset_all(); in fcn_tracker.m just before caffe.set_mode_gpu(); like @myfavouritekk, @zong209 has suggested, but my terminal output says Cleared 0 solvers and 0 stand-alone nets
And then I get the same error.

Anyone has any success??
Thanks.

@qiumei1101
Copy link

so who solve this problem now, thanks?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants