8.0 manage invalid data error

[gfcapalbo]

Problem:
I would have many jobs stuck in start. THese jobs Had all raised InvalidDataError.
Strangely the Exception was not catched in main.py, but the generic

Therefore the store function that had to write my failed job would never trigger

When explicitly importing InvalidDataError, the exception was catched (see commit

gfcapalbo/connector@5b77f3c)

I replaced except (FailedJobError, InvaliDataError, Exception):

with
except:

I am still not sure why this works for me, but it seems OK to set ALL of the forseeable exceptions to fail jobs.

[gfcapalbo]

Discussion continued from #187

[gfcapalbo]

@sbidoul @guewen is there a reason why main.py ==> except (FailedJobError, Exception): ?
I did a little python file with a minimal similar structure InvalidDataTypeError should be caught by Exception.

But it isn't/

Any insight is welcome/

[sbidoul]

@gfcapalbo I just tried this at the root of connector

import exception

try:
    raise exception.InvalidDataTypeError()
except (exception.FailedJobError, Exception):
    print "caught it"

It works as expected.

Can you provide a more detailed example?

[guewen]

I think the issue comes from

connector/connector/exception.py

Line 23 in b5c69d0

class ConnectorException(RuntimeError):

which should inherit from Exception and not RuntimeError. I don't know why the hell I derived it from RuntimeError...

[sbidoul]

@guewen

>>> isinstance(RuntimeError(), Exception)
True

So it must be something else, no?

[guewen]

I think the issue comes from

connector/connector/exception.py

Line 23 in b5c69d0

class ConnectorException(RuntimeError):

which should inherit from Exception and not RuntimeError. I don't know why the hell I derived it from RuntimeError...

Still RuntimeError derives from Exception.

[gfcapalbo]

I think that deriving from RuntimeError or Exception SHouldn't make any difference being that RuntimeError derives from Exception. In a minimal version made in a PY file the same exception Structure works as expected .( it catches InvalidDatarerror too With an except Exception:)

At the time of this posting I had a client waiting for a connector with jobs with invalid data stuck on start. so the content of this MR is clearly the Nuclear Option to deliver a fix.

Doing more tests later i hope to give you more insight.

[lasley]

I 👎 - a specific Exception guard is required everywhere IMO, even if just for Exception. As noted though, all non-system exiting exceptions in Python should inherit Exception in some way, so the global catch-all should be that.

There is also BaseException, but that's not something we're supposed to use out of core AFAIK.

I think the issue is instead possibly an incorrect type comparison deeper down the line. Maybe something less elegant than isinstance:

>>> class TestException(Exception):
...  pass
... 
>>> e = Exception()
>>> t = TestException()
<class '__main__.TestException'>
>>> type(t) == type(e)
False

@gfcapalbo do you maybe have a traceback from when this issue was occurring? Long-shot, but never know what it could hold.

[StefanRijnhart]

Hi, maybe related: today I am hit by jobs stuck in 'started' after a simple except_orm. This exception is caught correctly, but another exception is raised during the update of the job afterwards.

connector/connector/controllers/main.py(120)runjob()
job.set_failed(exc_info=buff.getvalue())
...
connector/connector/queue/model.py(152)write()
res = super(QueueJob, self).write(vals)
*** MissingError: ('MissingError', u'One of the documents you are trying to access has been deleted, please try again after refreshing.')

[StefanRijnhart]

This fixes my problem of jobs getting stuck in 'started' state after an exception was raised during the job: #214. In my case, one or more ORM records were created during the job before the exception was raised. @gfcapalbo was it the same in your case?

[guewen]

#214 is merged, please re-open if needed.