Add Stacktrace when DagFileProcessorManager gets killed

[msumit]

Scheduler - DagFileProcessorManager is the process that often gets hung due to various reasons. Found that having access to the stacktrace would be very helpful in debugging the root cause.

Should be helpful in resolving issue #7935

[XD-DENG]

Given we still officially support Python 3.5, maybe not the best time yet to introduce f-string (which is only supported since 3.6)?

[potiuk]

Given we still officially support Python 3.5, maybe not the best time yet to introduce f-string (which is only supported since 3.6)?

But we are already using a lot of f-strings. 3.5 is only supported in 1.10.* line. In master we are 3.6+

[XD-DENG]

Given we still officially support Python 3.5, maybe not the best time yet to introduce f-string (which is only supported since 3.6)?

But we are already using a lot of f-strings. 3.5 is only supported in 1.10.* line. In master we are 3.6+

Cool, makes sense to me.

[msumit]

@potiuk @XD-DENG can you guys approve it, if looks fine?

[XD-DENG]

👍

[potiuk]

Ah. And now @XD-DENG is going to kill me :) ...

I commented the "we are past 3.5" case, but when I looked at this closely .... I think you should not use the f-string in fact.

For a completely different reason though. It's OK to use f-strings in general - when it comes to logic or info messages, but in case of debug statements - it's not a good idea because of performance reasons. Debug statements are usually not printed in "production" environment. But if you use f-string or .format(), those interpolations are executed regardless of the setting of log level.

So for debug statements you should use this form:

    log.debug("Disposing DB connection pool (PID %s)", os.getpid())

As this will only run string interpolation when it is about to be printed.

[XD-DENG]

Change my review status based on latest comment from @potiuk

[XD-DENG]

Ah. And now @XD-DENG is going to kill me :) ...

I commented the "we are past 3.5" case, but when I looked at this closely .... I think you should not use the f-string in fact.

For a completely different reason though. It's OK to use f-strings in general - when it comes to logic or info messages, but in case of debug statements - it's not a good idea because of performance reasons. Debug statements are usually not printed in "production" environment. But if you use f-string or .format(), those interpolations are executed regardless of the setting of log level.

So for debug statements you should use this form:

    log.debug("Disposing DB connection pool (PID %s)", os.getpid())

As this will only run string interpolation when it is about to be printed.

Definitely would love instead of kill you😄😄 thanks for taking the second look and the update @potiuk !

[kaxil]

Nice one @msumit