AIRFLOW-1614 Replace inspect.stack() with sys._getframe()

[gbenison]

inspect.stack() is really expensive, and slows down processing of dags
having large numbers (100s, 1000s) of subdags.

Dear Airflow maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

This patch simply swaps one method of getting the caller's file name for another. There should be no change in behavior (other than performance-wise) for either tests or production code.

[codecov-io]

Codecov Report

Merging #2610 into master will decrease coverage by 0.08%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #2610      +/-   ##
==========================================
- Coverage   70.72%   70.63%   -0.09%     
==========================================
  Files         150      150              
  Lines       11565    11564       -1     
==========================================
- Hits         8179     8168      -11     
- Misses       3386     3396      +10

Impacted Files	Coverage Δ
airflow/models.py	87.02% <100%> (-0.01%)	⬇️
airflow/task_runner/bash_task_runner.py	93.33% <0%> (-6.67%)	⬇️
airflow/utils/helpers.py	53.44% <0%> (-2.88%)	⬇️
airflow/jobs.py	77.61% <0%> (-0.46%)	⬇️
airflow/executors/dask_executor.py	81.39% <0%> (+2.32%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9218a21...6729d07. Read the comment docs.

[saguziel]

Curious, why is this faster? Is there any documentation on this? (Will it be more brittle because it's a private function?)

[gbenison]

Initially, my thinking was that stack() would build a whole stack frame when we're only interested in the second-to-last frame, and that this was the source of the slowness (especially in the context of a web request, where the stack might be quite deep). However, it seems though that _get_frame is much cheaper even with a very short stack: import inspect import sys def f(): fileloc = inspect.getsourcefile(inspect.stack()[1][0]) # This version: 1.66 seconds # fileloc = sys._getframe().f_back.f_code.co_filename # This version: 0.06 seconds for n in range(10000): f() I don't have an explanation for that and I haven't found any documentation for it; it's just an empirical observation for now. As to whether this could be more brittle - it's a great question - I'm not sure at this point... that extra cost for stack() could very well be doing something important.

…

On Thu, Sep 14, 2017 at 3:44 PM, Alex Guziel ***@***.***> wrote: Curious, why is this faster? Is there any documentation on this? (Will it be more brittle because it's a private function?) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2610 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA6S2o2M6FM6cBhrfpXFrmwWcAvvc9IRks5siaxTgaJpZM4PYONy> .

[bolkedebruin]

https://github.com/python/cpython/blob/a72d15c97f9837f6c9c6bd476a62175c942cc588/Lib/inspect.py#L1062

It isn't doing too much I would say. The difference probably is a pointer vs a method that calls several other methods.

LGTM

[saguziel]

I think the difference is the first version may have to open the file for IO.

[saguziel]

From some simple testing, it appears to not break anything.