Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'NoneType' object is not callable when setting a tracing function #1769

Closed
trofimander opened this issue Jul 13, 2017 · 12 comments
Closed

'NoneType' object is not callable when setting a tracing function #1769

trofimander opened this issue Jul 13, 2017 · 12 comments

Comments

@trofimander
Copy link
Contributor

trofimander commented Jul 13, 2017

This bug breaks debugging of Cython with PyCharm and pdb. Below is a small project to reproduce the problem.

Consider this fork of a Cython tutorial at Europython 2017: https://github.com/traff/optimization-rimini-2017

  1. Go to prj folder.
  2. Execute python setup.py develop
  3. Run unit tests in ../tests/test_prj.py

The process will terminate with the following exception:

File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/main.py", line 93, in __init__
    self.parseArgs(argv)
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/main.py", line 140, in parseArgs
    self.createTests()
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/main.py", line 147, in createTests
    self.module)
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 219, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 219, in <listcomp>
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 153, in loadTestsFromName
    module = __import__(module_name)
  File "/Users/traff/PycharmProjects/optimization-rimini-2017_2/prj/tests/test_prj.py", line 2, in <module>
    from prj import _pyappend
  File "/Users/traff/PycharmProjects/optimization-rimini-2017_2/prj/prj/__init__.py", line 8, in <module>
    from ._mod import _append
  File "prj/_mod.pyx", line 3, in init prj._mod (prj/_mod.c:1342)
  File "prj/_mod.pyx", line 3, in PyMODINIT_FUNC PyInit__mod(void)
TypeError: 'NoneType' object is not callable

Note that when sys.setprofile(fun) is executed along with settrace there is no exception.

@trofimander
Copy link
Contributor Author

Correction: the error disappears not when sys.setprofile(fun) is called, but when the tracing function fun returns not None, but itself.
(I made the wrong conclusion because made 2 changes simultaneously).

@trofimander
Copy link
Contributor Author

I found the problem: in CPython a call event tracing sets the result returned by the tracing function to the filed frame->f_trace. And a line event tracing checks whether frame.f_trace is null or not and if not, then calls it.
Cython checks tstate->c_tracefun in both cases which seems to be a bug, that leads to the call on the Py_None object.

trofimander added a commit to trofimander/cython that referenced this issue Jul 16, 2017
…function that returns None (cython#1769)

CPython uses the return value of tracing function to assign it to frame.f_trace field and then it uses it for the frame instead of tstate.c_tracefunc. If it is None, the context shouldn't be traced line by line.
scoder added a commit that referenced this issue Sep 1, 2017
… results for both Python and Cython functions.

Fix "frame.f_trace" test in tracing code accordingly.
See #1769 and #1774 for background.
@scoder
Copy link
Contributor

scoder commented Sep 1, 2017

Does 71c1ad4 also solve this?

@scoder scoder added this to the 0.27 milestone Sep 13, 2017
@scoder
Copy link
Contributor

scoder commented Sep 13, 2017

I'll assume that this is fixed. Please re-open if it does not work for you.

@scoder scoder closed this as completed Sep 13, 2017
@PDiracDelta
Copy link

PDiracDelta commented Nov 20, 2018

I'm using Cython 0.28.5 (which should include this commit, right?) and I still get this error if I try to debug (with PyCharm) while my cythonCode.pyx file contains the linetrace=True compiler directive.

@scoder
Copy link
Contributor

scoder commented Nov 23, 2018

Should be fixed since 0.27, yes. Seems worth a new ticket. Please try 0.29 and provide all information you can gather, especially the Python version, a reproducer snippet and the exact error that you get. Also note that debugging Cython code is different from tracing it and is not necessarily supported by PyCharm, I don't know. It's compiled native code, after all, which is not directly introspectable. Probably worth asking on their side first, to see what their experience is and what support they promise.

Here are two (multi-file) tests that are known to work:
https://github.com/cython/cython/blob/master/tests/run/line_profile_test.srctree
https://github.com/cython/cython/blob/master/tests/run/line_trace.pyx

@PDiracDelta
Copy link

PDiracDelta commented Feb 13, 2019

@scoder damn, I tried to create a minimal working example... but I can't seem to reproduce the issue out of the scope of my real project, even when the new project uses the same conda environment! Do you think it could be an issue with how PyCharm handles debugging, or is that independent? I've been having other issues with that lately...

@scoder
Copy link
Contributor

scoder commented Feb 13, 2019

No idea. The last change in the profiling code was for 0.28, so 0.28.5 should have all there is. Unless you can cut down the crashing code somehow, it's unlikely that anyone else can do something about it. You could still try to get it to run under gdb to get more infos about the crash.

@pseudotensor
Copy link

repro (from https://stackoverflow.com/questions/32163436/python-decorator-for-printing-every-line-executed-by-a-function):

import sys

class debug_context():
    """ Debug context to trace any function calls inside the context """

    def __init__(self, name):
        self.name = name

    def trace_calls(self, frame, event, arg):
        # We want to only trace our call to the decorated function
        if event != 'call':
            return
        elif frame.f_code.co_name != self.name:
            return
        # return the trace function to use when you go into that
        # function call
        return self.trace_lines

    def trace_lines(self, frame, event, arg):
        # If you want to print local variables each line
        # keep the check for the event 'line'
        # If you want to print local variables only on return
        # check only for the 'return' event
        if event not in ['line', 'return']:
            return
        co = frame.f_code
        func_name = co.co_name
        line_no = frame.f_lineno
        filename = co.co_filename
        local_vars = frame.f_locals
        print ('  {0} {1} {2} locals: {3}'.format(func_name,
                                                  event,
                                                  line_no,
                                                  local_vars))

    def __enter__(self):
        print('Entering Debug Decorated func')
        # Set the trace function to the trace_calls function
        # So all events are now traced
        sys.settrace(self.trace_calls)

    def __exit__(self, *args, **kwargs):
        # Stop tracing all events
        sys.settrace = None


def debug_decorator(func):
    """ Debug decorator to call the function within the debug context """
    def decorated_func(*args, **kwargs):
        with debug_context(func.__name__):
            return_value = func(*args, **kwargs)
        return return_value
    return decorated_func

@debug_decorator
def testing() :
    a = 10
    b = 20
    c = a + b

testing()

With python:

 python testtracelines.py 
Entering Debug Decorated func
  testing line 57 locals: {}
  testing line 58 locals: {'a': 10}
  testing line 59 locals: {'a': 10, 'b': 20}
  testing return 59 locals: {'a': 10, 'b': 20, 'c': 30}

With pytest:

E
==============================================================================
ERROR: tracelines.py::testing
------------------------------------------------------------------------------
Entering Debug Decorated func

Traceback (most recent call last):
  File "/home/jon/h2oai.fullcondatest/tracelines.py", line 50, in decorated_func
    with debug_context(func.__name__):
  File "/home/jon/h2oai.fullcondatest/tracelines.py", line 40, in __enter__
    sys.settrace(self.trace_calls)
TypeError: 'NoneType' object is not callable

------------------------------------------------------------------------------
Ran 1 tests in 0.01s

FAILED (errors=1)

If ever one uses pytest, seems can't use this technique, which basically makes it impossible to use since pytest or nosetest are universally used for testing ones code.

@da-woods
Copy link
Contributor

da-woods commented Dec 27, 2020

@pseudotensor It works for me on the current 0.29.x and master branches.

You don't get any useful information from attempting to trace Cython functions of course (and you most likely never will), but it doesn't crash.

(It's possible I've misunderstood what you actually need to do to run it though)

@pseudotensor
Copy link

pseudotensor commented Dec 27, 2020

Python 3.6.10 | packaged by conda-forge | (default, Apr 24 2020, 16:44:11) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cython
>>> cython.__version__
'0.29.21'
>>> 

I stumbled upon this issue by googling. If this error is not related to cython itself, then I've found wrong place to post.

@da-woods
Copy link
Contributor

So to clarify:

  • if I run python3 testtracelines.pyI get the same result as you
  • If I run cythonize -if testtracelines.py then python3 -c "import testtracelines" I just get Entering Debug Decorated func and nothing else
  • If I run pytest I get 0 tests run in 0.01s (which suggests I'm using pytest wrong to be honest)

It isn't obvious to me from your description that you're using Cython at all here, so I think you possibly have found the wrong place

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants