Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow traceback objects to be instantiated/mutated/annotated #74764

Closed
njsmith opened this issue Jun 6, 2017 · 11 comments
Closed

Allow traceback objects to be instantiated/mutated/annotated #74764

njsmith opened this issue Jun 6, 2017 · 11 comments
Labels
3.7 interpreter-core type-feature

Comments

@njsmith
Copy link
Contributor

@njsmith njsmith commented Jun 6, 2017

BPO 30579
Nosy @brettcannon, @jcea, @ncoghlan, @njsmith, @ericsnowcurrently, @1st1, @miss-islington
PRs
  • #4793
  • #5653
  • #5655
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2018-01-07.13:31:58.450>
    created_at = <Date 2017-06-06.03:41:53.475>
    labels = ['interpreter-core', 'type-feature', '3.7']
    title = 'Allow traceback objects to be instantiated/mutated/annotated'
    updated_at = <Date 2018-06-09.13:58:32.107>
    user = 'https://github.com/njsmith'

    bugs.python.org fields:

    activity = <Date 2018-06-09.13:58:32.107>
    actor = 'jcea'
    assignee = 'none'
    closed = True
    closed_date = <Date 2018-01-07.13:31:58.450>
    closer = 'ncoghlan'
    components = ['Interpreter Core']
    creation = <Date 2017-06-06.03:41:53.475>
    creator = 'njs'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 30579
    keywords = ['patch']
    message_count = 11.0
    messages = ['295235', '295236', '295237', '295238', '295248', '308023', '309532', '309615', '309616', '312107', '312109']
    nosy_count = 7.0
    nosy_names = ['brett.cannon', 'jcea', 'ncoghlan', 'njs', 'eric.snow', 'yselivanov', 'miss-islington']
    pr_nums = ['4793', '5653', '5655']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue30579'
    versions = ['Python 3.7']

    @njsmith
    Copy link
    Contributor Author

    @njsmith njsmith commented Jun 6, 2017

    Currently, traceback objects don't expose any public constructor, are immutable, and don't have a __dict__ or allow subclassing, which makes it impossible to add extra annotations them.

    It would be nice if these limitations were lifted, because there are rare but important cases where code needs to manipulate tracebacks directly, and currently these have to use awful stuff like ctypes. For example:

    Jinja2:
    https://github.com/pallets/jinja/blob/bec0065c4e7e89e7d893eda7840ba0219824b23c/jinja2/debug.py#L298-L372

    Trio:
    https://github.com/python-trio/trio/blob/496493afecc22d7d1a17175b6a2748a9c3510066/trio/_core/_multierror.py#L233-L318

    (Notice that on PyPy there are no ctypes, but instead they have special extension that's supported for this case only to keep jinja2 working.)

    For the above cases, what's needed is the ability to instantiate and assign to the fields of traceback objects.

    In addition, in trio I'd like to be able to annotate traceback objects so that the traceback printing machinery can do things like hide "internal" tracebacks, or highlight places where exceptions jumped between tasks. This would be much easier if there were some way to attach data to tracebacks. Probably it doesn't make sense for traceback objects to have a __dict__ by default for speed/memory reasons, but we could add a dedicated metadata slot that is normally empty but can have arbitrary data assigned, or if they allowed subclassing then I could add a __dict__ in a subclass

    I'm CC'ing the "import machinery" interest list, because my understanding is that with the new import system there's a similar desire to hide "internal" traceback frames, and while the features requested in this bug report won't solve that problem directly they might (are intended to) provide some useful machinery for it. Feel free to un-CC if I'm wrong :Jinja2:
    https://github.com/pallets/jinja/blob/bec0065c4e7e89e7d893eda7840ba0219824b23c/jinja2/debug.py#L298-L372
    -).

    I think that this should be very straightforward to implement: it's just that no-one ever implemented setters etc. for tracebacks.

    @njsmith njsmith added 3.7 interpreter-core type-feature labels Jun 6, 2017
    @njsmith
    Copy link
    Contributor Author

    @njsmith njsmith commented Jun 6, 2017

    Uh, please ignore the random second paste of the jinja2 URL in the middle of the second to last paragraph.

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Jun 6, 2017

    Rather than allowing this on tracebacks directly, I'd prefer to build on the "TracebackException" work and allow traceback summaries in all the places where we currently require real tracebacks (including exception __traceback__ attributes): https://docs.python.org/3/library/traceback.html#tracebackexception-objects

    The essential requirement here would be to abstract out a "traceback protocol" from the current concrete traceback objects, and adapt affected parts of the interpreter (such as the traceback display on shutdown) to use that abstract protocol when dealing with a non-native traceback object.

    @njsmith
    Copy link
    Contributor Author

    @njsmith njsmith commented Jun 6, 2017

    My understanding is that the major difference between a real traceback object and a TracebackException object is that the latter is specialized for printing, so it can be lighter weight (no pinning of frame objects in memory), but loses some utility (can't do post-mortem debugging).

    If that's right, then that's definitely not a solution, because trio and jinja2 and import errors all need to support post-mortem debugging.

    I'm not against the idea of defining a traceback protocol, but it seems like a lot of work when the existing traceback objects are already perfectly good container objects that are just missing a few simple features.

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Jun 6, 2017

    Aye, if you need full post-mortem debugging support, then updating the public API of the actual traceback objects based on the current ctypes-based workaround does seem like a better path forward.

    TracebackException and friends would more come into play for the "make traceback printing easier to customise" part of your presentation.

    @njsmith
    Copy link
    Contributor Author

    @njsmith njsmith commented Dec 11, 2017

    PR 4793 provides everything that Jinja2 and Trio actually need right now. It doesn't provide any way to annotate tracebacks with extra data, but I'm not entirely sure if that's necessary or what it would look like, so I figured I'd at least get this part in for 3.7.

    @njsmith
    Copy link
    Contributor Author

    @njsmith njsmith commented Jan 6, 2018

    Ping -- anyone up for reviewing PR 4793?

    #4793

    It's pretty straightforward, and I figure better to ping now and beat the end-of-month rush :-)

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Jan 7, 2018

    New changeset e46a8af by Nick Coghlan (Nathaniel J. Smith) in branch 'master':
    bpo-30579: Allow TracebackType creation and tb_next mutation from Python (GH-4793)
    e46a8af

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Jan 7, 2018

    Thanks for the patch!

    @ncoghlan ncoghlan closed this as completed Jan 7, 2018
    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Feb 13, 2018

    New changeset aec7532 by Nick Coghlan in branch 'master':
    bpo-30579: Docs for dynamic traceback creation (GH-5653)
    aec7532

    @miss-islington
    Copy link
    Contributor

    @miss-islington miss-islington commented Feb 13, 2018

    New changeset 53374cc by Miss Islington (bot) in branch '3.7':
    bpo-30579: Docs for dynamic traceback creation (GH-5653)
    53374cc

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 interpreter-core type-feature
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants