Python 3.11.0b3 crashes dill

[DanielNoord]

This seems related to #432.

When running the test suite of pylint I get numerous of these errors:

pickler = <dill._dill.Pickler object at 0x1273546d0>
obj = <code object identity at 0x103cd23f0, file "/Users/daniel/.pyenv/versions/3.11.0b3/lib/python3.11/argparse.py", line 1770>

    @register(CodeType)
    def save_code(pickler, obj):
        log.info("Co: %s" % obj)
        if PY3:
            if hasattr(obj, "co_exceptiontable"):
                args = (
                    obj.co_argcount, obj.co_posonlyargcount,
                    obj.co_kwonlyargcount, obj.co_nlocals, obj.co_stacksize,
                    obj.co_flags, obj.co_code, obj.co_consts, obj.co_names,
                    obj.co_varnames, obj.co_filename, obj.co_name, obj.co_qualname,
>                   obj.co_firstlineno, obj.co_lnotab, obj.co_endlinetable,
                    obj.co_columntable, obj.co_exceptiontable, obj.co_freevars,
                    obj.co_cellvars
E                   AttributeError: 'code' object has no attribute 'co_endlinetable'

I guess some of these attributes changed again?

I'm not sure but some of the discussion in python/cpython#93516 might be relevant. I think they are also discussing whether endlinetable should always exist.

[mmckerns]

Thanks for reporting. Currently dill is tested against everything up to and including Python 3.110b1. I'll grab 3.110b3 now and see if anything has changed. Which version of dill are you working with? I'm assuming master...

[DanielNoord]

I was using 0.3.5.1. I couldn't find any report about this so I though this wasn't noticed yet.

However, looking at #480 I think you might have already fixed this. I'll retry with master.

[DanielNoord]

My bad! This has indeed already been fixed on master. Likely with the fixing of #480.

I should have checked before opening the issue.

Do you have a timeframe for the release of 0.3.6? Or potentially 0.3.5.2?

[mmckerns]

No worries, and thanks for the quick response. Yes, we have a soft release date of June 27.

[DanielNoord]

No worries, and thanks for the quick response. Yes, we have a soft release date of June 27.

Perfect! We'll keep an eye out then.

[DanielNoord]

@mmckerns Sorry for pinging you once again.

I was wondering if you have any update on the next release. No intention of rushing you as I know you're also only a volunteer but if there is anything we can do to help release a patch version with the fix for #480 please let us know. Because of the wide use of pylint within the Python ecosystem we'd like to start testing it.

[mmckerns]

@DanielNoord: everything is ready to go, essentially, with all the major items done. There are a few straggling open issues (#503, #527, #517, and #338) that should require very minimal effort to resolve, and I expect them to go into the release. I expect the straggling issues to be resolved in a matter of days. We also have two other issues (#346 and #475) tagged for the release, but I assume we will need to make a decision on whether to punt those to the next release or not. Push comes to shove, the blocker is #527 (and maybe #517, but I haven't decided how to handle that one yet as it's a python bug that python decided it won't resolve). @leogama @anivegesana: FYI

[DanielNoord]

@mmckerns Would you consider releasing a patch release with the 3.11 compatibility included? We'd like to start testing pylint for 3.11 and dill is a crucial dependency for us. I'm not sure how much effort that would be for you, but it would be much appreciated!

[mmckerns]

I expect to be closing #500 and #517 before next Monday. I'm punting #338. The blocker is #527, which either requires some surgical rollback or we finish the PR. My intent was to make a patch release a month ago, however we decided to include new development as well... some of which is now causing a delay. Hopefully @leogama and I can wrap up #527 shortly.

[leogama]

@DanielNoord FYI, my personal deadline for closing the major open PRs I'm involved with is this weekend. I'm pretty sure the technical challenges are already solved. What's missing is mostly unit testing and documentation.

[leogama]

My additions for the 0.3.6 milestone, both high and medium priority, are complete. Only some review is pending.

Brace yourselves, release is coming.

[DanielNoord]

Sorry for nagging on about this. Are you still planning on releasing 0.3.6? If not I think I'm going to fork the project just so we can start testing pylint on 3.11. We found some interesting bugs already locally on 3.11 but we probably need our CI to be up and running to be 100% sure.

Just to be clear, I don't intend to maintain such a fork after you have released 0.3.6 but we just need something to be available on PyPi.

Thanks for all the work you're doing 😄

[leogama]

I'm not sure about @mmckerns schedule this week, but he said he would review #475 today or tomorrow. This is the only open PR with the 0.3.6 milestone, so I assume the new version is about to be released in a matter of days. If you absolutely can't wait, you may fork the master branch if your code doesn't make use of the session saving feature (dill.dump_module(), previously dill.dump_session()), or the PR branch if it does use that.

FYI, in the last month, we worked on the handling of special cases for features introduced by #527 (which was merged with #475 lately), and I was busy documenting, annotating code, adding type hints, writing unit tests and fixing bugs of this massive PR. Since the weekend we have enough code coverage, and dill.dump_module() was tested with the new features over the Standard Library modules with 100% success for Python3.11rc1.

[mmckerns]

@DanielNoord: Do whatever you need to do. I expect the review and any follow-up might take several days. #475 is all we have left before 0.3.6, but it's a bit of a beast. One way or the other, we will likely have a resolution shortly after 3.11.0rc2 is released, and definitely a good bit before 3.11.0 final.

[EwoutH]

If 0.3.6 is a bit of a hassle because #475 is blocking it, would a 0.3.5.2 patch release with this fix be an option for the mean time? That might also take a bit of the pressure of #475.

[DanielNoord]

For anybody looking to get the fix from PyPI there is:
https://pypi.org/project/dill-pylint/

I screwed up the versioning but it's just a mirror/fork of main of 2/3 weeks ago. I don't maintain it or update it, but we use it in pylint to be able to run on 3.11.