Development focus areas for 2024

[JukkaL]

I'd like us to make progress in some of these areas in 2024. Feel free to propose more areas!

Python feature support

Add support for missing Python features. At least these are important:

• Real support for generator expressions
• Missing asyncio features
• Descriptors
• The final unsupported dunders (at least those that are commonly used)

Robustness and safety

• Fix bugs, focusing on deviations from CPython semantics.
• Fix type and memory safety issues.
• Improve testing.

Examples include safe # type: ignore comments and detecting stack overflows.

Consider adapting parts of the CPython test suite. Compile more open source code as regression tests.

Peak performance

Support low-level features such as unboxed floats and packed arrays to improve the maximum performance achievable with mypyc.

Reduce overhead of other fundamental operations, such as int and list operations.

Support free threading (no-GIL CPython builds)

Now that CPython is adding free threading support, it would great for mypyc to also support it.

Multiple subinterpreters

Support multiple subinterpreters with separate GILs on Python 3.12 (#993).

Usability

Improve usability. This includes improvements to the command line interface, and error handling and messages.

Performance bottlenecks

Address some of the worst performance bottlenecks in generated code. Focus particularly on cases where performance is worse than CPython, and cases where we generate bad code for common use cases.

Compilation speed

Improve compilation speed through incremental compilation, parallel compilation, making mypyc faster, generating smaller C code, etc.

[JukkaL]

Focus area update

Currently loop/list/integer heavy code can be quite slow relative to lower-level languages. We are faster than CPython, but major performance improvements seem possible here.

Promising improvements include fixed-width integers, packed arrays, and the removal of redundant reference count ops and runtime type checks.

This could also help with compilation speed, since mypyc itself contains a lot of code that currently isn't very efficient (e.g. data flow analysis). If we can speed up compilation, there's more room to implement additional optimizations.

[JukkaL]

Updated the focus areas for 2023 (edited the original summary in place).

[JukkaL]

Added multiple subinterpreter support (#993) as a focus area.

[JukkaL]

Updated for 2024 goals. The main new goal is no-gil support (#1038) -- this seems higher priority than multiple subinterpreters, though both are promising.