v3.0 Tracking Issue

[MikePall]

This is a locked tracking issue. Discussion takes place in individual issues. Related issues will be created by the admins, when needed.

Do not open issues or PRs for any of these, or for unsolicited features. Do not work on or submit code, unless a discussion has been fully resolved and you have been tasked with the implementation.

Issues listed here may be handled in the v3.0 development cycle, but there's no guarantee for that.

The following issues are grouped by their scope. The groups are listed in approximate dependency/implementation order.

Initial Bringup and Cleanup

• TBD: Scope and non-scope of v3.0 branch.
• TBD: Open design issues, implementation order, schedule.
• TBD: Sponsoring for continued maintenance and development.

• Open v3.0 branch
• Remove obsolete archs and arch-variants, see status page
• Remove obsolete target platforms
• TBD

Compile all/most standard library functions

• REQ: Simplify/unify this between interpreter and JIT backend.
• REQ: Do not compromise pure interpreter performance (Consoles, iOS).
• TBD: Make FFI mandatory? Needs pre-compiled equivalent for interpreter.
• TBD: More discussion and planning required.

• Reorganize simple builtins #10
• Replace more builtins with bytecode #15
• Replace package.* library with pure Lua code #22
• [ISSUE] Compile os.* and io.* functions

Compile all/most bytecodes

• [ISSUE] Compile closure creation, TBD effects on IR consistency

Syntax Extensions

• REQ: Must not affect existing source code. Bytecode changes are OK.

• Add bitwise operators #63, TBD metamethods or not
• [ISSUE] Ternary a?b:c operator
• [ISSUE] Optional dereference '?.' operator
• [ISSUE] Floor division operator // and __idiv metamethod
• [ISSUE] Soft continue keyword
• [ISSUE] Allow underscores in number literals (lexer, not tonumber)

Compile-Time Evaluation

• TBD: define scope, goals, non-goals and safety.

Garbage Collector

• TBD: Initial scope and MVP.

• New Garbage Collector #38
• [ISSUE] Public API to set allocator limits
• [ISSUE] Allocator hardening
• [ISSUE] Account for auxiliary memory in GC pacing, TBD ffi.gc size parameter
• [ISSUE] Memory profiler

Ports

• List of current and planned ports.

• [ISSUE] Enable unaligned access for more targets
• Add support for RISC-V #628, TBD RISC-V RV64 RVA22+, TBD unmergable

Cleanups, Semantic Consistency

• [ISSUE] Cleanup/simplify IR_HREF, TBD call helpers for uncommon paths
• [ISSUE] Consistent behavior of FP to unsigned int conversions (Fix x86-64 JIT conversion of negative floats to unsigned ints #415) (Miscompile on x64 with certain -march flags? #865 (comment))
• [ISSUE] 32/64 bit constants and widening (X64 widening elimination can interact badly with CSE #1086)
• Consider eliminating cur_L #1095

FFI Feature Completion

• FFI: Correctly handle redeclarations #28
• FFI: Handle all constant declarations #29
• FFI: Compile calls to C functions with aggregates passed or returned by value #57
• FFI: Add internal pre-processor #27
• Parse 64 bit bit fields (64bit struct Bit Fields invalid value near ':' #951)
• Parse 128 bit types

Bytecode Portability and Determinism

• TBD: Maybe convert all ports (even 32 bit) to LJ_FR2? LJ_GC64 has a performance impact, but LJ_FR2 very little.

JIT-Compiler Enhancements

• Fix table bump optimization #606
• [ISSUE] Allocation sinking enhancements (Stores to dead, unused objects prevent allocation sinking #651) (Improve Sink Optimization #652)

Pattern Matching Rewrite

• REQ: Only handles Lua patterns, not full regex. No extensions planned.
• REQ: Must be fast when only interpreted.
• REQ: Must be fast for one-off use, too. Avoid allocating memory.
• REQ: Handle dynamic patterns (misuse) reasonably.
• FYI: 2-phase parse/exec bytecode-based prototype available, but does not fulfill all REQ.
• TBD: Explore lazy parse + cache + trace approach. Explore QDFA.
• TBD: Prevent ReDOS or not, consider implementation complexity and efficiency.

• [ISSUE] Pattern matcher rewrite
• [ISSUE] Pattern matching methods for string buffers
• [ISSUE] JIT-compiled pattern matching

Repurpose Global+ Scope

• GOAL: Improve performance for resolving module methods.
• GOAL: Remove need for local abs = math.abs idiom, which is just red tape.
• GOAL: No change required for most modular codebases. Immediate benefit.
• REQ: Make this mode optional behind a CLI flag (name TBD).
• REQ: Bytecode generation shortcuts resolution to a constant (upvalue-like).
• REQ: Bind early at load. Collateral benefit: exposes misspellings like a linter.
• REQ: Lift top-level require to shortcut 2nd-level resolution, too.
• TBD: Specify exact constraints and effects. Check larger codebases.
• TBD: Consistency is same as the idiom. Subsequent writes do not affect the local copy.

• [ISSUE] Repurpose global+ scope

Debugging

• TBD: Check third-party extensions that dig into VM internals.
• TBD: Evaluate demand.

• [ISSUE] Dump on assertion, TBD dump format, dump analysis, hookable
• [ISSUE] Public API for VM introspection
• [ISSUE] Fast breakpoints, TBD bytecode/mcode patching

Unconfirmed, Unclear

• TBD: More likely to be moved to following versions.
• TBD: any enhancement issues not yet tagged with a version

• Make FFI mandatory
• Make compiler extensible by loadable modules
• Split optional features to loadable modules, default is builtin
• More serializers, e.g. JSON, REQ: optional/module