0.1 Roadmap

[thautwarm]

A short intro to the architecture (you can escape this and see the roadmap)

We have five parts to implement:

• language features. For instance, import statement is a lang feature. Such part includes PythonAST-to-TraffyASM compilation, and its interpretation in .NET side.
  related source code:

  • Traffy ASM interpreter
  • UnityPython.FrontEnd

• built-in functions and types: built-in functions like print, map, and builtin types like list and Exception
  related source code

  • UnityPython.BackEnd/src/Traffy.Runtime/Builtins.cs
  • UnityPython.BackEnd/src/Traffy.Asm/*.cs

• runtime mechanisms, such as inline cache (for fast attribute reading and writing), coroutine implementations, traceback system, fast method call, etc.
  backend code is more or less related to this topic, but for specific sub-topics:

  • inline cache:
    • UnityPython.BackEnd/src/ICInfrastructure/*.cs
    • UnityPython.BackEnd/src/Traffy.Objects/Object.IC.cs
    • UnityPython.BackEnd/src/Traffy.Objects/Object.IC.cs
  • coroutine (via .NET Task-like types)

  The implementation of Python data model (objects, classes, __init__, __new__, a lot) is roughly a duplicate of CPython, what I've done is just figuring out a maintainable and efficient approach.

• tooling, mainly about IDE support. UnityPython.BackEnd/traffy-types/builtins.pyi can instruct Pylance to analyze UnityPython source. When a new builtin is implemented, we should add type hints in traffy-types/builtins.pyi to support the new stuffs.

• Unity/game engine: we will finally generate Unity API bindings for UnityPython.

Roadmap

• language features
  Most of the language features have been already supported, see UnityPython.FrontEnd.
  The complete Python ASTs are defined at Python 3.10 ASDL

  What have been supported.

  • with
  • await, async, await Task.Yield
  • FormattedValue, JoinedStr (NOTE: ascii conversions ({x:a}) and format specifiers ({a:.2}) are not supported)
  • delete variables/attributes/subscripts: del a; del a[b]; del a.b
  • comprehension expressions: ListComp, SetComp, DictComp, GeneratorExp
  • raise
    • raise e, raise
    • raise e from cause
  • classes
  • other language features

• runtime mechanisms:

  • __abs__ (TODO: add abs builtin to call __abs__)
  • __enter__
  • __exit__ (the third argument of __exit__ is alway None in UnityPython!)

• built-in functions and types (only the necessary ones are included)

  • builtin types

    • object
    • int
    • str (TODO: methods)
    • float (TODO: methods)
    • bool (TODO: methods)
    • bytes (TODO: methods)
    • bytearray (TODO: methods)
    • slice (TODO: methods)
    • (Unimportant) complex, memoryview
    • type
      NOTE: type is the only metaclass in UnityPython!
    • ref: a UnityPython specific builtin, used for fast dictionary/attribute lookup

       if dct.get(key, valueref := ref()):
          print(f"found {key}", valueref.value)

    • classmethod
    • staticmethod
    • property: property.setter, property.getter
      NOTE: fget, fset are exposed yet; property.deleter is not implemented

  • (not in the plan) NotImplemented

  • builtin exception types (https://docs.python.org/3/library/exceptions.html):

    • BaseException
    • SystemExit
    • KeyboardInterrupt
    • GeneratorExit
    • Exception
      • ArithmeticError: subclasses
        • FloatingPointError (for future)
        • OverflowError
        • ZeroDivisionError
      • AssertionError
      • AttributeError
      • BufferError
      • ImportError
        • ModuleNotFoundError
      • LookupError
        • IndexError
        • KeyError
      • NameError
        • UnboundLocalError
      • RuntimeError
        • NotImplementedError
        • RecursionError
        • SyntaxError
      • SystemError
      • TypeError
      • ValueError
      • NativeError represents .NET exceptions

  • builtin functions/callables (https://docs.python.org/3/library/functions.html)

    • iter
    • Task.Yield (UnityPython specific): used for yield in async functions, the semantic is waiting for the next event loop.
    • print
    • map, filter
    • abs (__abs__ support?)
    • all, any
    • bin
    • callable
    • chr, ord, oct, hex
    • getattr, setattr
    • enumerate
    • hash
    • id (impossible to implement, at least cannot be an int)
    • isinstance, issubclass
    • range
    • repr: handle circular references (e.g., a list contained in the itself, deep or shallow)
    • reversed
    • round
    • super
    • sorted
    • sum
    • pow
    • zip
    • __import__
    • globals, vars, locals, breakpoint

• runtime mechanisms

  • inline cache

    object fields are stored in arrays rather than in dictionaries; writing and reading attributes are using inline cache (see IC.*.cs) for fast interpreter and low memory assumption

  • error positions in exception stacktrace

    Stack<int> traceback = frame.traceback;
    callstack.Push(current);
    // interpreter
    callstack.Pop();

    • traceback
    • metadata compression
    • (Enhancement) fold successive tracebacks

  • __init_subclass__

• tooling/IDE:

  • builtins.py: Pylance-specific stub files for UnityPython

• Unity Modules

[thautwarm]

we keep only __name__ and __package__ is not used, this is because the module spec is not dynamically imported. No need to resolve.

[thautwarm]

Intentionally not support __delattr__ because such feature is difficult to work with IDEs.

[thautwarm]

CPython __getattr__ cannot be implemented efficiently as we want to use native c# exceptions as Python exceptions.
Modified __getattr__ protocols can be efficient (getattr(o, attr, ref) -> bool), but this does not work with IDEs because the typing rules of a.b is hard-coded even in Pylance.

I'm going to drop magic methods like __getattr__ and __setattr__ to have fast getattr reflection.

[thautwarm]

we drop __missing__, the reason is similar to dropping __getattr__.

[thautwarm]

If one day any python static analyzer supports defining custom semantics for the syntax a[b] and a.b, we can support them easily.

At the current stage, it's doable to support them all, but this is a two-door design and causes unsolvable performance issues, which is the reason for not supporting them currently.

[thautwarm]

__import__ will not be implemented at the current stage.
We should provide a similar API to wrap RTS.import_from_module.

public static Stack<(TrStr, TrObject)> import_from_module(
       [AllowNull] string module,
       int level,
       Dictionary<TrObject, TrObject> globals,
       [AllowNull] string[] __all__
)

[thautwarm]

Focusing on tests: #26

[thautwarm]

The unsupported features will be documented.