Add marshal module with ability to serialize/unserialize bytecode functions

[dpgeorge]

Summary

This PR adds the marshal module, with marshal.dumps() and marshal.loads() functions. These functions can serialize/unserialize Python objects, but for now only bytecode functions are supported. The semantics of this module match CPython.

Motivation: the original motivation here was to be able to serialize an existing function, send it over a network connection (or otherwise) to a remote MicroPython device, then unserialize it and execute it. This way one MicroPython device can dynamically execute code on another MicroPython device.

And the original implementation was a bit simpler than this PR, it was a dedicated pair of functions in the micropython module that were just used to serialize/unserialize a (bytecode) function. Eg:

import micropython

def foo():
    print('hello')

serial_foo = micropython.serialize_function(foo)
# send serial_foo somewhere else

# ...
# on the device somewhere else:
foo = micropython.unserialize_function(serial_foo, globals())
foo()

But, instead of a MicroPython-specific API, it's definitely better to try and match an existing CPython API if possible. And in this case the marshal module does almost what was needed to serialize/unserialize bytecode functions. The main difference here is that marshal works with code objects, not functions. So the above code is:

import marshal

def foo():
    print('hello')

serial_foo = marshal.dumps(foo.__code__)

# ...
# at the other end
fun_type = type(lambda: 0)
code = marshal.loads(serial_foo)
foo = fun_type(code, globals())
foo()

Using marshal is a little more involved because you have to know about code objects (which compile() also returns, for example). But at least using marshal is fully compatible with CPython.

This PR implements the marshal module and the above marshal example works in MicroPython with this PR.

Testing

Tests are added to CI, and run on the unix coverage variant.

Trade-offs and Alternatives

The major trade-off/alternative here is a MicroPython-specific API vs a CPython-compatible API, namely the marshal module.

Using a MicroPython-specific API, eg micropython.serialize_function() and micropython.unserialize_function():

• Pro: No need to expose function.__code__.
• Pro: No need to add the ability to create functions from the function type, like type(lambda:0)(code_object, globals()).
• Pro: No added complexity to support a more sophisticated code object.
• Con: Not CPython compatible.
• Con: Not as general as the CPython/marshal approach.

Implementing the marshal module:

• Pro: CPython compatible and more general, eg works alongside the compile() function.
• Pro: Get an efficient implementation of function.__code__, which is what py/objfun: Add function.__code__ attribute #12280 attempted to do.
• Con: Quite a bit more complicated in the implementation, and may use a little more RAM at runtime to create additional objects when marshalling/unmarshalling.

Also note that the pickle module is no good here, because it cannot serialize functions. It simple serializes a reference to a function, which must already be in scope when the reference is unserialized.

[dpgeorge]

@iabdalkader What do you think about this approach using marshal and making it CPython compatible?

[github-actions]

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:   -32 -0.004% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:  +264 +0.072% TEENSY40
        rp2:    -8 -0.001% RPI_PICO_W
       samd:  +276 +0.103% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    -8 -0.002% VIRT_RV32

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.53%. Comparing base (3b62524) to head (e40a3fd).
Report is 6 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #16615      +/-   ##
==========================================
- Coverage   98.59%   98.53%   -0.06%     
==========================================
  Files         167      169       +2     
  Lines       21599    21807     +208     
==========================================
+ Hits        21295    21488     +193     
- Misses        304      319      +15     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[iabdalkader]

@iabdalkader What do you think about this approach using marshal and making it CPython compatible?

I think it's much better: CPython compatibility is always better than custom API, and in the future this can be extended to support other things. I thought I should give it a quick test, and I can confirm everything is still working fine. Amazing, thank you!

This type(lambda: 0) allocates a bit of memory right? Couldn't we instead use something like type(help)? At least if it's used internally in a frozen library. Or perhaps one day we'll have types.FunctionType.

[dpgeorge]

I think it's much better: CPython compatibility is always better than custom API, and in the future this can be extended to support other things.

OK, great. Then I'll stick with this approach of implementing the marshal module.

This type(lambda: 0) allocates a bit of memory right? Couldn't we instead use something like type(help)?

Yes, it allocates a little RAM. But type(help) won't work because that's a built-in function and has a different underlying type, not the same as a bytecode function.

If you have frozen code then lambda: 0 is also frozen. And you only need to get the type once, so I think it's not really a problem to do this.

Or perhaps one day we'll have types.FunctionType.

This already exists and you can use it here.

[dpgeorge]

OK, this PR is done and ready for final review:

• docs for the marshal module have been added
• the new tests now provide full coverage of all code added in this PR
• CI is fully green (except a glitch with codecov)
• ports that don't have features added don't increase in size

[andrewleech]

This looks great thanks! For reference my earlier attempts to expose a __code__ object / attributes was to support introspecting function variables to build a micropython pytest implementation supporting fixtures. From a quick skim / review of the code this certainly looks like it can support that really well.

[projectgus]

Few minor comments, but I'm really excited about this feature! Seems like it will enable some very interesting use cases for distributed MicroPython!

[iabdalkader]

Just to confirm, we don't need to enable MICROPY_PERSISTENT_CODE_SAVE anymore, correct?

[dpgeorge]

Just to confirm, we don't need to enable MICROPY_PERSISTENT_CODE_SAVE anymore, correct?

That is correct.