Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compile Python Scripts On the Go #2709

Open
timpur opened this issue Dec 20, 2016 · 60 comments

Comments

@timpur
Copy link

commented Dec 20, 2016

I was wondering if it was possible to compile python scripts on the go. The idea would be you can upload a script (.py) file to the board and get the board to compile and save the compiled code. No need to build custom firmware. The compiled script don't need to be loaded into ram and run from flash, just like the firmware and core modules.

The reason is, it would be nice for newbie (me a month ago) who doesn't know the complexity of creating a firmware to be able to have the benefits of frozen scripts (Minimum Ram Usage) without going to the trouble of setting up the SDK.

Having feature would also be awesome because, say you had an IOT device located remotely. You could webrepl into the device, upload a new script (.py) compile it on the board, move it to the lib folder (or something), remove the script (.py) and execute the new compiled script from flash. Now doesn't take any time to compile or use any ram to execute (the script doesn't live in ram, unlike .py scripts)

The saved compiled version could be a .mpy or .pyc, not sure about the details, that's why im asking you, the awesome developers. What do you think of a feature like this and is it possible? Tell me your thoughts.

It would be nice to create "Frozen Scripts" with out creating a custom firmware. Lets use that awesome builtin Interpreter to its full potential.

Functionality:

  • Python Scrips compiled on board and saved as a file in the flash

  • Compiled Scripts execute from flash (no loading into ram)

Outcome:
Minimum Ram Usage for custom scripts.

@dpgeorge

This comment has been minimized.

Copy link
Member

commented Dec 20, 2016

Yes this is entirely possible, and definitely a good idea. A lot of the building blocks are already in place but it would still take some effort to implement properly. Feel free to propose how a user would use this feature, eg what functions they would call to dynamically freeze a script, where it would go in flash, etc.

@timpur

This comment has been minimized.

Copy link
Author

commented Dec 21, 2016

I believe there is a Python 3 Module for this. I imagine we should implement this following the same as python 3. I have never used the module, but i think it does what ive mentioned above.

py_compile (https://docs.python.org/3/library/py_compile.html)

Following this module you would call something like this:
py_compile.compile(sourceFile, outputFile)
where the prams are string locations to files on the filing system eg '/folder/file.py'

On another note, in the documentation in Built-in Functions there is a 'compile()' function that hasn't been implemented on the esp8266 at lest (i only run MP on that)?. Not sure what the story is about that built in function?

It would be nice to see this come to life :)
Thanks for your interest.

@timpur

This comment has been minimized.

Copy link
Author

commented Dec 21, 2016

Or could we incorporate the "MicroPython cross compiler" or something alike that gets the job done?

@robert-hh

This comment has been minimized.

Copy link
Contributor

commented Dec 21, 2016

I use frozen bytecode and mpy-cross most of the time when the code is too large to get compiled on board. Embedding the compiler into the board would not help in that case. The compile pass would still run out of memory, I guess.
And since most, if not all builds can execute pre-compiled code from the file system, providing mpy-cross variants in the download section of MicroPython would do the same.

@timpur

This comment has been minimized.

Copy link
Author

commented Dec 22, 2016

I do understand that you can use the MicroPython cross compiler to achieve a similar outcome. Buts thats not the vision. The vision is, to achieve compiled python code without having to create a custom firmware. The Vision of only ever using the pre built MicroPython Firmware, but having the functionality of custom firmware.

This means for newbies and basic level programmers , dont have to touch the Toolchain and SDK to achieve advantages of precompiled scripts. Plus the compiler is already there, why not use it.

Also to my understanding, scripts that are interpreted (.py) get compiled when imported, this takes up ram temporarily during the compilation, but also once compiled the compiled version of the script lives in ram, which uses that ram till it is 'unloaded', on top of that the compiled script uses more ram as it executes and starts using resources (normal stuff).

The key idea here is to only compile once, and execute from flash. So the only ram being used the the normal ram used by execution (if any of that makes sense :P). So as you can see, this is a functional and usability feature, that i can imagine many can benefit from who use micropython.

Extending this functionality, it would be cool to see tools like ampy (https://github.com/adafruit/ampy) and webrepl, implent this as an aditional option to instead of only uploading a .py module, take that .py module compile it on the board and save it to the flash. The .py module is never saved to your board. Quick and Easy. Then we all could share and upload new module when ever and never needing to reflash the board just to use one new module.

Hopefully you guys also see my vision of how easy and useful this would be :).

@peterhinch

This comment has been minimized.

Copy link
Contributor

commented Dec 22, 2016

@timpur There is a second use-case which @robert-hh is referring to. That is the case where a module is so large that it cannot be compiled by the compiler on the target hardware because the compiler runs out of RAM. This code size limit is soon reached on RAM challenged targets like the ESP8266. On the Pyboard it typically occurs at about 1K LOC. In this case the only option is to cross-compile it.

This can currently be done in two ways, one being (as you're aware) to incorporate it into a firmware build as frozen bytecode. The second way is to cross compile it to a .mpy file, transfer that to the filesystem of the target, and execute it from there. Cross compiling it simply involves running the command line tool mpy-cross. However in this case the code is loaded into RAM for execution.

The ideal solution would cope with this use-case too, somehow cross-compiling the code before putting it into flash in such a way that it could be executed from there. I can't quite envisage how this would proceed but perhaps you have some ideas?

@robert-hh

This comment has been minimized.

Copy link
Contributor

commented Dec 22, 2016

You suggest to store compiled objects, similar to .mpy files created by mpy-cross, in flash in a format suitable for direct execution. I see one aspect where you suggestion definitely helps: boards without a file system, like the 512k esp8266 or . But these are typically also short of flash memory. About RAM: the question is, where code that is too large to fit in RAM for execution can be compiled on-board. At least that time it must fit into the RAM, even if it is stored in flash after that.
In your suggestion I see however the opportunity to pre-compile native, viper and assembler code too, because for these the absolute storage addresses can already be determined during compile time.
About re-flash: storing a module to flash is a (partial) re-flash. What about storing multiple modules? Freeing space? Would that require a re-flash? Maybe it's possible to find a way for executing modules out of the file system, given that they are stored properly = sequentially.

@dhylands

This comment has been minimized.

Copy link
Contributor

commented Dec 22, 2016

In general, the filesystem can't guarantee that sequential blocks in the file are sequential on the filesystem.

You would also only be able to use this for a fileystem on internal flash. sdcard or spi-flash based filesystems can't be executed from directly.

@peterhinch

This comment has been minimized.

Copy link
Contributor

commented Dec 22, 2016

@robert-hh Arm Thumb machine code is position independent is it not?

@chuckbook

This comment has been minimized.

Copy link

commented Dec 22, 2016

Before frozen bytecode was available I did some tests with a super primitive ROM 'filesystem' that allowed to use the idle flash space. However, such a fs image has to be compiled and downloaded. This is easy in the lab but for general usage there has to be an adequate infrastructure (flash erase, download, validation & programming, not to mention access locks etc.).
There is a chance that I have to reanimate this in the near future but I have no clue how this might fit for system architectures other than ARM (eg. where direct execution from flash is possible).

@robert-hh

This comment has been minimized.

Copy link
Contributor

commented Dec 23, 2016

So I can imagine two possible use cases for the on-board compile:

  1. w/o files: compile a script during loading and load it into flash into the space, where frozen bytcode resides. The dictionary would have to be extended, and some kind of space management would be required (startof free ares + length or remaining space). That would be extend-only. Erasing by re-flash of the device, Advantage for the user: Just a raw repl access needed, which woudl be possible for almost any platform/any device.
  2. with files: Compile on-board from a .py file or raw code coming in though the terminal interface into a .mpy file. The advantage for the user: no need for a cross compiler on the external platform.

In any case, the compile could support also platform specific properties like compile of native/viper/assembler code), which b.t.w. the cross compiler could already support now during creating of aflash image.. The drawback would be the same: large code would still have to be compiled externally and loaded as .mpy file or embeded into the build, and so the biggest advantage of frozen bytecode would still not be achieved, at least for me.
P.S.: Did someone ever think of making the frozen objects at least visible, like as read-only special objects in the file system?

@hoihu

This comment has been minimized.

Copy link
Contributor

commented Dec 31, 2016

I like the idea - it's a nice addition to frozen byte code and makes it updateable in the field (that's in my opinion the target use-case here). It makes sense not only for ressource constraint systems, but also to update mission critical modules (which shouldn't be part of the potential insecure file system).

Regarding space management: One of the advantages of frozen bytecode is its secureness (no data corruption possible). It would be nice if the update process is done in a way that there is little to no chance of data corruption.

A simplified update approach may be to use bank swapping for the frozen bytecode flash segment.

The idea would be to copy the frozen segement from bank A to bank B (or vice versa). While copying, patch/add the bytecode for the module that is being updated. Maybe some pointers in dictionaries must be updated too - i'm not familiar with the internas here. Since read and write address are on different memory banks, there is no need to buffer the content in RAM.

This way the risk of a corrupted state should be minimized since you would enable the target memory bank only if the copying/patching is successful.

The obvious downside is that you'll need a spare flash bank. On the F4 I believe that's in the 64k range.

@dhylands

This comment has been minimized.

Copy link
Contributor

commented Dec 31, 2016

On the F4 most of the the pages are 128K (except for the 16K/64K ones used by the filesystem).

The way that NOR flash works is that when you erase a page, the entire page becomes 0xff's. You can then change bits to zero by writing the flash. Once a bit has been changed to a zero, the only way to get back to a 1 is to erase the entire page again.

This means that you can erase the page to all 0xff's, and then keep appending modules until you fill the page. Each byte-code segment should have a header. The header should probably be 4 bytes, of which 3 bytes could be a length and one byte is an in-use marker. If the in-use marker is 0x00, then this module should be ignored. If the in-use marker is something else (say 0xEE) then it means that a valid entry is present. There should also be some type of CRC or checksum.

You'd write a new segment by ensuring the region you're going to write to is all 0xFF's. Then write the 0xEE/length, data, and CRC/checksum.

If you want to "delete" a module, you change the 0xEE to 0x00.

You could then use all of the flash from the end of the firmware to the end of flash, by appending modules, until you fill it up, and then you have to start over by erasing the flash and rewriting the firmware. If you had an SDcard (or a spare flash block), you could compact out the not-in-use modules.

Adding in failsafes (to ensure that everything is always valid even if the power is removed halfway during a flash write requires at least a spare flash block and probably some other compromises as well (like not allowing a bytecode segment to cross a 128K boundary).

@goatchurchprime

This comment has been minimized.

Copy link

commented Apr 25, 2017

Here's my 2c, after I saw @ShrimpingIt fighting over this issue for the last week, which resulted in building the entire esp8266 toolchain (in addition to the cross-compiler) to get frozen modules, which he things (though I don't know) consume less RAM than mpy files. Myself, I'm having to use just the cross-compiler, which is bad enough.

These devices we are using are very RAM constrained, but they've got way too much flash memory (it takes days to fill it up even with logging data sensors), so we're fine with solutions that waste it. A whole separate 0.5Mb program that compiled to mpy modules and embedded each one into its own individual page of flash within the main interpreter, so that the reset button alternately ran this "install system" or the real system, which meant you had to press it twice whenever you uploaded new code -- would be preferable than having no choice than to install the toolchain on a PC and insert an awkward command line operation between editing and uploading your python text files.

It would be a whole lot less disruptive for people who had hit the RAM limit and had gotten used to the convenience and debuggability of just running scripting languages on a microcontroller. (An absolute size limit on python modules due to the limitations of this on-board compiler is less of an issue, because you can usually work round it by breaking these files down.)

Maybe later we can think about merging such a development tool seamlessly into the main interpreter so no one even noticed it.

@aykevl

This comment has been minimized.

Copy link
Contributor

commented Oct 18, 2017

Here is a small test implementation for the ESP8266, a proof of concept if you will:
master...aykevl:mpy-to-flash

The idea is to use the flash space between the end of the .text section (.irom0.text) and the start of the FAT32 filesystem to store imported .mpy files.

TODO + ideas:

  • enable via a #define
  • limit it's size to the available size
  • do not overwrite on every restart
  • 'overflow' to RAM if flash is full
  • qstrs to flash?
  • only store specific modules in flash, not all .mpy files?
  • store regular .py files in flash?
  • ...
@aykevl

This comment has been minimized.

Copy link
Contributor

commented Oct 18, 2017

I've improved the PoC somewhat. It has a bit more safety built in and doesn't overwrite flash on every reset - as long as the order of import is kept the same.

Here are two ideas for an API:

Proposal 1

Add a new API (e.g. micropython.store_modules(*modules)) that does not actually import a module, but if it hasn't been imported yet loads it into flash and adds it to the internal list of loaded modules (the same list that's used to make sure a module is only imported once).
This call can then be added to the beginning of e.g. main.py or boot.py:

import micropython
micropython.store_modules('my_huge_module', 'foo', 'bar')
import my_huge_module
import foo
# bar is imported within foo

Trouble with this approach is that dependencies between modules are a bit less obvious. "importing" means 3 steps: compile/load the module, execute the module (which possibly imports other modules), and adding it to the parent namespace. With this API it's less clear when the execution step happens.

Proposal 2

Always load .mpy files into flash and load .py files into RAM during import. Then add a py_compile module to generate these .mpy files on the device itself, as @timpur proposed.
I think this is the easiest / least complex system, and is also compatible with CPython, which is nice.
The only downside is that there is now a separate compile step after every update to .py files, which might be non-obvious. And as there is usually no RTC we can't compare timestamps so will need to always choose the .py file or the .mpy file - probably the latter.

Result

The main use case is saving RAM. Currently the ESP8266 (in my build) has 44kB left between the end of the ROM and the beginning of flash storage, which can be used for this purpose.
What it doesn't cover is 'streaming' a .py or .mpy file over the REPL and storing it in flash, though in theory this could be added with some more work - it currently assumes the file is stored on the filesystem.

I'm in favor of proposal 2, for it's simplicity (skips questions of import order) and that it has a standard API. But I'm curious what other people think about this, or other ways to do it.

@adritium

This comment has been minimized.

Copy link

commented Nov 14, 2017

As a first step, why couldn't you save the structure returned by mp_obj_t mp_compile(mp_parse_tree_t *parse_tree, qstr source_file, uint emit_opt, bool is_repl) onto flash and then load it into RAM when you want to execute it?

@aykevl @goatchurchprime @dhylands @dpgeorge @timpur

@timpur

This comment has been minimized.

Copy link
Author

commented Nov 15, 2017

Beyond me, but sounds valid.

@adritium

This comment has been minimized.

Copy link

commented Nov 15, 2017

This can currently be done in two ways, one being (as you're aware) to incorporate it into a firmware build as frozen bytecode. The second way is to cross compile it to a .mpy file, transfer that to the filesystem of the target, and execute it from there. @peterhinch

Aren't those the same thing? To get frozen bytecode, you need to use the cross-compiler and then include the resulting _frozen_mpy.c into the build.

Or do you mean build as frozen .py?

@peterhinch

This comment has been minimized.

Copy link
Contributor

commented Nov 16, 2017

While both use the cross compiler to generate bytecode the runtime behaviour is different. A .mpy file is loaded from the flash filesystem into RAM for execution. By contrast the MicroPython VM can interpret frozen bytecode directly from flash; this can save substantial amounts of RAM.

Cross compiling to .mpy is good for the case where a large module fails to compile on the target hardware because the compiler runs out of RAM. Frozen bytecode comes into its own with bigger applications where the compiled bytecode itself runs out of RAM.

@adritium

This comment has been minimized.

Copy link

commented Nov 16, 2017

@peterhinch if you're going to cross-compile to .mpy, why wouldn't you want to then freeze it?
Is executing from RAM that big an advantage?

@aykevl

This comment has been minimized.

Copy link
Contributor

commented Nov 16, 2017

Is executing from RAM that big an advantage?

It's not. The advantage is that .mpy files can be loaded on a filesystem without modifying the firmware (MicroPython). Frozen bytecode requires recompiling the firmware, which may be inconvenient or impossible.

It is technically possible to load the contents of these .mpy files into flash for execution, but that's not the current system (see my branch above which implements it). It is technically very hard or impossible to execute .mpy files directly from the filesystem where they are stored due to fragmentation and limitations on some architectures (e.g. ESP8266).

@adritium

This comment has been minimized.

Copy link

commented Nov 16, 2017

@aykevl @peterhinch but there is another configuration to consider. If you have no filesystem, like my Cortex M4, it's still possible in theory to load in an .mpy without flashing the entire firmware.

If the FROZEN_MPY scheme is modified a little, you could write the new .mpy to some area in flash and then update the arrays that keep track of the location and size of the frozen .mpy files.

@aykevl

This comment has been minimized.

Copy link
Contributor

commented Nov 16, 2017

Some more questions for this scheme:

  • how are you going to transfer the Python source code to the device, without filesystem?
  • what about removing a module, should it be possible to reclaim space?
  • what kind of API are you thinking of? In other words, how do you store a module in flash?
@adritium

This comment has been minimized.

Copy link

commented Nov 16, 2017

how are you going to transfer the Python source code to the device, without filesystem?

UART/WiFi/Bluetooth/USB i.e. whatever method of external communication the module has

what about removing a module, should it be possible to reclaim space?

Absolutely. Also, even without a filesystem, there would still be a flash manager/driver that would be responsible for things like wear-leveling. If micropython is not the only thing running on the micro - as is the case with my application - this means that the micropython C-code wouldn't have direct access to the flash but might have to call the flash driver's I need X continguous bytes of flash (and of course you'd get some multiple of the flash page size) or I don't need **this** chunk of flash that you gave me awhile back.

what kind of API are you thinking of? In other words, how do you store a module in flash?

import dl_new_module from extern_comm

downloader = dl_new_module(source=wifi)
new_module = downloader.start(timeout=1000, all_at_once=true)
# optionally, make sure it's a valid .mpy
new_module.validate()
new_module.store()

At the python level, you shouldn't care where it's being stored as long as it's in a "permanent storage" drive.

It's in the implementation of dl_new_module that things would get fun.
I really do believe you want as much abstraction as possible so you can deal with the presence (or not) of a flash manager/driver. The core of dl_new_module should only be tasked with knowing how many bytes do I need to store this new module and then you would be in charge of how do I translate that into a request to my flash manager/driver.

If this is done right, I think it should be able to be integrated into the FROZEN_MPY code.

The _frozen_mpy.c that's generated to provide the frozen mpy would have to be organized in such a way to allow more modules to be added later on . . . but even then you couldn't get the full flexibility of a filesystem. You'd have to choose at compile-time the size of key arrays.

This would have to be a fixed size and the unused entries populated by "\0"

const char mp_frozen_mpy_names[] = {
"frozentest.py\0"
"\0"};

Same with this array

const mp_raw_code_t *const mp_frozen_mpy_content[] = {
    &raw_code_frozentest__lt_module_gt_,
};

Regarding QSTRs that are introduced by this new .mpy, we'd have to add those dynamically like

https://github.com/micropython/micropython/blob/dynamic-native-modules/extmod/modx/modx.c

#include "py/mpextern.h"

STATIC mp_obj_t modx_add1(const mp_ext_table_t *et, mp_obj_t x) {
    return et->mp_binary_op(MP_BINARY_OP_ADD, x, MP_OBJ_NEW_SMALL_INT(1));
}

MP_EXT_HEADER

MP_EXT_INIT
void init(const mp_ext_table_t *et) {
    mp_obj_t f_add1 = et->mp_obj_new_fun_extern(false, 1, 1, modx_add1);
    mp_obj_t list[6] = {MP_OBJ_NEW_SMALL_INT(1), MP_OBJ_NEW_SMALL_INT(2), MP_OBJ_NEW_SMALL_INT(3), et->mp_const_true_, MP_OBJ_NEW_QSTR(et->qstr_from_str("modx")), f_add1};
    et->mp_store_global(et->qstr_from_str("data"), et->mp_obj_new_list(6, list));
    et->mp_store_global(et->qstr_from_str("add1"), f_add1);
}
@adritium

This comment has been minimized.

Copy link

commented Nov 16, 2017

@aykevl well I feel stupid: this functionality is already there, isn't it @dpgeorge ?

persistentcode.c appears to contain implement what you'd need.

@aykevl

This comment has been minimized.

Copy link
Contributor

commented Nov 17, 2017

Can you describe the use case you're thinking of?
I see many problems, the most important one being that you haven't described how you want to update mp_frozen_mpy_content and similar. They're stored in flash, thus in read-only memory.

@dpgeorge

This comment has been minimized.

Copy link
Member

commented Nov 17, 2017

this functionality is already there, isn't it @dpgeorge ?

@adritium the discussion here can get confusing because there are many different ways to load a script, and there could be many more depending on the tradeoffs/features required. For example on an MCU without a filesystem but with a way to append contiguous chunks of data to the flash, you could save .mpy files to the flash and then load them from flash into RAM for execution (which is exactly what the loading code in persistencode.c does). That's quite easy to implement, it just requires to add a small bit of glue code to expose the append-only flash region as a pseudo, read-only filesystem such that persistentcode.c can read from it.

But this approach doesn't allow to execute bytecode from flash. To do that you'd need to take the output from persistentcode.c and save it to flash (instead of putting it in RAM). That's really what this issue here is about.

@adritium

This comment has been minimized.

Copy link

commented Nov 21, 2017

Load/save raw code loads/saves the type mp_raw_code_t.

Is it relevant whether it's the same thing as an .mpy file?

mp_obj_t  mp_compile(mp_parse_tree_t *parse_tree, qstr source_file, uint emit_opt, bool is_repl) {
    mp_raw_code_t *rc = mp_compile_to_raw_code(parse_tree, source_file, emit_opt, is_repl);
    // return function that executes the outer module
    return mp_make_function_from_raw_code(rc, MP_OBJ_NULL, MP_OBJ_NULL);
}
mp_raw_code_t *mp_raw_code_load(mp_reader_t *reader) {
    byte header[4];
    read_bytes(reader, header, sizeof(header));
    if (header[0] != 'M'
        || header[1] != MPY_VERSION
        || header[2] != MPY_FEATURE_FLAGS
        || header[3] > mp_small_int_bits()) {
        mp_raise_ValueError("incompatible .mpy file");
    }
    mp_raw_code_t *rc = load_raw_code(reader);
    reader->close(reader->data);
    return rc;
}


void mp_raw_code_save(mp_raw_code_t *rc, mp_print_t *print) {
    // header contains:
    //  byte  'M'
    //  byte  version
    //  byte  feature flags
    //  byte  number of bits in a small int
    byte header[4] = {'M', MPY_VERSION, MPY_FEATURE_FLAGS_DYNAMIC,
        #if MICROPY_DYNAMIC_COMPILER
        mp_dynamic_compiler.small_int_bits,
        #else
        mp_small_int_bits(),
        #endif
    };
    mp_print_bytes(print, header, sizeof(header));

    save_raw_code(print, rc);
}

@aykevl

This comment has been minimized.

Copy link
Contributor

commented Nov 21, 2017

Yes, this is the code to load an .mpy file.

@adritium

This comment has been minimized.

Copy link

commented Nov 21, 2017

So then I don't think the issue of qstrs that you brought up is a problem: since the structure mp_raw_code_t is an .mpy and .mpys

... have a special encoding for qstrs

then mp_raw_code_t also has the special encoding for qstrs

@adritium

This comment has been minimized.

Copy link

commented Nov 30, 2017

@timpur @dpgeorge @aykevl whoever can change the title, I'd recommend changing the title to

"Compile Python Scripts On the Go and run from flash"

You can already compile .py on the go.

@aykevl

This comment has been minimized.

Copy link
Contributor

commented Nov 30, 2017

You can already compile .py on the go.

No you can't. .py files are compiled before execution, but as the first post clearly states, it's about saving this to a .mpy file or similar.

@adritium

This comment has been minimized.

Copy link

commented Nov 30, 2017

Oh you want to compile them from the VM?

"And run from flash" should still be added.

@adritium

This comment has been minimized.

Copy link

commented Dec 1, 2017

@dpgeorge

@adritium note that the "raw code" will contain pointers to RAM which need to be relocated to point to flash, and it will also contain qstr values which may not be valid the next time the system starts (assuming you want to reuse the saved code after a reboot).

If I create my own load_raw_code_from_flash based on load_raw_code, then as long as I (1) set the RAM pointers to point to flash and (2) call the appropriate load*qstr* function . . . I should be able to execute from flash, correct?

@adritium

This comment has been minimized.

Copy link

commented Dec 8, 2017

So it turns out the C-API already exists to compile a .py that exists in flash (or RAM which would be the case if you're downloading. The reason I had so much trouble realizing it is because of the names of the functions.

See #3466 for complete thought process (or this comment for the punchline)

char * name; // name of my .py file
uint32_t name_len; 
char * py_file_content; // .py file text contents
uint32_t py_file_content_len; // size in bytes of .py file


// mp_lexer_t *mp_lexer_frozen_str(const char *str, size_t len)
qstr source = qstr_from_strn(name, name_len);
mp_lexer_t *lex = MICROPY_MODULE_FROZEN_LEXER(source, py_file_content, len, 0);

// int parse_compile_execute(const void *source, mp_parse_input_kind_t input_kind, int exec_flags)
mp_parse_tree_t parse_tree = mp_parse(lex, MP_PARSE_FILE_INPUT);
module_fun = mp_compile(&parse_tree, lex->source_name, MP_EMIT_OPT_NONE, exec_flags & EXEC_FLAG_IS_REPL);
mp_call_function_0(module_fun);

Someone needs to wrap that sequence in a Python function and one-half of the problem is solved.

@adritium

This comment has been minimized.

Copy link

commented Dec 8, 2017

The other half of your request is

The compiled script don't need to be loaded into ram and run from flash, just like the firmware and core modules.

There is a function void mp_raw_code_save(mp_raw_code_t *rc, mp_print_t *print). This function doesn't know how to write to any filesystem; it only knows how to use the interface mp_print_t to do the writing. So, YOU have to provide an mp_print_t implementation. In the case of writing to flash, print would know how to accumulate a page's worth of bytes and then write that page. If you had a filesystem, you'd use mp_raw_code_save_file.

The question is: now that you've successfully saved the mp_raw_code_t to flash, how do you "load" it and run from flash? That's the bad news.

mp_raw_code_t *mp_raw_code_load(mp_reader_t *reader) does NOT do what you want: it loads everything in RAM before executing but all is not lost.

With just a few tweaks of mp_raw_code_load, you can make your mp_raw_code_load_run_from_flash. Here's a guideline:

  1. all calls to m_new must be eliminated because they allocate RAM. Instead, the pointers that are assigned the memory from m_new must be pointed to the location in flash where the data in question is located (instead of copying from flash into RAM).

For example:

STATIC mp_raw_code_t *load_raw_code(mp_reader_t *reader) {
    // load bytecode
    size_t bc_len = read_uint(reader);
    byte *bytecode = m_new(byte, bc_len);
    read_bytes(reader, bytecode, bc_len);
    ...

would be replaced with

STATIC mp_raw_code_t *load_raw_code(mp_reader_t *reader) {
    // load bytecode
    size_t bc_len = read_uint(reader);
    byte *bytecode = get_ptr_and_increment(len);
    ...
  1. you need to keep the code that reads qstrs and "pushes" them into the Python context.
@aykevl aykevl referenced this issue Dec 8, 2017
@adritium

This comment has been minimized.

Copy link

commented Dec 8, 2017

@timpur

Following this module you would call something like this:
py_compile.compile(sourceFile, outputFile)
where the prams are string locations to files on the filing system eg '/folder/file.py'

Some micros do not have a filesystem - only flash - so /folder/filepy.py would be meaningless.
For that case, we'd need to implement some kind of "virtual file system" to serve as the interface to py_compile.compile(sourceFilePath, outputFilePath) and/or simply allow py_compile.compile the option to take in a Python array containing the actual .py file in text form.

@aykevl

This comment has been minimized.

Copy link
Contributor

commented Dec 9, 2017

@adritium I would suggest that you actually try to write what you propose to see if it's working.

@slavaza

This comment has been minimized.

Copy link

commented Mar 23, 2018

I'm not a compilers specialist, but let me just say one thing. I looked through the source and machine generated texts of the project and I want to suggest a way to runtime freeze modules with minor changes in the project. It can be simple if you to do a simultaneous reinstallation the pool of all frozen modules. In this case linking structures in the Flash will be minimal. Incremental building the pool of freeze modules is more complex and require significant modification of the linking logic.

@adritium

This comment has been minimized.

Copy link

commented Mar 24, 2018

There’s already a way: persistentcode.c

@slavaza

This comment has been minimized.

Copy link

commented Mar 24, 2018

If I correctly understood the file "persistent.с" contains only the functions of serializing the runtime structures of bytecode. I think problematically modify it to place the module code directly in the Flash, without pointers recalculation.

@adritium

This comment has been minimized.

Copy link

commented Mar 25, 2018

@timpur @dpgeorge please append “and run from flash” to the title.

@adritium

This comment has been minimized.

Copy link

commented Mar 25, 2018

@slavaza There is already a workflow to freeze .py files so that they can be executed from flash.

Look in the makefiles for a recipe that contains mpy-tool; the output is .mpy which is the same structure as the typemp_raw_code_t.

Next step is to turn .mpy into a c-file; this is done by a .py utility (I forget the name but it’s in the makefile).

A straightforward strategy is to replicate the code that produces this c-file.
The trick here is the linking of all the tables in the frozen-c file to the rest of the code.
What the linker would normally do in terms of address resolution, you’ll have to do.

Create a simple .py file - like “a=a+1” - and go through the workflow that produces the frozen-c file.
You’ll see all the tables that are produced and how they’re linked to the rest of the code.

@slavaza

This comment has been minimized.

Copy link

commented Mar 25, 2018

@adritium Thank you very match. I know this. I want have possibility to remote update the scripts in the controller. I like Python syntax and I want use it in my future project. But it need some improvement of the original project.

@tomlogic

This comment has been minimized.

Copy link
Contributor

commented Jun 17, 2018

I have some code running on an ARM platform that can relocate an mp_raw_code_t object to flash, including all references. I have modified the QSTR code to support a second QSTR pool in flash after the static pool embedded in the firmware image.

My current process runs outside of the REPL and runs the following steps:

  • perform a soft reboot
  • creates a QSTR pool in RAM (outside of heap) to match the size allocated in flash
  • compile a block of Python code with mp_compile_to_raw_code()
  • relocate compiled code from heap to addressable flash
  • write QSTR pool to addressable flash
  • perform another soft reboot

At this point, I can run that code by passing the entry point to mp_make_function_from_raw_code() and calling the generated function with mp_call_function_0().

I'm about to modify that code to take a list of .mpy files and embed them in addressable flash (along with their QSTRs) in such a way that mimics the use of mpy-tool to convert a .mpy file to C code for embedding in the firmware (aka a frozen mpy).

And just to summarize the reasons for doing this: on a low-RAM device, it's useful to have .mpy files in addressable flash such that an import of the module requires a limited amount of heap. By implementing it in this fashion, you no longer have to embed the .mpy file in the firmware image itself -- you can manage it on the device.

I've tried to write the current code in a way that splits the HAL layer for flash erase/write from the "bundling" layer that walks the mp_raw_code_t structure and relocates it to flash. I'm doing this work on a closed platform, but can share the generalized relocation code upstream.

Since I implemented os.compile() to convert .py to .mpy on the device (useful when breaking a program up into multiple modules that you can compile separately with a given heap size), I've planned to prototype this as os.bundle() since that's the term I came up with when I originally wrote the code over a year ago. Maybe I should call it micropython.freeze() instead?

@slavaza

This comment has been minimized.

Copy link

commented Jun 18, 2018

@tomlogic I've study a little about the operations performed by the mp_make_function_from_raw_code () function. I think the creation of a version of this function that can place the runtime structures directly into flash will be the best solution. This will be quite sufficient for a remote upgrade. However, this is not easy.

@janpom

This comment has been minimized.

Copy link

commented Jun 5, 2019

Has there been any progress on this? This would be helpful for over the air updates. I have a project where I allow multiple versions of the code to be installed and the user can switch between them. This is done by keeping each version of the code in a separate directory. In main.py, the directory of the desired code version is added to sys.path and the entry point function is executed. The problem is that the code resides in the filesystem and even if it's pre-compiled to .mpy, the modules still get loaded to RAM, which then quickly gets exhausted. Being able to execute the modules directly from flash would be very beneficial in this case.

@tomlogic

This comment has been minimized.

Copy link
Contributor

commented Jun 6, 2019

@janpom, I have working code, but I'm afraid I haven't generalized it enough to be of use elsewhere. If someone is genuinely interested in exploring it, they should contact me about getting a copy. Currently, I have a MicroPython method that manages an area of addressable flash on the host processor, and can erase that area and then write a QSTR table and execute-in-place versions of the code from multiple .mpy files. We use that as a way to get more executable code on the device without having it fill the heap.

I'll see if I can create a branch that includes my code, along with instructions on what's missing to get it working on a given platform.

@slavaza

This comment has been minimized.

Copy link

commented Jun 6, 2019

Linking frozen Python modules by the C language linker using direct addressing is a beautiful idea, but it making impossible to runtime decomposition them. Perhaps we should not break this effective conciseness.

@tomlogic

This comment has been minimized.

Copy link
Contributor

commented Jun 7, 2019

@slavaza I don't think anyone is talking about removing that feature (frozen python modules at compile time) but looking at exploring ways to add Python modules at run time. In my use case, MicroPython runs as a separate task of a closed-source firmware image. We want customers to have the ability to embed their Python modules in a way that reduces RAM usage. Relocating them to addressable flash, along with their QSTRs is a working solution.

@slavaza

This comment has been minimized.

Copy link

commented Jun 8, 2019

@tomlogic Yes, I was think about it too. On my opinion, one of the best and simple solution is modification the linker of micropython what make possible allocate loadable module directly into flash.

@pmp-p

This comment has been minimized.

Copy link
Contributor

commented Jun 18, 2019

@tomlogic how we should contact you ? is this code free, swapping seems good to fix #4187 too.

@tomlogic

This comment has been minimized.

Copy link
Contributor

commented Jun 18, 2019

@pmp-p I pasted some code in to #4187 for generating .mpy from .py on the device.

I'm afraid it will take some time/effort to untangle my routines for relocating compiled code to addressable flash, and they depend on some special HAL functions to erase/write the flash.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.