ports/esp32 add api support for ota updates #3576

andynd · 2018-01-21T01:28:33Z

add the following APIs:
esp.partition_find_first
esp.ota_set_boot_partition
esp.ota_get_boot_partition
esp.ota_get_running_partition
esp.ota_get_next_update_partition

move the vfs to its own partition if it exists
use an ota capable partition layout

add the following APIs: esp.partition_find_first esp.ota_set_boot_partition esp.ota_get_boot_partition move the vfs to its own partition if it exists use an ota capable partition layout

andynd · 2018-02-27T17:01:37Z

Any plans to merge this? Pull request was created more than a month ago.

ThomasWaldmann · 2018-03-19T00:57:41Z

Having OTA capability would be nice!

Do I understand it correctly that it would update the partition with micropython, but not the one with the filesystem?

ThomasWaldmann · 2018-03-19T00:59:06Z

ports/esp32/modules/_boot.py

@@ -5,6 +5,7 @@
 try:
    if bdev:
        uos.mount(bdev, '/')
+	print("Mounted /")


indentation looks wrong. tab? one space missing?

oh.. yes. thx Will fix it.

ThomasWaldmann · 2018-03-19T00:59:37Z

ports/esp32/modules/flashbdev.py

+	if not offset:
+		self.START_SEC = esp.flash_user_start() // self.SEC_SIZE
+	else:
+		self.START_SEC = offset // self.SEC_SIZE


again: indentation looks wrong.

andynd · 2018-03-19T01:06:45Z

Do I understand it correctly that it would update the partition with micropython, but not the one with the filesystem?

It moves the fs to its own partition, right now micropython ignores the partition table and uses space after micropython. This would corrupt a 2nd partition. You can update the fs too, but you do not need any code in mp to do it, it can be done in pure python code so this pull doesn't contain any code for it

Is the flash big enough for this? How would a typical layout with mp and fs look like?

Depends on your esp32 module. Most I work with have 4mb flash, enought for two mp partitions with 1.7mb each and a 512kb fs partition. My folder struktur is two folders at / one named "factory" and one named "ota_0". My boot.py then moves in the folder named after the boot partition and executes the boot.py and main.py there. Like that i can update the fs too, using a downloaded tar.

ThomasWaldmann · 2018-03-19T01:09:56Z

Thanks for explaining. C&P that to some docs file?

…nto ota

andynd · 2018-03-25T00:15:21Z

Updated to current master, fixed indentation. I will try to get our internal ota updater written in python added to the docu, just have to get an ok for it first. What more is needed to merge this?

app_update was included multiple times

andynd · 2018-08-11T09:57:08Z

I did not yet have time to publish our python side for this, but are there any plans to merge this?

Lisa999 · 2018-08-11T22:31:30Z

@andynd: can you fix the makefile error? It's the ULP which has been added to the makefile.
Some examples of use would be nice, like:
esp.partition_find_first(1,129,None)
esp.ota_get_boot_partition()

andynd · 2018-08-12T12:29:13Z

Merged upstream master. Any idea where to put the examples? I don't want to add yet another readme file like I had to for the ulp.

esp.ota_get_running_partition() esp.ota_get_next_update_partition(start_from=None)

andynd · 2018-08-12T16:18:27Z

I've added a README.ota.md on how to use this APIs. The code (in the README) is extracted from our product source code and some (irrelevant) parts removed. I have not tested it after I removed this parts but the production code is working so it should at most have some minor errors.

Lisa999 · 2018-08-12T20:09:19Z

I've integrated it into my micropython tree, works just fine (after fixing the makefile merge)!
Thx for the good work, this is just what i needed!

Ps, you can't use a 512K fatvfs! The minimum for 4K sectors is 512K, but.... an additional 16K is being used for the WL (wear level). Here's what espressif said:

With 4k sector size, the smallest FAT partition size is 512K (128 sectors), plus 4 sectors used by WL, makes the minimum of 528K.

https://www.esp32.com/viewtopic.php?t=1897

andynd · 2018-08-12T20:41:11Z

Well the micropython implementation doesn't seem to have this limitation. Micropython doesn't use any of the esp-idf filesystem code.

andynd · 2018-08-16T11:25:14Z

Merged upstream master again. What is missing to finally merge this? The original PR is 7 months old.

ThomasWaldmann · 2018-08-16T12:34:44Z

@andynd guess it is the usual "many contributions, few reviewers/maintainers / little time" issue.

HeMan · 2018-08-20T08:00:24Z

ports/esp32/Makefile

@@ -60,6 +60,8 @@ INC += -I$(BUILD)

 INC_ESPCOMP += -I$(ESPCOMP)/bootloader_support/include
 INC_ESPCOMP += -I$(ESPCOMP)/bootloader_support/include_priv
+INC_ESPCOMP += -I$(ESPCOMP)/bootloader_support/include_bootloader
+>>>>>>> upstream/master


This is a rest of a merge conflict, no?

Yes it was, thx for pointing it out. Funny how it still was able to built without errors. Fixed.

andynd · 2018-08-27T15:16:45Z

poke

andynd · 2018-09-05T15:48:57Z

more poke

andynd · 2019-01-29T20:41:15Z

Yes, because this pull request is over a year old. Unless somebody tells me he is going to merge it I will not again spend the time to fix it.

ThomasWaldmann · 2019-01-29T23:58:35Z

@dpgeorge ^^^

Lisa999 · 2019-03-18T22:28:35Z

@dpgeorge: Wake up call...

Lisa999 · 2019-03-18T23:00:05Z

@andynd : i've build my work on your pr, could you please fix the problem? I'll stalk @dpgeorge

Lisa999 · 2019-03-26T21:24:40Z

@dpgeorge: what does it take to get this merged?

Lisa999 · 2019-04-01T12:20:20Z

@dpgeorge ^^^

Lisa999 · 2019-04-13T21:18:14Z

@dpgeorge : as long as this doesn’t getting merged, i won’t create the PR requested in #4436 ...
What’s the point in making PR’s if they are open more then a year?

ThomasWaldmann · 2019-04-14T13:26:59Z

I am watching this since quite a while and I think this project needs more core developers who are trusted enough to review and merge PRs into this project.

I don't have personal interest to become a core dev (I don't like coding in C and also do not use micropython that much, I like micropython, but basically just played with it yet), but I know there are people who would qualify (just look at the PRs and forks).

Also, considering that revision control makes it easy to also revert stuff in case something is found to be problematic, maybe PRs should be accepted more easily. New stuff can be marked experimental for a while, without any guarantees (not even for staying).

Open a ticket about this?

Lisa999 · 2019-04-14T14:47:44Z

@ThomasWaldmann: I suspect that @dpgeorge wants to be in control and therefor doesn’t want to delegate the PR merge rights. Maybe that’s because of the breakup with pfalcon.
I don’t have the rights, so i can’t do it myself. My company is happy to pay to get this, and other PR’s, merged...
There are already three PR’s i need to get merged, so the question is who can do the merge and how much is that going to cost (in donation, contract work or whatever).

spacemanspiff2007 · 2019-04-14T17:36:56Z

@Lisa999: The lack of progress really is super frustrating and it's a shame that pfalcon is not working at this repo any more. Maybe you can team up with pfalcon (or someone else) on his repo or find a fork where there are more activity (maybe LoBoris)?
I'd really like to see the project advancing so it would be nice to have people working together.

dpgeorge · 2019-04-26T07:21:34Z

Sorry all for neglecting this. I'll try to make some progress here.

I think it's a good feature to have, so thanks @andynd for contributing it. But I think the way it's done, the way it is exposed at the Python level, needs to be changed.

The reason is because it looks like the approach here cannot support encrypted partitions (but please correct me if I'm wrong). That's because encrypted read/write must go through esp_partition_read/esp_partition_write (again, correct me if I'm wrong about this). If in the future encryption support was added to uPy we don't want to have to change everything, so it's good to think now how encryption would fit in.

As such my proposal would be to make the API more object-based, rather than pure function calls. In particular put emphasis on the concept of a Partition object. A Partition object would:

contain a pointer (at the C level) to a esp_partition_t struct
contain methods to retrieve the metadata (type, subtype, address, size, etc)
contain methods to read/write/erase the partition

Partition objects would be created by esp_partition_find_first(), esp_ota_get_boot_partition() and esp_ota_get_running_partition(), and be passed to esp_ota_set_boot_partition() and esp_ota_get_next_update_partition().

For example the API could be:

class Partition:
    @staticmethod
    def find_first(type, subtype, label) -> Partition: ...

    def metadata(self) -> (type, subtype, address, size, label, encrypted): ...

    def readinto(self, offset, buf): ....
    def write(self, offset, buf): ....
    def erase(self, offset, size): ....

def ota_get_boot_partition() -> Partition: ...
def ota_get_running_partition() -> Partition: ...
def ota_set_boot_partition(part): ...
def ota_get_next_update_partition(part) -> Partition: ...

By having a Partition object you don't need to worry about passing the correct offsets to esp.write_flash etc. And Partition could in the future support the block protocol allowing it to have filesystems on it and be mounted, etc (so it'd replace the FlashBdev class).

dpgeorge · 2019-04-26T07:22:00Z

ports/esp32/README.ota.md

+import esp
+import sys
+
+def bytecompare(a,b):


Why is this function necessary, isn't it enough to just do a == b? Or do I miss something?

Its not, (a == b) can be = false while bytecompare(a,b) returns true, because its compare by referenz and compare by value

Can you give an example where it's not the same? MicroPython compares by value, eg:

>>> bytearray(b'\x01\x02\x03')==b'\x01\x02\x03' True >>> bytearray(b'\x01\x02\x03')==bytearray(b'\x01\x02\x03') True >>> b'\x01\x02\x03'==bytearray(b'\x01\x02\x03') True

dpgeorge · 2019-07-09T05:17:21Z

See #4910 for a proposed Partition class.

Lisa999 · 2019-07-19T00:37:52Z

@andynd: #4910 partitions support is great, is this PR still needed?

waneksx · 2019-10-17T20:38:30Z

Is it posible to resolve those conflicts?

Lisa999 · 2019-10-17T21:59:24Z

Is it posible to resolve those conflicts?

#4910 is integrated into the master branch, #3576 is obsolete now...

mrwhy-orig · 2020-01-23T14:38:12Z

Where are we on this one? I got a bit lost with all those cross references... Is there documentation on how to make OTA possible?

jedie · 2020-01-23T17:07:48Z

ports/esp32/README.ota.md

+
+debug()
+run(host='0.0.0.0', port=8000)
+```


The Python code style looks bad, isn't it?

This is very small example server, it's fine for that but obviosly nobody should use that in production.

tve · 2020-01-23T17:57:05Z

Why is this PR still open? Is it hoping for someone to turn pieces of the README.ota.md into docs?

mrwhy-orig · 2020-01-23T18:36:39Z

My guess this README.ota.md is outdated. @dpgeorge implemented the Partition Class which serves as a replacement for another Class which is used in this pull request.
I think there is no Updated README.ota.md yet.

ThomasWaldmann · 2020-01-24T22:08:28Z

ports/esp32/README.ota.md

@@ -0,0 +1,405 @@
+# Over the air updates
+
+This is an example of how to do ota-updates using micropython and a very simple bottle server application. It assumes you have some way of notifying your esp of updates and transmit the update hashe securely (for example using ssl with certificate pinning).


typo "hashe"

how about cryptographically signed updates rather? the esp-idf / esp32 support ecdsa signatures and can verify that on update (flashing) / on boot (kind of "secure boot"). they even support encrypted flash images, but there is no requirement to use encryption if you just want to have signatures.

that way, the security of the server having the update files and that of the transmission channel does not matter.
if the signature is ok, you know the file is authentic, complete, correct (as released / signed by the author).
at least as long as you manage to keep your private signing key private.

Agreed. As long as there are no secrets or otherwise sensitive info embedded in the MP firmware being updated it should be fine to verify a hash over the firmware assuming the hash got transmitted securely. A signature is more flexible but much more of a hassle in many use-cases.

this has nothing to do with whether it has secrets inside, but whether you can authenticate the image. you can't authenticate with just a hash from same source.

It's save as long as you have a secure way to transfer this hash. We did not want signed updates because the signature verification was too complex and it was not needed for our usecase. We use certificate pinning and would have gained nothing except protection against a attacker controlled server, but that was not our attack scenario.

ThomasWaldmann · 2020-01-24T22:09:51Z

ports/esp32/README.ota.md

+
+def bytecompare(a,b):
+	if (len(a) != len(b)):
+		return False


tabs > spaces.

ThomasWaldmann · 2020-01-24T22:15:57Z

ports/esp32/README.ota.md

+def trackerprobe():
+	if do_update:
+		res['update'] = {}
+		res['update']['partition'] = {'url': '{}/chunk/part'.format(otahost), 'size': int(os.stat(partfile).st_size), 'hash': hashlib.sha256(open(partfile, 'rb').read()).hexdigest()}


the problem with this is that if somebody hacks your server and puts some tampered update binaries on it (instead the original good ones), the code will compute the hash for the tampered ones and the updater code on the esp32 will happily flash the bad update.

(see above how to avoid this)

That was not the attack vector we cared about. We wanted verified updates without confidentiality, thats why no encryption.

tve

I'm very interested in OTA (or rather, will be when I'm through with my current TODO list :-). But I'm not understanding what this PR is proposing. It's great to have a long readme, but I would prefer to first have a concise statement about how OTA is supposed to work and which use-cases it covers.

For example, it seems this PR intends the entire MP filesystem to be swapped with the OTA, as opposed to separating the filesystem from the MP firmware. Also, it seems this PR intends there to be 3 slots for firmware: factory, ota1, and ota2. Yet only 2 are really required, the factory one is optional (I wouldn't want it, for example).

I have used the Arduino Update class on the esp32 successfully implementing my own update and thought it abstracted the minutiae nicely without adding too many constraints. It would be worth taking a look at what it does: https://github.com/espressif/arduino-esp32/blob/master/libraries/Update/src/Updater.cpp

tve · 2020-01-26T05:44:41Z

ports/esp32/modesp.c

+STATIC mp_obj_t mp_esp_ota_get_boot_partition (void) {
+    const esp_partition_t* part = esp_ota_get_boot_partition();
+    void* partptr = &part;
+    return mp_obj_new_bytes(partptr, sizeof(partptr));


What is this returning in Python-speak? An opaque bunch of bytes?

Yes. The pointer as bytes.

tve · 2020-01-26T05:45:50Z

ports/esp32/modules/_boot.py

@@ -5,6 +5,7 @@
 try:
    if bdev:
        uos.mount(bdev, '/')
+        print("Mounted /")


tve · 2020-01-26T05:47:58Z

ports/esp32/partitions.csv

-nvs,      data, nvs,     0x9000,  0x6000,
-phy_init, data, phy,     0xf000,  0x1000,
-factory,  app,  factory, 0x10000, 0x180000,
+nvs,        data, nvs,     0x9000,  0x4000


Gratuitous whitespace change?
Why the size reduction?

tve · 2020-01-26T05:51:47Z

ports/esp32/README.ota.md

+
+## Folders
+
+We are using the following folder structure:


On the host or on the device?

tve · 2020-01-26T05:52:13Z

ports/esp32/README.ota.md

+└─ ota_1
+```
+
+The folders "factory", "ota_o" and "ota_1" musst have corresponding partitions. "VERSION" contains our currently running software version identifier, for example a git commit hash.


Why 3 partitions, i.e. copies of the firmware? Or am I misunderstanding?

in the product where this code is from, we used "factory" with the factory image that never got updated and was the fallback code for some specific scenarios. You can use only two partitions as well.

andynd · 2020-01-26T22:16:20Z

In general: please understand that this code was written for a specific product with specific tradeoffs and making it open source was just an afterthought. Now that there is a Partition class, you can write some parts different and in a nicer way.

Please understand as well that we wanted secure (as in someone who controlles the network can not tamper with them) and not confidential (an attacker can read firmware images, they are not secret and we do not care if someone gets them). An attacker who attacks our server and steals the SSL certificate or replaces the binary was not our concern. For this scenario it is fine to transfere a hash of the update over a ssl protected connection with certificate pinning and not to do signature verification and to transmit the update chunks unencrypted.

andynd · 2020-02-22T19:16:57Z

#4910 and #5027 implemented all changes I did here in a better way, closing this PR.

add support for ota updates

b19a464

add the following APIs: esp.partition_find_first esp.ota_set_boot_partition esp.ota_get_boot_partition move the vfs to its own partition if it exists use an ota capable partition layout

andynd changed the title ~~add support for ota updates for the esp32 port~~ ports/esp32 add api support for ota updates Feb 4, 2018

ThomasWaldmann reviewed Mar 19, 2018

View reviewed changes

Andreas Valder added 3 commits March 25, 2018 01:04

Merge branch 'master' of https://github.com/micropython/micropython i…

a4eead4

…nto ota

fix usage of mp_obj_new_str, broken by 4601759

28d357f

fix indentation

3a0e370

fix buildsystem errors

8c7bab1

app_update was included multiple times

Merge remote-tracking branch 'upstream/master' into ota

55221a8

Andreas Valder added 3 commits August 12, 2018 14:39

fixed partition table

f2883ec

add more more ota apis

b08cd00

esp.ota_get_running_partition() esp.ota_get_next_update_partition(start_from=None)

add README.ota.md with example code

56198f9

Merge remote-tracking branch 'upstream/master' into ota

9fe47e7

HeMan reviewed Aug 20, 2018

View reviewed changes

removed merge conflict stub

cd38062

Lisa999 mentioned this pull request Apr 13, 2019

ESP32: Non-volatile storage support #4436

Closed

dpgeorge reviewed Apr 26, 2019

View reviewed changes

Lisa999 mentioned this pull request Apr 26, 2019

esp32/modesp32.c: add esp32 non volatile storage support #4707

Closed

MrSurly mentioned this pull request Jul 3, 2019

Secure boot functionality for esp32 #4854

Open

dpgeorge mentioned this pull request Jul 9, 2019

esp32: Add esp32.Partition class to interface to OTA functions. #4910

Closed

nevercast mentioned this pull request Aug 16, 2019

esp32: make use of wear levelling from IDF #4999

Closed

jedie reviewed Jan 23, 2020

View reviewed changes

ThomasWaldmann reviewed Jan 24, 2020

View reviewed changes

tve reviewed Jan 26, 2020

View reviewed changes

andynd closed this Feb 22, 2020

		@@ -0,0 +1,405 @@
		# Over the air updates

		This is an example of how to do ota-updates using micropython and a very simple bottle server application. It assumes you have some way of notifying your esp of updates and transmit the update hashe securely (for example using ssl with certificate pinning).

ports/esp32 add api support for ota updates #3576

ports/esp32 add api support for ota updates #3576

Conversation

andynd commented Jan 21, 2018 • edited Loading

andynd commented Feb 27, 2018

ThomasWaldmann commented Mar 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andynd commented Mar 19, 2018

ThomasWaldmann commented Mar 19, 2018

andynd commented Mar 25, 2018 • edited Loading

andynd commented Aug 11, 2018

Lisa999 commented Aug 11, 2018 • edited Loading

andynd commented Aug 12, 2018

andynd commented Aug 12, 2018 • edited Loading

Lisa999 commented Aug 12, 2018 • edited Loading

andynd commented Aug 12, 2018

andynd commented Aug 16, 2018

ThomasWaldmann commented Aug 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andynd commented Aug 27, 2018

andynd commented Sep 5, 2018

andynd commented Jan 29, 2019

ThomasWaldmann commented Jan 29, 2019

Lisa999 commented Mar 18, 2019

Lisa999 commented Mar 18, 2019 • edited Loading

Lisa999 commented Mar 26, 2019

Lisa999 commented Apr 1, 2019

Lisa999 commented Apr 13, 2019

ThomasWaldmann commented Apr 14, 2019

Lisa999 commented Apr 14, 2019

spacemanspiff2007 commented Apr 14, 2019

dpgeorge commented Apr 26, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dpgeorge commented Jul 9, 2019

Lisa999 commented Jul 19, 2019 • edited Loading

waneksx commented Oct 17, 2019

Lisa999 commented Oct 17, 2019 • edited Loading

mrwhy-orig commented Jan 23, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tve commented Jan 23, 2020

mrwhy-orig commented Jan 23, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tve left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andynd commented Jan 26, 2020

andynd commented Feb 22, 2020

andynd commented Jan 21, 2018 •

edited

Loading

ThomasWaldmann commented Mar 19, 2018 •

edited

Loading

andynd commented Mar 25, 2018 •

edited

Loading

Lisa999 commented Aug 11, 2018 •

edited

Loading

andynd commented Aug 12, 2018 •

edited

Loading

Lisa999 commented Aug 12, 2018 •

edited

Loading

Lisa999 commented Mar 18, 2019 •

edited

Loading

Lisa999 commented Jul 19, 2019 •

edited

Loading

Lisa999 commented Oct 17, 2019 •

edited

Loading