Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How about OTA updater? :) #55

Open
h-david-a opened this issue Apr 21, 2024 · 11 comments
Open

How about OTA updater? :) #55

h-david-a opened this issue Apr 21, 2024 · 11 comments
Labels
enhancement New feature or request

Comments

@h-david-a
Copy link
Contributor

There are new features almost every day!
I'd like to get them directly to the device :)
I've tried to adapt micropython_ota: https://github.com/mattjackets/micropython_ota/blob/main/micropython_ota.py
But it fails to download 'big' files: in my case it was failing on memory alocation for 44kb
Googled a bit, and it appears that there's a mrequests lib providing chunked downloads (https://github.com/SpotlightKid/mrequests/tree/master)

Looking into it. Any inputs are welcome!

@echo-lalia
Copy link
Owner

I like the idea, but I'm not sure I can envision how this should ideally work for MicroHydra.

Most people are running the compiled firmware, I think. That means that there would need to be a totally separate solution for OTA updates on the firmware, vs running it on normal MicroPython firmware.

Maybe if there was a built-in app that made it easy to download files from GitHub, that could be a good solution? That way, people could fetch new apps from the app repo without needing a computer, and people who want the newest MH updates could download from this repo.

I'm not sure! I'd love to hear your input on this :)

@h-david-a
Copy link
Contributor Author

h-david-a commented Apr 22, 2024

So, my perfect setup looks like this:

  • I've got M5 stick launcher to select different firmwares if needed (https://github.com/bmorcelli/M5Stick-Launcher/tree/main?tab=readme-ov-file)
  • Firmwares stored on SD card
  • Currently running micropython clean rom.
  • MicroHydra installed from sources (via Thonny) to tinker around.
    Fun fact, that reflashing (e.g to Bruce or demo roms) doesn't affect MH files (Would be nice if someone explained that to me :))

As for OTA, I see three use cases:

  1. For tinkering with a non-compiled version of MH - obviously github downloader is needed.
    I've got some trials in my repo:

  2. For updating 'extra' apps of the compiled version - github downloader with store-like functionality is needed (see previous point)

  3. For an OTA update of the compiled ROM - some collaboration with M5Stick-Launcher repo needed.

Sidenote for perfectionists: to do OTA sustainably, proper release process is needed for MicroHydra repo (but that's another story once OTA will work at least somehow)

@echo-lalia
Copy link
Owner

That's a good point; Now that M5Launcher supports MicroHydra, (and that they're adding OTA to it), it basically handles all of the 'OTA' functionality for the compiled version for us.

Fun fact, that reflashing (e.g to Bruce or demo roms) doesn't affect MH files (Would be nice if someone explained that to me :))

I think this happens because MicroPythons virtual filesystem is just not being overwritten by the other firmwares.
Afaik, the normal MicroPython firmware is only taking up a ~2mb partition on it's own, and then the rest of the flash becomes a filesystem. Then, if you overwrite with another firmware, as long as the firmware isn't huge in size, it'll probably only overwrite the MicroPython partition, and not overwrite the filesystem.

You can actually encounter some issues if you, just as an example, install UIFlow, and then install MicroHydra (without erasing the flash first) because the old UIFlow filesystem will be there instead of MH's filesystem.

@echo-lalia
Copy link
Owner

adjusted 'original' ota script (it provides app store-like functionality, BUT fails on big files of MicroHydra itself):

Would you be willing to share what line of code in your first attempt is running out of memory? I haven't had the chance to try it out myself, but I have encountered a lot of memory issues throughout this project, so I'm curious to know if this will be a familiar issue for me :)

@h-david-a
Copy link
Contributor Author

That's a good point; Now that M5Launcher supports MicroHydra, (and that they're adding OTA to it), it basically handles all of the 'OTA' functionality for the compiled version for us.

Are they adding OTA?.. Where?

I think this happens because MicroPythons virtual filesystem is just not being overwritten by the other firmwares.

Btw, do you know by chance if there's an 'empty' firmware available? Just to make a clean install? (or that should be a feature request to M5 launcher?)

Would you be willing to share what line of code in your first attempt is running out of memory?

Sure!
This is the line (+-1)
https://github.com/h-david-a/Cardputer-MicroHydra/blob/67de65a3552fd6d01d7c3c7a6865b7f9deff9506/MicroHydra/apps/ota.py#L105

@echo-lalia
Copy link
Owner

Pirata, the author of M5launcher, has been sharing updates on the unofficial Cardputer Discord about progress on a launcher that can install from SD and install from the M5 burner repo
SmartSelect_20240422_155019_Discord.jpg

Really exciting stuff! They're making progress quickly 😁

And as for erasing the old filesystem... Usually that is something done by the flashing tool you're using, so it makes sense to me that M5 launcher would get that functionality.

@echo-lalia
Copy link
Owner

Sure! This is the line (+-1) https://github.com/h-david-a/Cardputer-MicroHydra/blob/67de65a3552fd6d01d7c3c7a6865b7f9deff9506/MicroHydra/apps/ota.py#L105

Interesting. So, it looks like it's not failing to fetch the content from the internet, but instead failing to read the content/create the file, right?

If that's the case, you might be able to solve it by doing something similar to what the Files app does. Here's a snippet from the 'paste' function in the files app:

with open(source,"rb") as old_file:
    with open(dest, "wb") as new_file:
        while True:
            l = old_file.read(512)
            if not l: break
            new_file.write(l)

It's not actually reading/writing the file all at once, but scanning through in small chunks, in order to not use too much memory.

I still haven't actually gotten the chance to run your code myself, so I could be totally off with this. But I think it might just be failing because it's trying to fully open/decode the entire file all at once, before writing it to a new file.

@h-david-a
Copy link
Contributor Author

h-david-a commented Apr 24, 2024 via email

@h-david-a
Copy link
Contributor Author

h-david-a commented Apr 24, 2024

So, the issue is with the urequests module.
And some nice person already patched that 6 years ago:
https://github.com/chrisb2/micropython-lib/blob/master/urequests/example_iter.py :))

@h-david-a
Copy link
Contributor Author

After messing around got to the No free space on device. Wiping firmware doesn't really help.
I was checking available space with shutil (print(shutil.disk_usage('/'))), got total space 1048576 bytes.
And current microhydra is +-450KB. OTA downloads +-450KB more. And there are some logs and other leftovers which causes "system crash".

Also, s3 stamp specs mention 8MB flash (http://docs.m5stack.com/en/core/StampS3)

Checking partitions with esp32 method shows also strange results:

print(esp32.Partition.find())

[<Partition type=0, subtype=32, address=65536, size=851968, label=app0, encrypted=0>,
<Partition type=0, subtype=16, address=983040, size=5242880, label=app1, encrypted=0>]

So, app0 is 832KB and app1 - 5MB. And that doesn't match any schema described by m5 launcher
https://github.com/bmorcelli/M5Stick-Launcher/wiki/Explaining-the-project#custom-partitioning

OTA appears to be a more complicated think :)

@echo-lalia echo-lalia added the enhancement New feature or request label Aug 21, 2024
@echo-lalia
Copy link
Owner

FYI, newest version of MicroHydra now has a built-in way to download apps from the github repo. This isn't an OTA updater at all because it can't download/overwrite MicroHydra's files, but I could imagine a future OTA update functionality reusing some of the same logic 😁

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants