Upload failed (Internal Server Error) when update to version v3.5.0b2

[rickroetenberg]

can't update to version v3.5.0b2, will get Upload failed (Internal Server Error). Tried different browsers without any success..

Current version v3.4.2b5

[Stevehans]

Same here.

[MichaelDvP]

If the ems-esp was origially usb-flashed with a version before 27.02.2021 the partitions are to small for the new update.
Via OTA the partition table can not be changed, a usb-flash is required. Use esptool and erase chip before flashing.
You can save your settings before as json and reload it to AP after new flash.

[proddy]

The partition sizes haven't changed? Not sure why it's not working

[MichaelDvP]

See this commit, It was ems-esp v2 (esp32). But the current v 3.5.0 should also fit in min_spiffs (0x1E0000) size, is there an overhead when flashing?

[proddy]

not sure, but the partition table is exactly the same so not sure why its not working. We could use the old partition table XLS and try that, or look into what has changed in the latest IDF/espressif-arduino core. Maybe we missed something

[MichaelDvP]

I've tested with some GUI-Flashers and most of them do not flash the partition table or a wrong one,
Espressive have a GUI for windows that works.
Here is a zip containing windows-gui with all files needed and the actual python commandline-tool .
Flash_EMS-ESP.zip

In GUI select ESP32 and set these files:

Tested with a node-mcu32 with other partitions before.
After flashing with the GUI disconnect esp32 from power and connect again.

[rickroetenberg]

so we can't upgrade anymore because we don't have a usb flash tool? strange if you ask me

[MichaelDvP]

@rickroetenberg You can upgrade via ota or upload if your esp32 is partitioned correctly. I don't know why you have earlier flashed a wrong partition scheme. I hoped that this tool will help people to fix the issue, which is NOT an EMS-ESP issue. If you dont want to use it, stay with 3.4.2. Not my problem.

[MichaelDvP]

Sorry, it seem i have misunderstood some things. I thought the wrong partition scheme is always done by users (via usb-flash) and could not understand why a new usb-flash is now problematic.
But:

• The partition scheme is from tasmota flasher with 1856k program partition and 320k data. This fits for v3.4.2 (1782k) but not for v3.5.0 (1860k). This flasher was sometimes recommended here. EMS-ESP v3 partition scheme is 1984k program partition, 64k data (v2 was 1920k program, 200k data).
• The Tasmota flasher loads the partitions online from https://github.com/Jason2866/ESP_Flasher and ignores local settings.
• BBQKees has also used this flasher, so some of his deliveries have the smaller partition size.

The Tasmota flasher is much easier to handle than the espressive tool, so i've modified the tasmota flasher to use local bootloader/partitions and call it EMS-ESP-Flasher.
Download compiled version with ems-esp settings: https://github.com/MichaelDvP/EMS-ESP_Flasher/releases/tag/Release

@proddy: Or should the flasher load the partitions.bin online from here(dev)

[Stevehans]

So are you saying that this is a BBQKees issue for some of us in that it's been partitioned wrongly?

[proddy]

So are you saying that this is a BBQKees issue for some of us in that it's been partitioned wrongly?

yes, he's using a flash tool with preset partitions that are too small for v3.5.0

[MichaelDvP]

Let's wait for BBQKees come back.

IMHO: I'm not sure if we can call it an issue. His gateways comes with preflashed, working software. No promise of endless updates.
With software development there were always some breaking changes, v3 does not run on old esp8266. Now 3.5 does not run on the prepartitioned modules. In first case v2->v3 the module has to be changed, now a serial (USB) flash is required. I think most users have a micro-usb-cable.
In not to far future the ota-program memory will be full, for further updates a new module (16M) or a different partition scheme with small factory and large OTA partition is needed, that will have a different update procedure. Be prepared that you'll need experience with serial flashing.

[proddy]

true. BBQKees confirmed this and will be using a new method to flash the firmware on new gateways. For existing users we should provide a script (both Windows, Linux/OSX) for the CLI tools.

[MichaelDvP]

@proddy Should we add a flash info, not for v3.4.2->v3.5, but maybe for 16M chips formated to 4M? (This is formated 16M):

[proddy]

Yes @MichaelDvP, that'll be good to add. With a couple of recommendations

• show Kb everywhere instead of bytes. That's been on my list to do for a while.
• Don't use the word Sketch (it's from the old arduino days) but use Application size

[MichaelDvP]

OK, i'll make the change. Next word for translation.

[proddy]

@MichaelDvP I just tried uploading the firmware from the Github Release assets via the WebUI and experiencing the same issue. I actually think the bug is in the CI build scripts and not related to how the Gateways are flashed. I'm investigating...

[MichaelDvP]

Can not reproduce, loaded the 3.5.0b2 from here and there, for both web upload working on a 4M MH-ET.

[proddy]

Ok.

[proddy]

it was the comment from Thomas that triggered this (https://discord.com/channels/816637840644505620/816652040305901628/1022973202454032414) . Which I still can't explain...

[MichaelDvP]

I don't know what was wrong in this upload. The upload routine give error 500 for all possible errors back. A size check is done on start of upload, if the error 500 is immediatly it is the size. If upload starts and error 500 comes later it is a lost/corrupt ip-package. I had this error a few times while uploading with different sorftware, a second try always works.
Or the download from github had an error, we don't have a MD5 to check.

[proddy]

I tried it a 2nd time and it worked, like you said. Weird. I'll see if I can improve the error messaging on a failed upload in UploadFileService::handleUpload(). I'll need to use the same Flasher tool as BBQKees uses. The MD5 check has been an ask for a while (#240 (comment)). I'll open a new issue so we don't forget.

[MichaelDvP]

Afaik BBQKees used this tool, it loads partitions online, see here. I've forked and load from local directory depending on flash size of target, so it will work on 4M and 16M Gateways.
First step for md5 check is to add md5-info to release, than people can check after download (only very few will do).

[proddy]

The Tasmota flasher is much easier to handle than the espressive tool, so i've modified the tasmota flasher to use local bootloader/partitions and call it EMS-ESP-Flasher. Download compiled version with ems-esp settings: https://github.com/MichaelDvP/EMS-ESP_Flasher/releases/tag/Release

@proddy: Or should the flasher load the partitions.bin online from here(dev)

That's really cool! I say either hard code the partition settings or read them from the dev repo. And then add the flasher to the EMS-ESP project, or put the executables in the script folder so others can use it. We do need to say some kind words to the original owners of the tool for re-using their code.

[MichaelDvP]

I like tools that do not need a online connection. Maybe first look in the local directory and load online if this fails?

This tool is very common used, i think original is the nodemcu-flasher, forked to esphome-flasher and then to tasmota-flasher. I have not checked yet how to build binaries for mac/linux, oly tested for windows. BTW: if used as commandline tool you can specify partition and bootloader, but not in the gui.

[bbqkees]

An EMS-ESP specific upload tool for our users would be a nice idea.
You see that with more projects.

We should add a warning to EMS-ESP that the upload of 3.50 via the web may cause serious problems.
People who have their Gateway at a remote location or are unprepared to do a flash via USB will not be pleased when their device does not boot anymore.
Or we should maybe temporary remove the beta update feature in EMS-ESP.

For production uses I'm going to switch to the Espressif factory flash version, erasing those 16MB flash each time takes too long, so I can do more in parallel.

[MichaelDvP]

Uploading via web is safe. The upload reports error 500 and old software stays. Only usb-flashing with wrong partition table ends up with not booting esp32, but then someone can reflash with right partitions.

[proddy]

yes, then let's make a warning in the Wiki so when this comes up again (and it will) we can just point people to the FAQ

[bbqkees]

Ok great I misread it then somewhere during my holiday.
I'll update the BBQKees Wiki in the coming days to include your flash tool instead of the older one.