Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade ESP32 to V3.1 of esp-idf #1524

Closed
jumjum123 opened this issue Oct 15, 2018 · 29 comments
Labels

Comments

@jumjum123
Copy link
Contributor

@jumjum123 jumjum123 commented Oct 15, 2018

Espressif released V3.1 some weeks ago. New binary is 1308KB now, which is huge...
There are some major changes to get it running for Espruino, please see files in targets/ESP32/ChangesV3.1 for more detailled information. Especially some changes in sdkconfig.

Major changes:

  • some deprecated api calls are replaced
  • mbedtls in standard Espruino does not work with V3.1, an option is added to make for compiling with board specific mbedtls
  • support of PSRAM is working now, WROVER boards now have 20000 jsvars
  • additional includes and args in make

To Do:

  • check changes for side effects (you never know)
  • check building with Wilberforce tools
  • once this is done, merge to Espruino
  • check for size of heap. Right now we calculate free heap of 40k, should be working with 30K, which would result in 3000 jsvars for
  • any idea to reduce size ?
@gfwilliams

This comment has been minimized.

Copy link
Member

@gfwilliams gfwilliams commented Oct 15, 2018

Hi, the changes look good. However I'm a bit confused by https://github.com/espruino/Espruino/compare/ESP32-V3.1#diff-0310c52e4f91e3090ce7a7bfd24eb752R23

Is it possible to just modify the actual config file? https://github.com/espruino/Espruino/blob/master/libs/crypto/mbedtls/config.h

Or failing that, maybe just add them to defines in ESP32.make for mbedtls?

That would be way preferable to a board-specific hack (since where the defines are, they won't actually change the compilation for mbedtls itself).

@jumjum123

This comment has been minimized.

Copy link
Contributor Author

@jumjum123 jumjum123 commented Oct 15, 2018

Moved defines to make/family/ESP32.make and pushed to ESP32-V3.1 branch.
Much better solution than before

@gfwilliams

This comment has been minimized.

Copy link
Member

@gfwilliams gfwilliams commented Oct 15, 2018

Thanks! I'm definitely happy with that.

@jumjum123

This comment has been minimized.

Copy link
Contributor Author

@jumjum123 jumjum123 commented Oct 15, 2018

To correct a kind of typo, its moved to make/crypto/ESP32.make

@gfwilliams

This comment has been minimized.

Copy link
Member

@gfwilliams gfwilliams commented Oct 16, 2018

How ready are we to merge this? I'd like to get a new release of Espruino out this week (ideally tomorrow) - is it likely we'd get it all sorted by then, or should I leave this for the next release?

@jumjum123

This comment has been minimized.

Copy link
Contributor Author

@jumjum123 jumjum123 commented Oct 16, 2018

I would prefer to wait for wilberforce, to get it running with his build tool

@wilberforce

This comment has been minimized.

Copy link
Member

@wilberforce wilberforce commented Oct 16, 2018

Having trouble compling on buildtools at the moment.....

CC /mnt/c/Users/rhys/esp32/V3.1/EspruinoBuildTools/esp32/build/app/build/esp32/hwcrypto/aes.o
/mnt/c/Users/rhys/esp32/V3.1/EspruinoBuildTools/esp32/build/esp-idf/components/mbedtls/mbedtls/library/base64.c:37:30: fatal error: mbedtls/platform.h: No such file or directory
In file included from /mnt/c/Users/rhys/esp32/V3.1/EspruinoBuildTools/esp32/build/esp-idf/components/bt/bluedroid/common/include/common/bt_target.h:1950:0,
                 from /mnt/c/Users/rhys/esp32/V3.1/EspruinoBuildTools/esp32/build/esp-idf/components/bt/bluedroid/device/controller.c:19:
/mnt/c/Users/rhys/esp32/V3.1/EspruinoBuildTools/esp32/build/esp-idf/components/bt/bluedroid/common/include/common/bt_trace.h:35:21: fatal error: esp_log.h: No such file or directory

The espruino V3.1 changes are compling, struggling to get the libraries to build at present.

@jumjum123

This comment has been minimized.

Copy link
Contributor Author

@jumjum123 jumjum123 commented Oct 17, 2018

My way to get it running is

  • create executable file setenv.sh
#!/bin/bash
export ESP_IDF_PATH=/home/esp32/V3.1/esp-idf
export IDF_PATH=/home/esp32/V3.1/esp-idf
export ESP_APP_TEMPLATE_PATH=/home/esp32/V3.1/template
export COMPONENT_PATHS=/home/esp32/V3.1/esp-idf/components
export BOARD=ESP32
export RTOS=1
export USE_BLUETOOTH=1
[[ ":$PATH:" != *":/home/esp32/xtensa-esp32-elf/bin:"* ]] && PATH="/home/esp32/xtensa-esp32-elf/bin:${PATH}"
  • create executable file compile.sh
#!/bin/bash
export ESP_IDF_PATH=/home/esp32/V3.1/esp-idf
export IDF_PATH=/home/esp32/V3.1/esp-idf
export ESP_APP_TEMPLATE_PATH=/home/esp32/V3.1/template
export BOARD=ESP32
export RTOS=1
export USE_BLUETOOTH=1
[[ ":$PATH:" != *":/home/esp32/xtensa-esp32-elf/bin:"* ]] && PATH="/home/esp32/xtensa-esp32-elf/bin:${PATH}"
cd Espruino
make clean all
make
  • download sources from github
git clone -b v3.1 --recursive https://github.com/espressif/esp-idf.git esp-idf
git clone https://github.com/espressif/esp-idf-template.git template
git clone -b ESP32-V3.1 https://github.com/espruino/Espruino.git
cp Espruino/targets/esp32/Changes_V3.1/* template
  • compile template
. ./setenv.sh
cd template
make
cd ..
  • compile Espruino
./compile.sh
@wilberforce

This comment has been minimized.

Copy link
Member

@wilberforce wilberforce commented Oct 20, 2018

To correct a kind of typo, its moved to make/crypto/ESP32.make

A normal make clean now fails with:
Makefile:548: make/crypto/LINUX.make: No such file or directory

@jumjum123

This comment has been minimized.

Copy link
Contributor Author

@jumjum123 jumjum123 commented Oct 20, 2018

Oh sorry, there was a problem in check for file existance for non ESP32-boards in make/crypto directory

@wilberforce

This comment has been minimized.

Copy link
Member

@wilberforce wilberforce commented Oct 22, 2018

@jumjum123

managed to get a successful build with 3.1 and the latest changes from master merged.

https://www.espruino.com/binaries/travis/4f2b50f97887874339db5fae1f643fb6a6a47489/espruino_esp32.bin

Please try.

@jumjum123

This comment has been minimized.

Copy link
Contributor Author

@jumjum123 jumjum123 commented Oct 22, 2018

Downloaded binary and started, looks good.
I used partition and boot from my testing

Next downloaded https://www.espruino.com/binaries/travis/4f2b50f97887874339db5fae1f643fb6a6a47489/espruino_2v00.18_esp32.tgz which includes boot and partition, and get this error
E (29) esp_image: image at 0x20000 has invalid magic byte
E (29) boot: Factory app partition is not bootable
E (29) esp_image: image at 0x170000 has invalid magic byte
E (33) boot: OTA app partition slot 0 is not bootable
E (38) boot: No bootable app partitions in the partition table

Could this be the change in partition table where partition needs to be commented ?
ESP-IDF V3.1 does not like the entry partition,data,0,0x8000,4K anymore

@wilberforce

This comment has been minimized.

Copy link
Member

@wilberforce wilberforce commented Oct 22, 2018

@jumjum123

ESP-IDF V3.1 does not like the entry partition,data,0,0x8000,4K anymore

Sorry - I don't follow the difference? I looked at your partitions.csv and it looked the same?

I have had strange messages about the partition table being larger than 4Mb as well so had to resize the last partition (flashfs) as smaller - which is a bit weird!

@jumjum123

This comment has been minimized.

Copy link
Contributor Author

@jumjum123 jumjum123 commented Oct 22, 2018

take this one https://github.com/espruino/Espruino/blob/ESP32-V3.1/targets/esp32/Changes_V3.1/partitions_espruino.csv
Espressif guys told me, partition is not a real partition, and during change for larger partition file, they made this decision.
Agree, that this is weird, but Espressif is the owner, ....

@wilberforce

This comment has been minimized.

Copy link
Member

@wilberforce wilberforce commented Oct 22, 2018

@jumjum123

Thanks - The partition is building now after commenting that out.

I'll see if I can compile the whole thing and do a build that also replaced all of the partitions.

I have notice that some of the sdkconfig I set have been dropped:

CONFIG_LWIP_MAX_SOCKETS=10
CONFIG_LWIP_SO_RCVBUF=Y

So at the end of this update - we'll need to make sure your sdkconfig is the same.

@jumjum123

This comment has been minimized.

Copy link
Contributor Author

@jumjum123 jumjum123 commented Oct 23, 2018

Great, searched for the your changes, and couldn't find.
Could we add the actual sdkconfig and partitions_espruino.csv to targets/esp32/ on github ?

This would make it easier for everybody to build its own toolchain.
last not least, I still have to correct the documentation. It would simplify this ugly part of the job ;-)

BTW, I found a way to create board specific API-Docu. Only had to add an argument to gordons script.
See example http://jumspruino.jumware.com/ESP32/docu/functions.html
In my toolchain, make creates a new file and adds it to tgz-file

@wilberforce

This comment has been minimized.

Copy link
Member

@wilberforce wilberforce commented Oct 23, 2018

I have added sdkconfig changes to the buildtools at this stage.

The latest travis build should have the correct boot loader and partition bins - sorry I've not had a chance to test yet.

@jumjum123

This comment has been minimized.

Copy link
Contributor Author

@jumjum123 jumjum123 commented Oct 24, 2018

Very good news first, I've tested your binary and got it running.
First test and big surprise, its much faster compared to my latest build. And the size is about 20 kb smaller.
I've tested with a simple loop
var x,d = new Date();for(var i = 0; i < 1000;i++)x = i;d = new Date() - d; console.log(d);
With your binary it takes 390 ms and with mine 470

Now strange news begin
Downloaded everything from zero and compiled on my ubuntu server
Result, it works, but its much slower, compared to your binary, and size is still 20Kb more

Since I already had the plan for long time, to switch to your buildtools, built it up
Got your tool running without any problems and I like it.
But the binary was still big and slow

There was a chat on espressif server about a new toolchain https://esp32.com/viewtopic.php?f=10&t=7400 for testing.
Gave it a chance and now I've the speed of your binary with my binary, success :-)
But there is still a gap in size to your binary of about 15KB

Do you have any idea, what I'm doing wrong ?

@jumjum123

This comment has been minimized.

Copy link
Contributor Author

@jumjum123 jumjum123 commented Oct 24, 2018

After next testing with new compiler, the result is not funny
There are some problems like reset() not working, variables are not recognized correctly, etc. etc.
Actual status, your binary still works fine, but I don't get it built :-(

@wilberforce

This comment has been minimized.

Copy link
Member

@wilberforce wilberforce commented Oct 25, 2018

Very good news first, I've tested your binary and got it running.
First test and big surprise, its much faster compared to my latest build. And the size is about 20 kb smaller.

It will be due to the debug vs release flags.

For the travis builds the final espruino make is done like his:
BOARD=ESP32 RELEASE=1

This could explain the difference in speeds as the assert statement would be dropped.

After next testing with new compiler, the result is not funny

Sound like we should just get this release with existing compiler out first and do that as a later step.

@jumjum123

This comment has been minimized.

Copy link
Contributor Author

@jumjum123 jumjum123 commented Oct 25, 2018

Great got it working with RELEASE=1, thanks a lot.
Will do some more testing during weekend.
I agree to ignore new compiler for now.
@gfwilliams, are there any plans on your side to use GCC 8.2 ?

@gfwilliams

This comment has been minimized.

Copy link
Member

@gfwilliams gfwilliams commented Oct 26, 2018

are there any plans on your side to use GCC 8.2 ?

Not until there's an arm-none-eabi one I can get easily from https://developer.arm.com/open-source/gnu-toolchain/gnu-rm/downloads

Honestly I usually wait a bit as well as I've found the first release to be a bit flaky. I'm using 5.4.1 at the moment for ARM, but when I first started using GCC 5 it broke a bunch of stuff - and I didn't move to GCC 6 initially because that broke stuff too.

@wilberforce

This comment has been minimized.

Copy link
Member

@wilberforce wilberforce commented Nov 6, 2018

@jumjum123

Have you completed your testing? Are we ready to bring this into the master branch?

@jumjum123

This comment has been minimized.

Copy link
Contributor Author

@jumjum123 jumjum123 commented Nov 6, 2018

Looks like we were both waiting for OK ;-)
I did a few testing, and everything runs fine.
To my surprise, ESP32 is slow, even some percent slower as first Espruno.
All together we are ready to bring it to master.
Big pro is support of up to 20000 Jsvars for WROVER boards

@wilberforce

This comment has been minimized.

Copy link
Member

@wilberforce wilberforce commented Nov 6, 2018

All together we are ready to bring it to master.
@jumjum123 - I think @gfwilliams is away at the moment - so we'll ask to merge once he returns.

Big pro is support of up to 20000 Jsvars for WROVER boards
We'll need to add this to the Changelog

It seems that neopixel is still broken - I have no idea why this stopped?

To my surprise, ESP32 is slow, even some percent slower as first Espruno.
Do you have some benchmarks to compare?

@jumjum123

This comment has been minimized.

Copy link
Contributor Author

@jumjum123 jumjum123 commented Nov 7, 2018

Sorry, no idea about neopixel.
There was a change in API, where a call was replaced by a new one.
This did'nt solve the problem.
Its still on my todo list but not on first place.
Right now I'm working on a general solution for 64x32 LED boards

@wilberforce

This comment has been minimized.

Copy link
Member

@wilberforce wilberforce commented Nov 13, 2018

@wilberforce wilberforce reopened this Nov 27, 2018
@wilberforce

This comment has been minimized.

Copy link
Member

@wilberforce wilberforce commented Nov 27, 2018

We have issues with 3.1 not connecting to wifi, when there is no previously saved connection.

Need to work out why wifi.connect() does not complete or revert back to 3.0.x

See
#1571

@wilberforce wilberforce added the ESP32 label Nov 27, 2018
@jumjum123

This comment has been minimized.

Copy link
Contributor Author

@jumjum123 jumjum123 commented Nov 29, 2018

To add some info:
Errormessage 1105 is not enough space. phy data is stored in nvs data partition or in a special phy partition. I checked creating a phy partition. This added one more binary, and errormessage did not appear anymore.
Next I had the problem that I got event SYSTEM_EVENT_STA_START, and 3secs later SYSTEM_EVENT_STA_DISCONNECTED. After playing around with definition of wifi_config_t in jswrap_wifi_connect (jswrap_esp32_network.c) right now I get a connection. Tested this with 2 different boards. Problem is that I don't know, why this works now.
I will do some more testing and playing around, but family will take their time.
Oh before I forget, this all was done with a corrected version of ESPV3.1.1
BTW, contact to Espressif is like walking through honey

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.