Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ESP32: https request no longer works #1613

Open
wilberforce opened this issue Feb 4, 2019 · 25 comments
Open

ESP32: https request no longer works #1613

wilberforce opened this issue Feb 4, 2019 · 25 comments
Labels

Comments

@wilberforce
Copy link
Member

@wilberforce wilberforce commented Feb 4, 2019

See:

http://forum.espruino.com/conversations/330049/#comment14604279

var http = require("http");
http.get("https://www.google.com", function(res) {
  res.on('data', function(data) {
    console.log(data);
  });
});
Uncaught InternalError: Failed! mbedtls_ssl_setup: Not enough memory
@wilberforce wilberforce added the ESP32 label Feb 4, 2019
@GabrielGab
Copy link

@GabrielGab GabrielGab commented Feb 9, 2019

yes , I have verified same problem on version 2.01 for ESP32 but ESP8266 still working fine for https, on same version.

@wilberforce
Copy link
Member Author

@wilberforce wilberforce commented Feb 12, 2019

Working:

>process.env
={
  "VERSION": "1v97",
  "GIT_COMMIT": "74b2511",
  "BOARD": "ESP32",
  "FLASH": 0, "RAM": 524288,
  "SERIAL": "240ac400-2f90",
  "CONSOLE": "Serial1",
  "MODULES": "Flash,Storage,fs," ... "r,crypto,neopixel",
  "EXPTR": 1073417244 }
> 

{ "free": 2601, "usage": 2399, "total": 5000, "history": 53,
  "gc": 0, "gctime": 2.061 }
E (64222) esp_ota_ops: ota data invalid, no current app. Assuming factory
{
  "sdkVersion": "v2.1",
  "freeHeap": 129480, "addr": 65536,
  "partitionBoot": "factory"
 }


Fail:

process.env
={
"VERSION": "1v98",
"GIT_COMMIT": "1e06400",
"BOARD": "ESP32",
"FLASH": 0, "RAM": 524288,
"SERIAL": "240ac400-2f90",
"CONSOLE": "Serial1",
"MODULES": "Flash,Storage,fs," ... "r,crypto,neopixel",
"EXPTR": 1073485304 }
ESP32.getState()
={
"sdkVersion": "v3.0",
"freeHeap": 37668 }


Current:

>   console.log(process.env);
{
  "VERSION": "2v01.33",
  "GIT_COMMIT": "adb3b7d",
  "BOARD": "ESP32",
  "FLASH": 0, "RAM": 524288,
  "SERIAL": "240ac400-2f90",
  "CONSOLE": "Serial1",
  "MODULES": "Flash,Storage,heatshrink,fs,net,dgram,tls,http,NetworkJS,Wifi,TelnetServer,crypto,neopixel",
  "EXPTR": 1073484860 }
=undefined
{ "free": 3965, "usage": 135, "total": 4100, "history": 17,
  "gc": 0, "gctime": 3.262 }
=undefined
>  console.log( ESP32.getState());
{
  "sdkVersion": "v3.1.2-dirty",
  "freeHeap": 20412, "BLE": false, "Wifi": true, "minHeap": 10984 }
=undefined


@wilberforce
Copy link
Member Author

@wilberforce wilberforce commented Feb 12, 2019

Got this working by reducing jsvars. The current esp-idf build is using more heap. Looking at reducing the dynamic buffers - also set to 16 top buffers when 10 in the past.

@gfwilliams
Copy link
Member

@gfwilliams gfwilliams commented Feb 12, 2019

Normal Espruino builds actually allocate space for HTTPS out of JsVars and that was disabled for ESP32 - potentially that could be re-added I guess?

@jumjum123
Copy link
Contributor

@jumjum123 jumjum123 commented Feb 12, 2019

@wilberforce, did you check minHeap in ESP32.getState ?
In theory, this should show minimum size of free heap, happened sometime during execution
Idea was to use this to adjust number of jsvars
freeHeap is 37 kb, which is a lot.

@wilberforce
Copy link
Member Author

@wilberforce wilberforce commented Feb 12, 2019

Ah - thanks - that explains it.
I think that when @jumjum123 changed to use the esp-idf version on mbedTLS that was lost. This was due to the esp-odd version being several versions ahead of the version use in espruino and there where missing functions and it would not compile.

@wilberforce
Copy link
Member Author

@wilberforce wilberforce commented Feb 12, 2019

@jumjum123 - I don't think free heap is that large on the current travis build - perhaps 32k.

I also recall there was a https buffer size setting in espruino - as the default is something like 15k.

A bit more reseach!

@wilberforce
Copy link
Member Author

@wilberforce wilberforce commented Feb 13, 2019

Looking at the available heap in the versions above v2 has much more heap than v3!

It looks like the callow used can be changed here:

https://github.com/espressif/esp-idf/blob/master/components/mbedtls/Kconfig

And the buffer size from 16k to 512 and also turn off suites not used to match the espruino/mdebtls/config.h

@MaBecker
Copy link
Contributor

@MaBecker MaBecker commented Mar 7, 2019

@GabrielGab

but ESP8266 still working fine for https, on same version.

Nice, do you like to share some details using https with ESP8266?

@thedamian
Copy link

@thedamian thedamian commented Aug 23, 2019

@MaBecker

    var http = require("http");
    http.get("https://www.google.com", function(res) {
      res.on('data', function(data) {
        console.log(data);
      });
    });

But let's get back to how we'll fix this.

@thedamian
Copy link

@thedamian thedamian commented Aug 25, 2019

I can confirm that this is a FIXED issue on v2.04 for BOTH ESP32 and ESP8266

I suggest this issue is marked "SOLVED"

This was the code I used in BOTH devices successfully.
Devices it was successfully tested on:
ESP8266: D1 Mini & NodeMcu Lua ESP8266 ESP-12F
ESP32: ESP32-CAM , the LILYGO® TTGO & official expressif ESP32-WROOM-32D

Write up with screenshots and details is here: http://forum.espruino.com/conversations/330049/#comment14873565

var ssid = '***';
var password = '****';
var port = 80;
var wifi = require('Wifi');
console.log("started");
wifi.connect(ssid, {password: password}, function() {
    console.log('Connected to Wifi.  IP address is:', wifi.getIP().ip);
    var http = require('http');
    http.get("https://www.google.com", function(res) {
      res.on('data', function(data) {
        console.log(data);
      });
    });
    // wifi.save(); // Next reboot will auto-connect
});
@gfwilliams gfwilliams closed this Aug 27, 2019
@olliephillips
Copy link

@olliephillips olliephillips commented Oct 12, 2019

@wilberforce I see this in v2.04 on ESP32. I'm struggling to understand what the solution is from the issue thread, although it's closed. Are there any pointers you can provide?

@wilberforce
Copy link
Member Author

@wilberforce wilberforce commented Oct 13, 2019

@olliephillips

With some sdk updates, related to ble, the mbedtls was switched from the esprunio version, for the esp-idf version. This was due to functions that the ble code used that was not in the espruino version( it is behind the esp-idf version). The espruino code was modified to use jsvars as memory buffers, so the current implementation is trying to allocated from the heap and there is not enough free memory for this.

@olliephillips
Copy link

@olliephillips olliephillips commented Oct 13, 2019

@wilberforce
Copy link
Member Author

@wilberforce wilberforce commented Oct 13, 2019

@olliephillips
You mean https? - it should be fine for http.

A work around for a custom build would be to reduce the number of jsvars, so that these is more heap space left.

1 similar comment
@wilberforce
Copy link
Member Author

@wilberforce wilberforce commented Oct 13, 2019

@olliephillips
You mean https? - it should be fine for http.

A work around for a custom build would be to reduce the number of jsvars, so that these is more heap space left.

@wilberforce wilberforce reopened this Oct 13, 2019
@wilberforce
Copy link
Member Author

@wilberforce wilberforce commented Oct 13, 2019

@olliephillips
You mean https? - it should be fine for http.

A work around for a custom build would be to reduce the number of jsvars, so that these is more heap space left.

@olliephillips
Copy link

@olliephillips olliephillips commented Oct 14, 2019

@MaBecker
Copy link
Contributor

@MaBecker MaBecker commented Mar 13, 2020

@olliephillips did you get https working? If yes please share your changes - thanks.

@olliephillips
Copy link

@olliephillips olliephillips commented Mar 13, 2020

Hey Mark, I didn't, no. I had a go at it, but quickly parked it I recollect, and I've not had much time to spend with Espruino recently.

@MaBecker MaBecker mentioned this issue Mar 21, 2020
3 of 15 tasks complete
@23RoMax
Copy link

@23RoMax 23RoMax commented May 17, 2020

@MaBecker I've tried it with three different heapsizes/heapvar configurations.

As you have recommended here:

heapVars = (esp_get_free_heap_size() - 60000) / 16;
--> HTTPS Calls are possible
--> New issue I encountered: mbedtls_ssl_handshake returned -0x4290

heapVars = (esp_get_free_heap_size() - 65000) / 16;
--> HTTPS Calls are possible
--> Same issue
--> Additionally: Memory warnings

heapVars = (esp_get_free_heap_size() - 70000) / 16;
--> Memory warnings, no connection to wifi has been established by my ESP32 (coincidence? - not sure if it is related)

Now that I got HTTPS working, I still can't make use of it due to the handshake error above.
Some research suggested a memory leak.
EDIT: Second resource suggesting a memory issue here.

Did anyone encounter this issue and solve it?

@23RoMax
Copy link

@23RoMax 23RoMax commented May 24, 2020

Hi @MaBecker ,

any chance you would find a minute to have a look at my previous comment :) ?
Highly appreciated to get a hint :)

@MaBecker
Copy link
Contributor

@MaBecker MaBecker commented May 24, 2020

Hi 23RoMax,

sure, please share the url that is causing the mbedtls_ssl_handshake returned -0x4290.

Your build is using this: heapVars = (esp_get_free_heap_size() - 70000) / 16;

Have you removed some modules ?

If you like we can chat over gitter.im.

@23RoMax
Copy link

@23RoMax 23RoMax commented May 24, 2020

Thank you!
I tried several RESTful APIs as I'm trying to extend my coin tracker.

The first endpoint I tried was https://api.coinbase.com/v2/prices/BTC-EUR/buy

Another one I am successfully using with http requests is: https://api.coindesk.com/v1/bpi/currentprice/EUR.json

Also https://google.com yields the same result.

My latest build that seemed stable was with
heapVars = (esp_get_free_heap_size() - 60000) / 16;

I noticed that anything higher caused issues and crashes.
Didn't remove any other modules.

I don't have a gitter.im account yet, will set one up.

Thank you for your help!

@MaBecker
Copy link
Contributor

@MaBecker MaBecker commented May 30, 2020

Ok, I also disabled the dynamic var allocation by commenting

'DEFINES+=-DJSVAR_MALLOC', # Allocate space for variables at jsvInit time

and set

'variables' : 2500, # JSVAR_MALLOC is defined below - so this can vary depending on what is initialised

to 1800.

this will give you a very large heap space

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.