Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Building Crypto on ESP8266 - compile errors #807

Closed
wilberforce opened this issue Feb 13, 2016 · 82 comments
Closed

Building Crypto on ESP8266 - compile errors #807

wilberforce opened this issue Feb 13, 2016 · 82 comments
Labels
ESP8266 This is only a problem on ESP8266 devices

Comments

@wilberforce
Copy link
Member

wilberforce commented Feb 13, 2016

Adding the the Makefile:

else ifdef ESP8266_BOARD
EMBEDDED=1
USE_NET=1
USE_TELNET=1
USE_CRYPTO=1
BOARD=ESP8266_BOARD
CC libs/crypto/jswrap_crypto.o
libs/crypto/jswrap_crypto.c: In function 'jswrap_crypto_error':
libs/crypto/jswrap_crypto.c:91:3: error: invalid initializer
   if (e) jsError(e);
   ^
libs/crypto/jswrap_crypto.c: At top level:JsVar
void jswrap_crypto_error(int err) {
  const char *e = jswrap_crypto_error_to_str(err);
>>>  if (e) jsError(e);
  else jsError("Unknown error: -0x%x", -err);
}

The issue appears to be the flash_str #define:

[#define](http://forum.espruino.com/sear­ch/?q=%23define) jsError(fmt, ...) do { \
    FLASH_STR(flash_str, fmt); \
    jsError_flash(flash_str, ##__VA_ARGS__); \
  } while(0)

root@esp8266-VirtualBox:~/Espruino# grep '#define FLASH_STR' src/*.h
src/jsutils.h:#define FLASH_STR(name, x) static char name[] __attribute__((section(".irom.literal"))­) __attribute__((aligned(4))) = x

I tried changing:

#define FLASH_STR(name, x) static char name[] __attribute__((section(".irom.literal"))) __attribute__((aligned(4))) = x

to
#define FLASH_STR(name, x) static char * name __attribute__((section(".irom.literal"))) __attribute__((aligned(4))) = x

libs/network/esp8266/jswrap_esp8266_network.c:151:3: error: initializer element is not constant
__wr0, __wr1, __wr2, __wr3, __wr4, __wr5, __wr6, __wr7, __wr8, __wr9, __wr10,
^
libs/network/esp8266/jswrap_esp8266_network.c:151:3: error: (near initialization for 'wifiReasons[0]')

However then it fails here:

FLASH_STR(__ev0, "#onassociated");
FLASH_STR(__ev1, "#ondisconnected");
FLASH_STR(__ev2, "#onauth_change");
FLASH_STR(__ev3, "#onconnected");
FLASH_STR(__ev4, "#ondhcp_timeout");
FLASH_STR(__ev5, "#onsta_joined");
FLASH_STR(__ev6, "#onsta_left");
FLASH_STR(__ev7, "#onprobe_recv");
static char *wifi_events[] = { __ev0, __ev1, __ev2, __ev3, __ev4, __ev5, __ev6, __ev7 };

What is the best way to solve this?

@wilberforce
Copy link
Member Author

wilberforce commented Feb 13, 2016

Further to above, if the line is changed:
if (e) jsError(e);
to
if (e) jsError("error");

so it complies, then the following error occurs:

root@esp8266-VirtualBox:~/Espruino# ./build.sh
CC src/jslex.o
CC src/jsvar.o
CC src/jsvariterator.o
CC src/jsutils.o
CC src/jsnative.o
CC src/jsparse.o
CC src/jspin.o
CC src/jsinteractive.o
CC src/jsdevices.o
CC src/jstimer.o
CC src/jsspi.o
Generating JS wrappers
WRAPPERSOURCES = src/jswrap_array.c src/jswrap_arraybuffer.c src/jswrap_date.c src/jswrap_error.c src/jswrap_espruino.c src/jswrap_flash.c src/jswrap_functions.c src/jswrap_interactive.c src/jswrap_io.c src/jswrap_json.c src/jswrap_modules.c src/jswrap_pin.c src/jswrap_number.c src/jswrap_object.c src/jswrap_onewire.c src/jswrap_pipe.c src/jswrap_process.c src/jswrap_serial.c src/jswrap_spi_i2c.c src/jswrap_stream.c src/jswrap_string.c src/jswrap_waveform.c libs/math/jswrap_math.c libs/network/jswrap_net.c libs/network/http/jswrap_http.c libs/network/js/jswrap_jsnetwork.c libs/network/esp8266/jswrap_esp8266_network.c targets/esp8266/jswrap_esp8266.c libs/network/telnet/jswrap_telnet.c libs/crypto/jswrap_crypto.c
DEFINES = -DGIT_COMMIT=e42ca309cdd9e55cea592bf28ff08cfbfa5c95fb -DBUILDNUMBER="262" -DUSE_DEBUGGER -DUSE_TAB_COMPLETE -DUSE_HEATSHRINK -DUSE_MATH -DUSE_NET -DUSE_ESP8266 -DUSE_TELNET -DUSE_OPTIMIZE_PRINTF -D__ETS__ -DICACHE_FLASH -DXTENSA -DUSE_US_TIMER -DEMBEDDED -DESP8266
CC libs/crypto/jswrap_crypto.o
CC targets/esp8266/spi.o
CC targets/esp8266/user_main.o
CC targets/esp8266/log.o
CC targets/esp8266/jshardware.o
CC targets/esp8266/i2c_master.o
CC targets/esp8266/esp8266_board_utils.o
CC gen/jswrapper.o
LD espruino_esp8266_partial.o
LD espruino_esp8266_user1.elf
LD espruino_esp8266_user2.elf
/home/esp8266/esp-open-sdk/xtensa-lx106-elf/lib/gcc/xtensa-lx106-elf/4.8.2/../../../../xtensa-lx106-elf/bin/ld: address 0x3fffd488 of espruino_esp8266_user2.elf section `.bss' is not within region `dram0_0_seg'
/home/esp8266/esp-open-sdk/xtensa-lx106-elf/lib/gcc/xtensa-lx106-elf/4.8.2/../../../../xtensa-lx106-elf/bin/ld: address 0x3fffd488 of espruino_esp8266_user2.elf section `.bss' is not within region `dram0_0_seg'
collect2: error: ld returned 1 exit status
make: *** [espruino_esp8266_user2.elf] Error 1
make: *** Waiting for unfinished jobs....
/home/esp8266/esp-open-sdk/xtensa-lx106-elf/lib/gcc/xtensa-lx106-elf/4.8.2/../../../../xtensa-lx106-elf/bin/ld: address 0x3fffd488 of espruino_esp8266_user1.elf section `.bss' is not within region `dram0_0_seg'
/home/esp8266/esp-open-sdk/xtensa-lx106-elf/lib/gcc/xtensa-lx106-elf/4.8.2/../../../../xtensa-lx106-elf/bin/ld: address 0x3fffd488 of espruino_esp8266_user1.elf section `.bss' is not within region `dram0_0_seg'
collect2: error: ld returned 1 exit status
make: *** [espruino_esp8266_user1.e
```lf] Error 1

My understanding is that the linked code is too large?

So does something need to be cut back to make it fit?

I'm adding crypto so that I have access to the SHA1 function for use in web sockets.

@gfwilliams
Copy link
Member

gfwilliams commented Feb 15, 2016

Ok, it looks like in the error case, the issue is that ESP8266 has to store strings in flash unless you access them in a special way - there's a hack for jsError because its strings are almost always constant. Try: if (e) jsError("%s", e);

And yes... the other issue you have is probably also related to that. On ESP8266, unless you add special hacks, all constant data is put in RAM. The best solution for now would be to make sure you add "ifndef":"ESP8266" to the JSON for AES and SHA256/etc functions - so they will not get included and RAM will not get used up for their constant data.

@wilberforce
Copy link
Member Author

wilberforce commented Feb 16, 2016

Thanks. The jserror %s hack now compiles.

I tried added the ifndef around the other functions, however still ran out of space.

@gfwilliams
Copy link
Member

gfwilliams commented Feb 16, 2016

You could try also looking at libs/crypto/mbedtls/config.h and removing mention of AES/etc from there.

@urish
Copy link
Contributor

urish commented Feb 23, 2016

Hi, I encountered the same issue and sent a pr with a fix (#810).

@wilberforce did you find out how to make it fit?

@wilberforce
Copy link
Member Author

wilberforce commented Feb 24, 2016

@urish Sorry, no.

I tried what Gordon suggested with "ifndef":"ESP8266" but it was not enough. I think we need to exclude functions as I thing we are running out of rom space.

@urish
Copy link
Contributor

urish commented Feb 24, 2016

@tve any inputs on this one?

@wilberforce
Copy link
Member Author

wilberforce commented Feb 24, 2016

I'm considering writing a JavaScript module, based on the sha1.js code that @jumjum has published in the forums.

It could be a standalone require("crypto") to use on boards where sha1 is not in firmware.

This way a separate firmware would not be require and it's only about 1k.

In my use case for web sockets, the module is used during negotiation so could be unloaded once the socket is established.

@wilberforce
Copy link
Member Author

wilberforce commented Feb 25, 2016

@urish

If you save this in a projects folder in your web ide, the ws module find this module and uses it:

https://raw.githubusercontent.com/wilberforce/EspruinoDocs/master/modules/crypto.min.js

I'll post to the Forum with an example:
http://forum.espruino.com/comments/12817146/

@urish
Copy link
Contributor

urish commented Feb 25, 2016

thanks @wilberforce !

In my case, I would like to achieve TLS connections, but I am sure that crypto.js can be useful for other people :)

@wilberforce
Copy link
Member Author

wilberforce commented Feb 25, 2016

@urish
TLS will need a lot more functions, so I'm not sure if there will be enough space?

@gfwilliams
Copy link
Member

gfwilliams commented Feb 25, 2016

No, I think you're out of luck there. In it's default (standards-compliant) mode, mbedtls needs at least 16kB RAM, which is going to mean very little left for Espruino.

If you want TLS, get a Pico and attach an ESP8266 to it. The Pico has enough RAM to handle it.

@urish
Copy link
Contributor

urish commented Feb 26, 2016

@wilberforce @gfwilliams thanks for the feedback! I know that the Espressif SDK does have TLS support (I use it with NodeMCU). Do you know whether there was any work done on using their TLS support with Espruino?

@gfwilliams
Copy link
Member

gfwilliams commented Feb 27, 2016

No, there wasn't any work done.

IMO it's worth seeing if mbedtls can be tweaked to fit, because then the same code will be used for all Espruino platforms.

If not we're going to end up with a bunch of code that's just left to rot, and that won't get the bugfixes and improvements that the main mbedtls-based code gets.

@gfwilliams
Copy link
Member

gfwilliams commented Mar 4, 2016

Just had a quick look at this, and it seems that mbedtls will compile just fine, but as-is there's not enough memory.

You can modify libs/crypto/mbedtls/config.h to remove features and can then remove bits of libs/crypto/jswrap_crypto.c that used the other features, and I think it'll then compile.

@gfwilliams gfwilliams added the ESP8266 This is only a problem on ESP8266 devices label Mar 4, 2016
@wilberforce
Copy link
Member Author

wilberforce commented Mar 5, 2016

@gfwilliams
You have mentioned the hashlib module will be depreciated in favor of the mdedtls, is there any easy way to remove this from being complied - as that would free up some memory.

I was also looking at the optimisations that @tve had done moving strings to flash, and wondering how much scope there was to free up more ram?
#697

@tve
Copy link
Contributor

tve commented Mar 5, 2016

I was also looking at the optimisations that @tve had done moving strings to flash, and wondering how much scope there was to free up more ram?
Try make topstrings (read the relevant makefile section first)

@wilberforce
Copy link
Member Author

wilberforce commented Mar 8, 2016

@tve thanks.

I tired to build using the top strings, but using a recent git pull with the standard Makefile, I can no longer build...

Sorry can't paste here, the error is here:

https://gitter.im/espruino/Espruino?at=56de471268ddef776469772f

It appears that the user1 elf is bigger than 480k and won't fit in the iram1_0_seg _section ?

@urish
Copy link
Contributor

urish commented Mar 12, 2016

Seems like it is possible to fit mbedtls into the ESP8266, as it has been done in esp-open-rtos

@wilberforce
Copy link
Member Author

wilberforce commented Mar 13, 2016

@tve
I have, ./build-topstrings.sh:

root@esp8266-VirtualBox:/home/esp8266/Espruino# cat build-topstrings.sh 
#! /bin/bash
export DISABLE_LTO=1 
export ESP8266_BOARD=1
export FLASH_4MB=1
export PATH=/home/esp8266/esp-open-sdk/xtensa-lx106-elf/bin:$PATH
export COMPORT=/dev/ttyUSB0
make $* 
make clean
./build-topstrings.sh
./build-topstrings.sh topstrings
root@esp8266-VirtualBox:/home/esp8266/Espruino# ./build-topstrings.sh topstrings

Top 20 from ./topstrings:
To get details: xtensa-lx106-elf-objdump -j .rodata.str1.4 -s src/FILENAME.o
root@esp8266-VirtualBox:/home/esp8266/Espruino# ls -l topstrings 
-rw-r--r-- 1 root root 0 Mar 13 15:23 topstrings

The file size is 0 - what am I doing wrong?

@MaBecker
Copy link
Contributor

MaBecker commented Mar 13, 2016

@wilberforce use xtensa-lx106-elf-objdump -h filename.o to check the used named

in my case it is .rodata.str1.1 and not .rodata.str1.4, so change the Makefile

xtensa-lx106-elf-objdump -h ./src/jswrap_pin.o

./src/jswrap_pin.o:     file format elf32-xtensa-le

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .literal      00000000  00000000  00000000  00000034  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  1 .text         000003ce  00000000  00000000  00000034  2**2
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  2 .data         00000000  00000000  00000000  00000402  2**0
                  CONTENTS, ALLOC, LOAD, DATA
  3 .bss          00000000  00000000  00000000  00000402  2**0
                  ALLOC
  4 .rodata.str1.1 00000028  00000000  00000000  00000402  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  5 .irom.literal 0000004d  00000000  00000000  0000042c  2**2
                  CONTENTS, ALLOC, LOAD, DATA
  6 .comment      00000022  00000000  00000000  00000479  2**0
                  CONTENTS, READONLY
  7 .xtensa.info  00000038  00000000  00000000  0000049b  2**0
                  CONTENTS, READONLY
  8 .xt.lit       00000048  00000000  00000000  000004d3  2**0
                  CONTENTS, RELOC, READONLY
  9 .xt.prop      000001c8  00000000  00000000  0000051b  2**0
                  CONTENTS, RELOC, READONLY

@wilberforce
Copy link
Member Author

wilberforce commented Mar 13, 2016

Great! Thanks. here is the output:

Top 20 from ./topstrings:
7250 ./espruino_esp8266_partial.o
765 ./src/jsinteractive.o
542 ./libs/network/socketserver.o
512 ./gen/jswrapper.o
438 ./libs/network/esp8266/jswrap_esp8266_network.o
387 ./libs/network/esp8266/ota.o
295 ./src/jslex.o
245 ./src/jsvar.o
225 ./src/jsparse.o
216 ./libs/network/telnet/jswrap_telnet.o
214 ./src/jswrap_process.o
207 ./targets/esp8266/esp8266_board_utils.o
207 ./src/jswrap_json.o
196 ./src/jswrap_io.o
196 ./libs/network/socketerrors.o
176 ./libs/network/esp8266/network_esp8266.o
151 ./targets/esp8266/user_main.o
146 ./src/jswrap_object.o
137 ./src/jswrap_espruino.o
132 ./src/jswrap_pipe.o

@wilberforce
Copy link
Member Author

wilberforce commented Mar 13, 2016

In jsinteractive - there is the load banner, and quite a few calls to jsiConsolePrint which I don't think is using the FLASH_STR #define, so quite a few of these strings could go into flash memory

@urish
Copy link
Contributor

urish commented Mar 13, 2016

@wilberforce @gfwilliams gfwilliams I have spent some time on it and after reducing the number of js variables from 1400 to 1000 (inside boards/ESP8266_BOARD.py), it seems to happily compile.

However, make complains about:

** user1.bin uses 529828 bytes of 491520 available

Two questions:

  1. Do you see any reason why this would not be usable with 1000 js vars instead of 1400?
  2. Since I have a 4MByte board, does it make sense that I would be able to use this firmware somehow despite its size? Any pointers about this one?

@wilberforce
Copy link
Member Author

wilberforce commented Mar 14, 2016

@urish
When did you last do a pull, as a recent update of the mathlib reduces the size of the limage:

http://forum.espruino.com/comments/12848811/

This might be small enough.

However, I think you will run out of available RAM as the TLS stuff uses quite large buffers - bigger than the ESP8266 will have available (16K or so from memory). You might be able to compile with a smaller buffer size, perhaps 1K - and see if your https:// server will still connect to this.

One way of testing this might be to use the linux install, and try the smaller packet size with this first.

Is it https:// requests you want to do? You could use a proxy service, or use a pico and esp8266-01, as you won't have the memory limitation then

@gfwilliams
Copy link
Member

gfwilliams commented Mar 14, 2016

Yeah, you may run out of memory fot HTTPS (at least without tweaking the config.h to use non-standard (lower than normal) buffer sizes) - however I got the impression you might have just wanted Crypto for the SHA1 and other features - in which case I'd imagine you're fine.

Strange about the high usage from jsinteractive.c - if jsiConsolePrintf could be tweaked with a define the same way the errors have, you might be able to save quite bit of RAM.

@gfwilliams
Copy link
Member

gfwilliams commented Mar 14, 2016

Just to add, are you building with RELEASE=1? That will reduce firmware size a whole load.

As I understand it, you could use more flash, but it would require some linker tweaks and effectively stops you being able to do firmware updates over the air.

@urish
Copy link
Contributor

urish commented Mar 14, 2016

@gfwilliams building with RELEASE=1 indeed. Any pointers for for linker tweaks?
@wilberforce building with -lmirom, is this the new math lib?

For those who need crypto - I managed to compile a firmware just with crypto (without SSL), and the image size seems to fit (haven't tested it yet on the real device, though)

In my case - I want SSL indeed (for HTTPS and secure MQTT), and my use case is presenting the ESP8266 as an IoT capable device on tech talks I am giving. Currently, I use NodeMCU, but most of my audience is not familiar with Lua, so being able to use Espruino can be a game changer in my case.

@gfwilliams
Copy link
Member

gfwilliams commented Mar 14, 2016

I'm not sure exactly. @tve might have some ideas, or I think it's just a matter of googling. You need to spread the firmware over user1 and user2 areas.

Having said that, you could add the SAVE_ON_FLASH define in the Makefile, and remove the debugger and command completion defines if they were in there. It might save you a bit of space.

Also you can edit libs/crypto/libmbedtls/config.h (iirc) and remove any types of crypto you think you can do without.

@urish
Copy link
Contributor

urish commented Mar 15, 2016

good work :)

@urish
Copy link
Contributor

urish commented Mar 15, 2016

>[tls.connect({host:'www.google.com',port:443}),process.memory(),esp.getState().freeHeap]
Connecting with TLS...
Performing the SSL/TLS handshake...
=[
  { "type": 4,
    "opt": {
      "host": "www.google.com",
      "port": 443 },
    "sckt": 15 },
  { "free": 133, "usage": 667, "total": 800, "history": 59 },
  2240 ]
ERROR: Failed! mbedtls_ssl_handshake returned -0x2

Now need to guess which one is missing - heap or js vars

@wilberforce
Copy link
Member Author

wilberforce commented Mar 15, 2016

I removed the IF statement in

if (jsvIsObject(options)) {
    if (!ssl_load_cacert(sd, options) ||
        !ssl_load_owncert(sd, options) ||
        !ssl_load_key(sd, options)) {
      ssl_freeSocketData(sckt);
      return false;
    }
  }

Looks like at least one is required in the or.. I think the forum post mentions these options...

tls.connect(options, callback)
Description
Create a socket connection using TLS
Options can have ca, key and cert fields, which should be the decoded content of the certificate.

To use https:// you should not need to use the cert options... did you try:

require("http").get("https://www.google.­com", function(res) {
res.on('data', function(data) { console.log(data); });
});

@wilberforce
Copy link
Member Author

wilberforce commented Mar 15, 2016

Now need to guess which one is missing - heap or js vars

Looks like heap:

https://github.com/espruino/Espruino/search?utf8=%E2%9C%93&q=SOCKET_ERR_MEM

case ESPCONN_MEM: err = SOCKET_ERR_MEM; break;

507 int err = code;
508 switch (code) {
509 case ESPCONN_MEM: err = SOCKET_ERR_MEM; break;
510 case ESPCONN_ABRT: err = SOCKET_ERR_RESET; break;

1001 releaseSocket(pSocketData);
1002 return SOCKET_ERR_MEM;
1003 }

@urish
Copy link
Contributor

urish commented Mar 15, 2016

Yup, seems like it is heap - less js vars, and now we are getting -0x7200 nearly all the time.

@urish
Copy link
Contributor

urish commented Mar 15, 2016

Well, after playing with esp-open-rtos for a while, it seems like MBEDTLS_SSL_MAX_CONTENT_LEN has to be somewhere around 3900 bytes (or larger), otherwise the handshake fails.

@urish
Copy link
Contributor

urish commented Mar 15, 2016

@gfwilliams is there a way to allocate a big buffer off the js-vars space?

@gfwilliams
Copy link
Member

gfwilliams commented Mar 15, 2016

@urish all the memory needed for TLS comes straight out of the JsVars space already - so you need to increase the amount of variables as much as possible.

If you want to change that, you'll have to modify config.h again to replace calls to Espruino's jsvar malloc implementation with malloc and free.

@urish
Copy link
Contributor

urish commented Mar 15, 2016

@gfwilliams thanks for the response! I will look into it further, some more questions that come into my head:

  1. How much bytes does a single JsVars contribute to the JsVars space size?
  2. I see that two of the mbedTls modules have strings that consume great amounts of RAM (nearly 4k), which can make a huge difference on the ESP:
root@dd700b1d591e:/c/Users/Uri/Documents/esp-ssl# cat topstrings
10989 ./espruino_esp8266_partial.o
2493 ./libs/crypto/mbedtls/library/oid.o
1717 ./libs/crypto/mbedtls/library/x509_crt.o

Where can I find information how to move those strings into the Flash so they won't consume any RAM?

Thanks!

@gfwilliams
Copy link
Member

gfwilliams commented Mar 15, 2016

  1. It's worth looking at http://www.espruino.com/Internals - 12 or 16 bytes, depending on if you have >1023 vars or not.
  2. This has quite a lot of info where it's been attempted with Graphics: More built-in code for ESP8266 #770

But actually without much success, so I'm not 100% sure what's going on there. There should be a bunch of stuff on the net about moving ESP8266 stuff from RAM into flash though

@wilberforce
Copy link
Member Author

wilberforce commented Mar 15, 2016

  1. Here is the reference to work that was done to move some constants to flash
    Constant strings (and other data) moved to flash memory #697

@tve already did a whole bunch here: 294f3ff

It involves #define to wrap the strings.

@wilberforce
Copy link
Member Author

wilberforce commented Mar 15, 2016

Yikes . The oid.c is full of strings! Are you sure you need it for tls v 1.2? Might be worth looking into the switches you need for a minimum build, I reference a minimum config.h above that might be worth a shot..

@gfwilliams
Copy link
Member

gfwilliams commented Mar 15, 2016

Actually you should note that some of this stuff works by first copying the string from flash into RAM - but you don't need to do that (and probably shouldn't in the case of the crypto). It'd be better to read directly from flash

@wilberforce
Copy link
Member Author

wilberforce commented Mar 15, 2016

following on what @gfwilliams is saying, here are the macro's that pack into iRom, and then read from it:
#770 (comment)

@urish
Copy link
Contributor

urish commented Mar 16, 2016

Progress report - oid.c is indeed needed for TLS, and I believe that so is x509_crt.c.

So far, the only module that I was able to disable without impacting the ability to establish TLS connection is MBEDTLS_SSL_SRV_C.

I tried to move the big tables into the ROM by adding __attribute__((section(".irom.literal"))):

static const oid_x520_attr_t oid_x520_attr_type[] __attribute__((section(".irom.literal")) =
{
    {
        { ADD_LEN( MBEDTLS_OID_AT_CN ),          "id-at-commonName",               "Common Name" },
        "CN",
    },
...
}

But this seems to have no affect on the size in topstrings (whereas commenting the entire struct content definitely reduces the size of the module in topstrings). Am I doing it right?

@MaBecker
Copy link
Contributor

MaBecker commented Mar 16, 2016

check object file with

xtensa-lx106-elf-objdump -h path/name_of_object_file.o

for a line with .irom.literal to make sure that it will be stored in flash

@wilberforce
Copy link
Member Author

wilberforce commented Mar 16, 2016

here is the definition of the macro:

#define FLASH_STR(name, x) static char name[] __attribute__((section(".irom.literal"))) __attribute__((aligned(4))) = x

#define FLASH_STR(name, x) static char name[] __attribute__((section(".irom.literal"))) __attribute__((aligned(4))) = x
It also aligns of 4 byte boundarys which is required to read from flash.

and here is example use in the simple case:

FLASH_STR(expect_cb, "Expecting callback function but got %v");

FLASH_STR(expect_cb, "Expecting callback function but got %v");
#define EXPECT_CB_EXCEPTION(jsCB) jsExceptionHere_flash(JSET_ERROR, expect_cb, jsCB)

I think in this case you are declaring arrays of strings, so this would need to be modified....
I don't see why your addition (with the aligned(4) would not work now for proof of concept...

Also here is using an array of strings, but there has got to be a better way!

// Reasons for which a connection failed

@wilberforce
Copy link
Member Author

wilberforce commented Mar 16, 2016

Looks like this is done a cunning way with arduino build:
http://www.esp8266.com/viewtopic.php?f=9&t=5446

lfaustini wrote:how can I push the WString.cpp.o into the .irom0.text section?
I believe that ESP8266-Arduino IDE uses a special linker script that puts many functions that are marked for the .text section into .irom0.text. The excerpt below comes from esp8266/Arduino on GitHub:
CODE: SELECT ALL
.irom0.text : ALIGN(4)
{
irom0_text_start = ABSOLUTE(.);
*core_esp8266
_.o(.literal_, .text_)
*spiffs_.o(.literal_, .text_)
.cpp.o(.literal, .text_)
libm.a:(.literal .text .literal._ .text.)
*libsmartconfig.a:(.literal .text .literal.
.text.
)
*(.irom0.literal .irom.literal .irom.text.literal .irom0.text .irom.text)
_irom0_text_end = ABSOLUTE(.);
_flash_code_end = ABSOLUTE(.);
} >irom0_0_seg :irom0_0_phdr

So this ld file moves stuff into flash, without having to use macros.

I think it works on functions too.... Looks promising..

https://github.com/esp8266/Arduino/blob/55e5bdfc6cb28fa79f94953f292f66b38804fc67/tools/sdk/ld/eagle.app.v6.common.ld

@gfwilliams
Copy link
Member

gfwilliams commented Mar 16, 2016

You could try it. I wonder if that's program code only though - He's talking about WString.o - not WStrings in general :)

The issue is that accesses to flash need to word-only and word-aligned. Just dumping a string into ROM will be fine until you go to access a character from it. Hence the macros.

Until the compiler is smart enough to auto-generate only word accesses for stuff in ROM, you will have no choice but to manually modify code.

@wilberforce
Copy link
Member Author

wilberforce commented Mar 27, 2016

@tve @gfwilliams

This technique installs an exception handler that catches the mis-aligned reads, and corrects them.

https://github.com/cesanta/smart.js/blob/master/smartjs/platforms/esp8266/user/esp_flash_bytes.c

It looks a lot simpler that having to copy all the ROM strings from iRom to ram before use...

@tve
Copy link
Contributor

tve commented Mar 27, 2016

What is the performance impact of this, though?

@wilberforce
Copy link
Member Author

wilberforce commented Mar 27, 2016

I have no idea - convenience vs performance?

@tve
Copy link
Contributor

tve commented Mar 28, 2016

well, the performance already isn't all that great. At some point it just sucks. You're gonna have to use some macro to put the strings into flash anyhow, is there that much to be gained by not having a macro that does the copy? How many bytes of strings are left? How many of those could possibly be moved to flash?

@wilberforce
Copy link
Member Author

wilberforce commented Mar 28, 2016

Top 20 from ./topstrings:
7250 ./espruino_esp8266_partial.o
765 ./src/jsinteractive.o
542 ./libs/network/socketserver.o
512 ./gen/jswrapper.o
438 ./libs/network/esp8266/jswrap_esp8266_network.o
387 ./libs/network/esp8266/ota.o
295 ./src/jslex.o
245 ./src/jsvar.o
225 ./src/jsparse.o
216 ./libs/network/telnet/jswrap_telnet.o
214 ./src/jswrap_process.o
207 ./targets/esp8266/esp8266_board_utils.o
207 ./src/jswrap_json.o
196 ./src/jswrap_io.o
196 ./libs/network/socketerrors.o
176 ./libs/network/esp8266/network_esp8266.o
151 ./targets/esp8266/user_main.o
146 ./src/jswrap_object.o
137 ./src/jswrap_espruino.o
132 ./src/jswrap_pipe.o

If the constants in jswrapper were aligned on 4 byte boundaries we could keep the reference tables in flash... That would save a heap of space".

The non- string tables might be any easy target, as they are generated - these are not showing in the top strings that you have kindly written.

@urish
Copy link
Contributor

urish commented Mar 28, 2016

we will probably also be able to get mbedTLS working on the ESP :-)

@gfwilliams
Copy link
Member

gfwilliams commented Mar 29, 2016

Assuming we could catch unaligned accesses, we could just copy all the const data to flash and remove the irom defines completely. It sounds very tempting.

the performance already isn't all that great

I guess it depends how you look at it. It's not great, so would people notice if it got a bit worse? :) The main complaint at the moment actually seems to be lack of RAM, which this might help with.

If someone has some free time/inclination, maybe they could try this just dump everything into ROM, removing all the ESP8266-specific flash copying/etc, and using that exception handler. The only way to know for sure is to do some benchmarks and see how much it hits performance.

If it's loads then I think it'd be a bad idea, but if it's a few percent I think it's a hit worth taking.

From my point of view, it'd be nice to remove the slightly iffy preprocessor stuff around print statements, and I think if you're printing/making error messages then you don't really care about performance either.

The one thing I think could hit performance is the symbol table, but then a few very simple ESP8266-specific tweaks in jswrapper.c could fix that.

Perhaps some of the mbedTLS stuff could end up being quite slow, but then if it's a choice of having it slow or not fitting it in at all, I think it's worth having it in.

@wilberforce
Copy link
Member Author

wilberforce commented Mar 29, 2016

@gfwilliams has merged the build with sha crypto for the esp8266 - so now the standard build will support web sockets:

#824

@wilberforce
Copy link
Member Author

wilberforce commented Apr 6, 2016

In aes.c there is a save-to-rom macro, however in the esp8266 case the tables still end up in ram.

There is still not enough heap space for the buffers, so unless more code is moved from ram to irom, TLS support for https can't work without reducing jsvars.

My initial aim of supporting web sockets via SHA1 is done, so closing issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ESP8266 This is only a problem on ESP8266 devices
Projects
None yet
Development

No branches or pull requests

5 participants