esp8266 ScratchABit error

[sheinz]

Hi @pfalcon,

I'm trying to get ScratchABit + ida-xtensa2 to work for esp8266 disassembly and having the following error:

> python3 scratchabit.py ../esp8266.def
Loading CPU plugin xtensa
Loading /Volumes/esp8266/esp-open-rtos/examples/http_get/build/http_get.out (elf plugin)
Adding area: Area(0x40000000-0x4000ffff, {'name': '.bin', 'access': 'RX'})
Loading /Volumes/esp8266/esp-elf-rom/bootrom.bin @0x40000000
Processing section: entrypoints
Traceback (most recent call last):00
  File "scratchabit.py", line 950, in <module>
    engine.analyze(_progress)
  File "/Volumes/esp8266/ScratchABit/engine.py", line 861, in analyze
    insn_sz = _processor.ana()
  File "/Volumes/esp8266/ScratchABit/plugins/cpu/xtensa.py", line 460, in ana
    val = get_full_val(ea, 4)
  File "/Volumes/esp8266/ScratchABit/idaapi.py", line 293, in get_full_val
    return ADDRESS_SPACE.get_data(ea, val_sz)
  File "/Volumes/esp8266/ScratchABit/engine.py", line 221, in get_data
    val = val | (area[BYTES][off + i] << 8 * i)
TypeError: 'NoneType' object is not subscriptable

Here's my configuration file:

cpu xtensa

# Show up to this many raw bytes of code/data
show bytes 4

# Load an arbitrary esp8266 ELF file
# See example.def if you want to load raw binaries instead
load /Volumes/esp8266/esp-open-rtos/examples/http_get/build/http_get.out elf

# First attempt at loading BootROM binary image and symbols
# These files came from https://github.com/jcmvbkbc/esp-elf-rom
area .bin 0x40000000(0x10000) rx
load /Volumes/esp8266/esp-elf-rom/bootrom.bin 0x40000000
[entrypoints]
load "/Volumes/esp8266/esp-elf-rom/eagle.rom.addr.v6.ld"

Python version 3.5.2
Is there something wrong with my configuration?
Thanks

[pfalcon]

Well, this is definitely not ida-xtensa2 bug, but ScratchABit's ;-). (But nevermind)

So, some unexpected condition happens when loading http_get.out file. It's fully dependent on it that file, so I won't be able to tell much without it. Feel free to attach it to this ticket if you're interested in me looking into it.

In the meantime, did you try examples included with ScratchABit, as described in its README (those are x86/x86_64, right)? And as a next step, did you try https://github.com/pfalcon/xtensa-subjects/tree/master/2.0.0-p20160809 , which is known good (and pretty big for a test) Xtensa case?

[sheinz]

Here is the file: http_get.tar.gz

I tried the included example for x86 and it works fine.

Also https://github.com/pfalcon/xtensa-subjects/tree/master/2.0.0-p20160809 loads fine.

I tried to use configuration from xtensa-subjects for my file:

cpu xtensa
show bytes 4

load /Volumes/esp8266/esp-open-rtos/examples/http_get/build/http_get.out elf

area .sysram 0x3fffc000-0x3fffffff rw
area .bootrom 0x40000000(0x10000) rx

# Hardware registers
area .dport0 0x3ff0_0000(0x100)   rw
area .wdev   0x3ff2_0000(0x10000) rw
area .periph 0x6000_0000(0x10000) rw

load /Volumes/esp8266/xtensa-subjects/bootrom/bootrom.bin 0x40000000
[entrypoints]
load "/Volumes/esp8266/xtensa-subjects/bootrom/eagle.rom.addr.v6.ld"

And get different error:

Loading CPU plugin xtensa
Loading /Volumes/esp8266/esp-open-rtos/examples/http_get/build/http_get.out (elf plugin)
Adding area: Area(0x3fffc000-0x3fffffff, {'access': 'RW', 'name': '.sysram'})
Adding area: Area(0x40000000-0x4000ffff, {'access': 'RX', 'name': '.bootrom'})
Adding area: Area(0x3ff00000-0x3ff000ff, {'access': 'RW', 'name': '.dport0'})
Adding area: Area(0x3ff20000-0x3ff2ffff, {'access': 'RW', 'name': '.wdev'})
Adding area: Area(0x60000000-0x6000ffff, {'access': 'RW', 'name': '.periph'})
Loading /Volumes/esp8266/xtensa-subjects/bootrom/bootrom.bin @0x40000000
Processing section: entrypoints
Traceback (most recent call last):00
  File "./scratchabit.py", line 950, in <module>
    engine.analyze(_progress)
  File "/Volumes/esp8266/ScratchABit/engine.py", line 843, in analyze
    assert fl in (ADDRESS_SPACE.CODE, ADDRESS_SPACE.UNK)
AssertionError

[pfalcon]

Ok, so the problem is funkily code-generated noreturn functions, and the fact that ScratchABit isn't (yet? at all?) well designed for not fully valid code. See the disassembly with assertion disabled:

│40100e68          fatal_exception_handler:                                                                                            │
│40100e68 12c1e0       $sp -= 0x20                                                                                                     │
│40100e6b c961         *(u32*)($sp + 0x18) = $a12                                                                                      │
│40100e6d 2901         *(u32*)$sp = $a2                                                                                                │
│40100e6f 30c074       $a12 = bitfield($a3, /*lsb*/0, /*sz*/8)                                                                         │
│40100e72 026107       *(u32*)($sp + 0x1c) = $a0                                                                                       │
│40100e75 05faff       call fatal_handler_prelude                                                                                      │
│40100e78 31fbff       $a3 = 0x401010d8                                                                                                │
│40100e7b 2801         $a2 = *(u32*)$sp                                                                                                │
│40100e7d 4803         $a4 = *(u32*)$a3                                                                                                │
│40100e7f 3d0c         $a3 = $a12                                                                                                      │
│40100e81 c00400       call $a4                                                                                                        │
│40100e81          ; End of function 'fatal_exception_handler' (as set by loader (detected: 0x40100ea7))                               │
│40100e84                       ; xref: r 0x40100ea2 (abort+0x1a)                                                                      │
│40100e84          dat_40100e84:                                                                                                       │
│40100e84 90b82040     dd      0x4020b890                                                                                              │
│40100e88          ; Start of function 'abort'                                                                                         │
│40100e88          abort:                                                                                                              │
│40100e88          ; UNEXPECTED value: 12 flags: 03                                                                                    │
│40100e89          ; UNEXPECTED value: c1 flags: 02                                                                                    │
│40100e8a          ; UNEXPECTED value: f0 flags: 03                                                                                    │
│40100e8b          ; UNEXPECTED value: 09 flags: 02                                                                                    │
│40100e8c          ; UNEXPECTED value: 31 flags: 02                                                                                    │
│40100e8d c921         *(u32*)($sp + 0x8) = $a12                             

So __attribute__((noreturn)) function ends with a dangling call, instead of a more natural jump. A leisurely question to @jcmvbkbc : was that ever fixed in a newer gcc version, I remember (if not mixing up) that gcc-xtensa (4.x) at all has issues with optimizing tail calls. @sheinz : Can you confirm this was built with gcc 4.x? Nevermind, I see it in your ELF: /Volumes/case-sensitive/esp-open-sdk/crosstool-NG/.build/src/gcc-4.8.2/libgcc.

In the meantime, I'm definitely looking into trying to alleviate this issue in ScratchABit.

[pfalcon]

gcc-4.8.2

Btw, even esp-open-sdk now (half year at least) uses 4.8.5. And as I mentioned to @projectgus long ago, esp-open-sdk's role is very simple: provide a known-good toolchain for novices or busy people who don't know how to build toolchain of or don't want to bring more instability (on gcc version side) to otherwise already very unstable esp8266 environment. Advanced users should use gcc-xtensa from upstream crosstool-NG (and work with @jcmvbkbc to test whatever isn't yet upstream and provide motivation for him to upstream it).

This is of course all offtopic wrt to ScratchABit breakage.

[jcmvbkbc]

was that ever fixed in a newer gcc version, I remember (if not mixing up) that gcc-xtensa (4.x) at all has issues with optimizing tail calls

I'm pretty sure that no gcc version does tail call optimization for xtensa, for windowed ABI due to the calling convention, and for call0 because I didn't think about it when I did the port, so it inherited windowed ABI behavior. But it should be possible with call0, I'll take a look at it.

[pfalcon]

But it should be possible with call0, I'll take a look at it.

Thanks for the prompt reply, as usual. ;-). At your convenience of course, I don't think that anybody in hurry on it.

[pfalcon]

Ok, so back to the subject of the reported issue - the underlying cause is described pfalcon/ScratchABit#14 , it's something which I was afraid might bite some time, thanks for providing a testcase for that. So I by now have a change which make http_get.out to be properly disassembled, but it regressed function boundary detection functionality, so need more work yet.

[sheinz]

Thanks for such a quick response.

[pfalcon]

@sheinz : Sorry that it took extra time to finish it. Current ScratchABit master with current ida-xtensa2 master (both need to be updated) works on your http_get example. If you look further into it, feel free to report any issues you see, I'm set to make ScratchABit reliably working for various usages.