Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FritzBox 4040 bootloader image broken #1766

Open
RalfJung opened this issue Jun 24, 2019 · 35 comments

Comments

@RalfJung
Copy link
Contributor

commented Jun 24, 2019

Bug report

What is the problem?

A user is reporting the following problem:

Nach dem flash nach der einschlägigien Anleitung https://fritz-tools.readthedocs.io/de/latest/ quitiert die Fitzbox mit einem wiederholten doppelten blinken der Info LED in rot ihren Dienst.
Eine IP um ein den Config Mode zukommen vergibt sie auch nicht.

Translation: trying to boot the FritzBox with the Gluon bootloader fails, the info LED just double-blinks in red.

The same problem has also been reported in another community at https://forum.freifunk.net/t/installation-der-firmware-auf-fritzbox-4040-schlaegt-fehl/20624.

What is the expected behaviour?

Booting the FritzBox with the image should work.

Gluon Version:

2018.2.1

The bootloader from 2018.2 works fine, so as a work-around our users are now installing a previous firmware version and then doing a sysupgrade.

Site Configuration:

https://git.hacksaar.de/FreifunkSaar/gluon-site/blob/6b83c15caaabfe1159ea912882d05696681e5a32/site.conf

Custom patches:

None

@mweinelt

This comment has been minimized.

Copy link
Contributor

commented Jun 24, 2019

Thanks for your report.

Can you provide a serial log of the flash procedure? The theory is, that if we see a Red LED flashing, we will likely see a related error message in the serial log.

@RalfJung

This comment has been minimized.

Copy link
Contributor Author

commented Jun 24, 2019

I don't even own the device, sorry. I just forwarded this on behalf of a user in our community, and because I saw similar a report from another community. I will ask our user about giving you more feedback.

@belzebub40k

This comment has been minimized.

Copy link
Contributor

commented Jun 24, 2019

Between v2018.2 and v2018.2.1 there was one commit for uboot-4040.

91d3b87353 uboot-fritz4040: fix crash caused by interaction with gcc 7.1+

@mweinelt

This comment has been minimized.

Copy link
Contributor

commented Jun 24, 2019

I've just successfully tested the flash procedure for a firmware based on eed810a (v2018.2.1-7-geed810aa):

https://firmware.darmstadt.freifunk.net/images/1.6.1/other/gluon-ffda-1.6.1-avm-fritz-box-4040-bootloader.bin

Eva_AVM >.............................[tcp_check_ack] <wrong ack> 2049 0
.[tcp_check_ack] <wrong ack> 2251 0
................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Eva_AVM >
flash ..........................................................................<Reboot Device>

The Info-LED is continously red while the flash procedure is running.

@mweinelt

This comment has been minimized.

Copy link
Contributor

commented Jun 24, 2019

Master build based on 7d386a6 (v2018.2-157-g7d386a66).

https://firmware.darmstadt.freifunk.net/images/1.5~20190621/other/gluon-ffda-1.5~20190621-avm-fritz-box-4040-bootloader.bin

Eva_AVM >[autodetect_Handler] urlader Version error
[autodetect_Handler] urlader Version error
..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Eva_AVM >
flash ..........................................................................<Reboot Device>

I can't reproduce this with recent Gluon versions.

@mweinelt

This comment has been minimized.

Copy link
Contributor

commented Jun 24, 2019

The FRITZ!Box 4040 I'm testing with advertises it's bootloader as

(AVM) EVA Revision: 1.3243
(C) Copyright 2005 AVM Date: Dec 13 2016 Time: 16:13:32 (0) 3 0x0-0x240D

and it can't boot the image provided by Münsterland either

[FLASH:] MACRONIX Uniform-Flash 32MB 256 Bytes WriteBuffer
[FLASH:](Eraseregion [0] 512 sectors a 64kB) 
[SYSTEM:] CortexA9 
<create jffs2 from 0x1F00000 len 0x100000 jffs2_size 16>
Eva_AVM >
<ERROR: FIRMWARE_ILLEGAL_CONFIG>

https://firmware.freifunk-muensterland.de/domaene04/versions/v4.0.5/other/gluon-ffmsd04-v2018.2.1%2B4.0.5-avm-fritz-box-4040-bootloader.bin

@mweinelt mweinelt changed the title FritzBox 4040 bootloader image broken [regression] FritzBox 4040 bootloader image broken Jun 24, 2019

@mweinelt

This comment has been minimized.

Copy link
Contributor

commented Jun 24, 2019

[FLASH:] MACRONIX Uniform-Flash 32MB 256 Bytes WriteBuffer
[FLASH:](Eraseregion [0] 512 sectors a 64kB) 
[SYSTEM:] CortexA9 
<create jffs2 from 0x1F00000 len 0x100000 jffs2_size 16>
Eva_AVM >
<ERROR: FIRMWARE_ILLEGAL_CONFIG>

https://mgmt.saar.freifunk.net/firmware/experimental/other/gluon-ffsaar-1.7.3-avm-fritz-box-4040-bootloader.bin

@RalfJung

This comment has been minimized.

Copy link
Contributor Author

commented Jun 24, 2019

Ah right, I could have thought earlier of pointing you to our firmware images. ;)

Here is the diff of our site config and build script between the two versions. We did change the build invocations a bit, but the change looks all right to me and the fact that another community also has this problem makes me believe that's not it.

I will hopefully soon build a version based on Gluon 2018.2.2, which might be closer to the v2018.2.1-7-geed810aa you were testing.

@RalfJung

This comment has been minimized.

Copy link
Contributor Author

commented Jun 25, 2019

We now have a firmware based on Gluon 2018.2.2, the FritzBox bootloader is at https://mgmt.saar.freifunk.net/firmware/1.7.4~rc1/other/gluon-ffsaar-1.7.4~rc1-avm-fritz-box-4040-bootloader.bin.

@mweinelt

This comment has been minimized.

Copy link
Contributor

commented Jun 25, 2019

We now have a firmware based on Gluon 2018.2.2, the FritzBox bootloader is at https://mgmt.saar.freifunk.net/firmware/1.7.4~rc1/other/gluon-ffsaar-1.7.4~rc1-avm-fritz-box-4040-bootloader.bin.

FWIW: It's not the FRITZ!Box Bootloader, but the image that can be flashed via the FRITZ!Box Bootloader (EVA).

[FLASH:] MACRONIX Uniform-Flash 32MB 256 Bytes WriteBuffer
[FLASH:](Eraseregion [0] 512 sectors a 64kB) 
[SYSTEM:] CortexA9 
<create jffs2 from 0x1F00000 len 0x100000 jffs2_size 16>
Eva_AVM >
<ERROR: FIRMWARE_ILLEGAL_CONFIG>
@RalfJung

This comment has been minimized.

Copy link
Contributor Author

commented Jun 25, 2019

Well that's strange. None of the commits you mentioned you built yourself exactly coincides with the ones used to build these FW, but you got one on the 2018.2 branch that is between 2018.1.2 and 2018.2.2...

@mweinelt

This comment has been minimized.

Copy link
Contributor

commented Jun 25, 2019

So what we built for our latest stable firmware was actually this https://github.com/blocktrron/gluon/commits/ffda-1.6.1.

I just tested last nights master build and that works. Going to test v2018.2.2 next.

@RalfJung

This comment has been minimized.

Copy link
Contributor Author

commented Jun 25, 2019

Maybe it's something about the build environment? Compiler versions and so on? We are using the following docker file to generate ours:

FROM debian:stretch

RUN apt-get update
RUN apt-get install -y build-essential git subversion python2.7 gawk unzip libncurses5-dev zlib1g-dev libssl-dev wget cmake pkg-config curl ca-certificates bsdmainutils
RUN cd /tmp && wget https://git.universe-factory.net/libuecc/snapshot/libuecc-6.zip && unzip libuecc-6.zip && cd libuecc-6 && mkdir build && cd build && cmake -D CMAKE_BUILD_TYPE=RELEASE .. && make && make install && cd /tmp && rm -rf libuecc-6.zip libuecc-6 && ldconfig
RUN cd /tmp && wget https://github.com/tcatm/ecdsautils/archive/ab4eda463b4cdbb58136cec171a686fd11694c4e.zip && unzip ab4eda463b4cdbb58136cec171a686fd11694c4e.zip && cd ecdsautils-ab4eda463b4cdbb58136cec171a686fd11694c4e && mkdir build && cd build && cmake -D CMAKE_BUILD_TYPE=RELEASE .. && make && make install && cd /tmp && rm -rf ab4eda463b4cdbb58136cec171a686fd11694c4e.zip ecdsautils-ab4eda463b4cdbb58136cec171a686fd11694c4e

RUN useradd ci
USER ci

It's been a while since we last re-generated it though, we are using versions of everything from 2018-04-27. But I doubt Debian stable has moved that much since then.

@mweinelt

This comment has been minimized.

Copy link
Contributor

commented Jun 25, 2019

I'm currently compiling our images on a Debian Buster machine. Maybe try the Dockerfile given in contrib/Dockerfile instead? It doesn't have ecdsautils, but that can be installed from apt in buster.

@RalfJung

This comment has been minimized.

Copy link
Contributor Author

commented Jun 25, 2019

I'd rather not build on a not-yet-stable distribution. But thanks for pointing out that Dockerfile, we might switch to it once buster is released!

It doesn't have ecdsautils,

It does, though?

@mweinelt

This comment has been minimized.

Copy link
Contributor

commented Jun 25, 2019

Buster is already in full-freeze since march, so I wouldn't worry too much.

https://release.debian.org/buster/freeze_policy.html

Edit: fixed typo in my last message x(

@RalfJung

This comment has been minimized.

Copy link
Contributor Author

commented Jun 25, 2019

I'm aware of the Debian freeze process, but still I prefer to wait for a release.

@RalfJung

This comment has been minimized.

Copy link
Contributor Author

commented Jun 25, 2019

So I guess the working hypothesis is that something between Gluon 2018.2.1 and 2018.2.2 introduced a regression only when using GCC 7 GCC 6 (or some other part of the toolchain that changed between stretch and buster)?

@mweinelt

This comment has been minimized.

Copy link
Contributor

commented Jun 25, 2019

I pulled in this change at the end of january 2019 when updating the OpenWrt reference (fcb08ea).

91d3b87353 uboot-fritz4040: fix crash caused by interaction with gcc 7.1+

openwrt/openwrt@91d3b87

@RalfJung

This comment has been minimized.

Copy link
Contributor Author

commented Jun 25, 2019

Actually stretch is on GCC 6.3. So maybe that patch broke older compilers?

@mweinelt

This comment has been minimized.

Copy link
Contributor

commented Jun 25, 2019

Just built v2018.2.2 using the Gluon Docker container and the resulting image works.

@rotanid

This comment has been minimized.

Copy link
Member

commented Jun 25, 2019

next up: someone tries the same dockerfile but with stretch ;)

@mweinelt

This comment has been minimized.

Copy link
Contributor

commented Jun 26, 2019

Yes, I can confirm that building on Debian Stretch with GCC 6.3 results in <ERROR: FIRMWARE_ILLEGAL_CONFIG>.

@mweinelt

This comment has been minimized.

Copy link
Contributor

commented Jun 26, 2019

If someone else wants to reproduce this, I used this environment:

FROM debian:stretch-slim

RUN apt update && apt install -y --no-install-recommends \
    ca-certificates \
    file \
    git \
    subversion \
    python \
    build-essential \
    gawk \
    unzip \
    libncurses5-dev \
    zlib1g-dev \
    libssl-dev \
    libelf-dev \
    wget \
    time \
    lua-check \
  && rm -rf /var/lib/apt/lists/*

RUN useradd -d /gluon gluon
USER gluon

VOLUME /gluon
WORKDIR /gluon
@Adorfer

This comment has been minimized.

Copy link
Contributor

commented Jun 26, 2019

just for my understanding (and perhaps until it's fixed here or upstream):
which versions of gcc are now considering as OK for building FB4040-bootloader?
(Just in order to be on the safe side when creating images today)

@kevin-olbrich

This comment has been minimized.

Copy link
Contributor

commented Jun 26, 2019

Maybe OT:
For my understanding: Until I saw this issue, I always thought that OpenWRT builds it's toolchain for every target - for example:

.../gluon/openwrt/build_dir/toolchain-mipsel_24kc_gcc-7.3.0_musl
image

This issue will lead to a lot of issues because stretch uses GCC 6.3 while Fedora 30 is at v9.1.

If the system GCC is used, whats the reason for building GCC then? Compiling the compiler?

@NeoRaider

This comment has been minimized.

Copy link
Member

commented Jun 26, 2019

The system GCC can't directly affect the bootloader, as the bootloader is built using the target GCC. The system GCC could affect the build in two ways:

  • Miscompilation of target GCC
  • Miscompilation of host utilities (for example firmware image build tools that run during the build)
@rotanid

This comment has been minimized.

Copy link
Member

commented Jun 28, 2019

i can't reproduce this issue, it's working for us.
after having no problems with our regular builds on Debian Stretch, i took an empty folder and built completely from scratch and without any patches, only the Gluon tag v2018.2.2.
Even with this build there was no issue with flashing it to a fresh 4040, it worked on the first try.

@rotanid rotanid added this to the 2019.1 milestone Jun 29, 2019

@MPW1412

This comment has been minimized.

Copy link
Contributor

commented Jul 4, 2019

Our broken images (Gluon v2018.2.1) here at Freifunk Münsterland where build on a Hetzner Cloud machine (CX51). So the OS was Ubuntu 18.04. The server has been deleted after the build was finished unfortunately.

The confirmed working images(Gluon v2018.2) were build on Ubuntu 16.04 though.

So it might have something to do with the OS they were build on and or the Gluon version.

@rotanid

This comment has been minimized.

Copy link
Member

commented Jul 4, 2019

So someone needs to do a new build completely from scratch and without custom patches on another hosting platform than Hetzner but with Ubuntu 18.04.
With full debug logging like described in https://github.com/freifunk-gluon/gluon/wiki/Troubleshooting so we can search the logs.

@rotanid

This comment has been minimized.

Copy link
Member

commented Jul 4, 2019

my own build of v2018.2.2 on Ubuntu 18.04 (docker image) works. so it's not only the OS.

@NeoRaider suggested on IRC that there is something wrong with the padding in some cases because of issues with https://github.com/chunkeey/FritzBox-4040-UBOOT/blob/master/fritz/fritzcreator.sh

@RalfJung

This comment has been minimized.

Copy link
Contributor Author

commented Jul 4, 2019

Our confirmed working and confirmed broken images were both built in the same Debian Stretch Docker container (linked above). The only difference is the Gluon version.

I can do another build of that and provide logs if that's helpful, but it would not be Ubuntu.

@rotanid

This comment has been minimized.

Copy link
Member

commented Jul 4, 2019

if you can reproduce broken images, that's great.
let's wait with further testing for an updated fritzcreator

@RalfJung

This comment has been minimized.

Copy link
Contributor Author

commented Jul 4, 2019

Our 1.7.3 and 1.7.4~rc1 were both broken, so it looks fairly reproducible. Someone could test 1.7.4~rc2 and 1.7.4 to confirm.

@NeoRaider

This comment has been minimized.

Copy link
Member

commented Jul 20, 2019

I've pushed a potential fix to the branch wip/fritz4040, please test.

@mweinelt mweinelt referenced this issue Aug 18, 2019
2 of 4 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.