Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not working on U-Boot 2017.07-RELEASE-g78ed34f31579 (Sep 29 2017 - 07:43:44 -0700) #13

Open
0xFelix opened this issue Dec 25, 2018 · 246 comments

Comments

@0xFelix
Copy link

0xFelix commented Dec 25, 2018

Seems like Cisco blocks the old method of getting a serial prompt in this uboot version?

The ubootwrite.py script is sending xyzzy, but no prompt appears.

Maybe CONFIG_AUTOBOOT_STOP_STR was changed?

Where can one obtain the latest GPL sources of Meraki's uboot?

@0xFelix
Copy link
Author

0xFelix commented Dec 25, 2018

Update:

I tried pressing space instead on boot...
Resulted in the device saying:

Secure boot NOT enabled! Blowing fuses... Resetting now.

It is dead now, won't boot and the orange LED stays lit.

Maybe Cisco is not so keen on replacing this device's firmware afterall.

@Ordspilleren
Copy link

Looks like this is now reflected in the flashing documentation in Google Drive. Is there any chance this will be solved, and is there something we can do to help?

@riptidewave93
Copy link
Owner

@Ordspilleren best thing to do at this point is to request the latest u-boot GPL source from Meraki. This should hopefully show us what Blowing fuses... Resetting now. actually does to the device.

@Ordspilleren
Copy link

I got the U-boot source from Meraki. From at quick look at the code, the whole Blowing fuses... Resetting now. ordeal seems to be related to the Qualcomm QFPROM. I am not qualified enough to figure out much more, or the solution for that matter.
Here's a link for the source: https://drive.google.com/file/d/1fIs0rZI2a6UOqal6JS4nxI-Ld4co0O_B/view

@argentdeux
Copy link

My Meraki licence expires this year, I wait hopeful that the 2017.07 version of U-Boot is able to be flashed before then! Unfortunately this is beyond my ability.

@Flole998
Copy link

Looks like what you're doing there is enabling secureboot. xyzzy should still work (with secureboot disabled) though, at least it seems like that in the code/config, I only had a brief look at the source though.

Some fuses are write once and can never be deactivated (at least not in software), I don't know if that fuse is one of those.

@0xFelix
Copy link
Author

0xFelix commented Mar 20, 2019

In the 20170427 u-boot sources the file include/configs/ipq40xx_cdp.h contains CONFIG_AUTOBOOT_STOP_STR set to xyzzy and in the newer u-boot sources of @Ordspilleren CONFIG_AUTOBOOT_STOP_STR is completely absent. So I guess they disabled it on purpose.

Instead the newer ipq40xx_cdp.h file defines new u-boot commands including boot_meraki_qca. This function forces secure boot and also "blew" the fuse on my device. (Looking at the code this should make no difference and it should already have been blown on my device, maybe for devices upgrading from older u-boot versions?) Devices with 20170427 u-boot seemingly do not use secure boot and are therefore able to run arbitrary code while devices with the newer u-boot only run signed code?

@Richy78
Copy link

Richy78 commented Apr 30, 2019

any news on the issue? I'm stuck on"uploading image" no prompt...

@Flole998
Copy link

If it's really an efuse then you blow it once and that's it, no way to recover.

What you can always do when secureboot is inactive is desolder the flash and replace the uboot.

@ajkenah
Copy link

ajkenah commented Aug 10, 2019

I found this document on Secure boot :https://www.qualcomm.com/media/documents/files/secure-boot-and-image-authentication-technical-overview.pdf
Looks like secure boot has a efuse location for a root certificate hash to validate boot images. When you fail to boot a secure image it blows the fuses with the root cert.. or thats how i read the above.
Not sure if Cisco even uses it though as it seems to mention that you can store certificates in ROM as well.

@ajkenah
Copy link

ajkenah commented Aug 10, 2019

Has anyone managed to JTAG the MR33? I've got a couple of fresh ones and I would like to backup the firmware so that I can "roll-back" the Meraki bootloader later when it gets updated.. I suspect that the pinout is the 10x2 holes on the side, but I'm getting weird readings on my multi-meter and the traces don't seem to line up whats specified in the standard ARM 20-pin JTAG Layout...

@satis4action
Copy link

Guys, any news?
My version is: U-Boot 2017.07-RELEASE-g78ed34f31579 (Sep 29 2017 - 07:43:44 -0700)

@Flole998
Copy link

What did you do so far? Already pressed space and seen Secure boot NOT enabled! Blowing fuses... Resetting now.? In that case its already bricked, most likely forever. If you havent seen that yet the easiest way is most likely to replace uboot.

@shanehull
Copy link

any news on the issue? I'm stuck on"uploading image" no prompt...

Did you work this out?

I have the same issue. Verbose mode gives me "Waiting for prompt.." then nothing.

I've tried setting the baud rate, data bits and stop bit, but still nothing. Have verified that I have a working serial connection in/out.

@ajkenah
Copy link

ajkenah commented Nov 6, 2019

I never got this working. I blew two rasperryPi's and then the AP itself trying to JTAG in and dump the memory so that I could flash it back if necessacary... Not that it means it doesn't work, jus tthat im not very good at soldering and double checking cable pinouts :(

@owenashurst
Copy link

Any update on this?

@Flole998
Copy link

Flole998 commented Dec 1, 2019

The method I mentioned above still works. Desolder flash, replace uboot, solder it on again.

@owenashurst
Copy link

owenashurst commented Dec 1, 2019

The method I mentioned above still works. Desolder flash, replace uboot, solder it on again.

@Flole998 are there any instructions on this? I've never flashed a NAND flash chip before, removing it should be ok, I have a heat gun but only really done this many years ago. Also, is it the chip under the tape?

@Flole998
Copy link

Flole998 commented Dec 1, 2019

There are datasheets available that tell you everything you need to know about that flash. The basic steps are dump, seperate OOB/ECC, modify, calculate new ECC and write back.

What might work aswell is to short the flash when uboot tries to load the kernel, depending on it's configuration it might fall back to a console.

@kitor
Copy link

kitor commented Dec 1, 2019

Was this tested anywhere?

As I'm not familiar with how wear leveling, etc works. But I have two MR33 - one running OpenWRT, one on stock software (fortunately still 2y of license to wait for any developments).

I thought about booting modded one into initrd and then hot-swapping flash and reflashing it from linux - would this work? (let's skip physical part of swapping flash in the answer)

If I understand correctly, in this kind of environment software (uboot/linux kernel) is responsible for all the leveling/ecc stuff, not memory controller like on flash drives?

@Flole998
Copy link

Flole998 commented Dec 1, 2019

Actually I've done it before, it's been quite some time ago though. I'm not sure if someone else has done it or if this is documented somewhere.

I've heard of hot-swapping flash causing a brick before, however that was on a different device.

What could work is cloning the other NAND if you don't want to do the modifications manually, if it contains calibration data though that would need to be written back afterwards.

@owenashurst
Copy link

owenashurst commented Dec 1, 2019

The thing is, I've waited until the boot process finishes and I accidentally tried to exit out of screen using CTRL+C, and this gave me a Meraki> prompt. Now I don't know at which step that it causes the fuse to blow. Is it once it reboots with the uploaded openwrt u-boot and realises it's not signed or something? Because I've replicated what I've done on the same device over several reboots and I always get the prompt.

Also, I took a full back up when I successfully flashed another MR33 yesterday of MTD. Is this useful for the MR33 with the newer firmware?

@satis4action
Copy link

satis4action commented Dec 9, 2019

@Flole998
sorry to bothering you, can I use mr33-uboo.bin from this source? https://drive.google.com/drive/folders/1jJa8LzYnY830v3nBZdOgAk0YQK6OdbSS
address should be ?
0x000000700000-0x000000900000 : "u-boot"
or gerneric processes can be the same?

1.Create a full dump of the SPI Flash, and store it in a safe place
2.Erase and Clear 0x0-0x3ffff on the SPI Flash
3.Flash U-Boot to 0x0
4.Proceed to the OpenWRT Install Procedure

referrer : https://openwrt.org/toh/aruba/aruba_ap-105

@Flole998
Copy link

The basic process is the same, yes. Just that this is not a SPI Flash and the address is obviously different. But dumping, replacing and writing back are the correct steps, just that in this case I desoldered the Flash because I don't want the CPU to interfere with my programming operation.

@owenashurst
Copy link

owenashurst commented Jan 15, 2020 via email

@satis4action
Copy link

connect terminal to uart port and you must see on the begining of the boot process, like this

"U-Boot 2012.07-g97ab7f1 [local,local] (Oct 06 2016 - 13:07:25)"
...

39400208-4807a69e-4b2c-11e8-8002-42f6f78b2ab6

@kechie
Copy link

kechie commented Jan 18, 2020

If you observe the boot process when you're connected via UART, it will tell you

On Wed, 15 Jan 2020, 12:27 Richard L. Alhama, @.***> wrote: hi how would you determine

I have U-Boot 2017.07-RELEASE-g78ed34f31579 (Sep 29 2017). Guess I have to hone my soldering skills then.

Thanks

@cryptage21
Copy link

Hi guys,

Is there a way to know the bootloader version (without opening) by booting the MR33 and checking firmware version by example ?

Thanks

@urbaniak
Copy link

I've tried even unsoldering the flash chip, but without success - looks like it's glued and then soldered.

So the only idea I've got is clamp on tsop-48 (https://www.aliexpress.com/item/32838230005.html?spm=a2g0s.9042311.0.0.27424c4dsE1VcC) and then writing old uboot.

@Flole998
Copy link

If it's glued down and they didn't put too much glue on it you can just remove it with a heatgun.

The problem with those clamps is that the CPU must be held in reset and the pins for the flash must not be pulled in either direction (low or high), otherwise the CPU will interfere and you will get weird results.

@Flole998
Copy link

Well you can also desolder and re-solder the memory, what's cheaper depends on how much your time is worth. If that's "nothing" them you can definitely spend an hour desoldering and resoldering that part. You should also factor in the risk of breaking the device when figuring out what's the cheapest method.

@sijans
Copy link

sijans commented May 28, 2023

I was able to replace the uboot with the old 2012 version with a Raspberry Pi! For instructions and code see https://github.com/sijans/rpi-tsop48-nand

sorry, wait... how can you do this without desoldering the nand? have you solder +12 cable to the nand pinouts??? how? i would like to test this but the nand pinuouts are too thiny and too close together.

@CRAWLER-V
The clip is the best solution, but too expensive for an experiment and only one use.
I soldered directly to the NAND using a thin tip and enamel wire. Was not that difficult for me.

nand

@CRAWLER-V
Copy link

this is insane... very good to you.
Have you a jpg/png about NAND pinout to solder with?
i get the image for raspberry pinout, missing the NAND pinout...

curious to try this job.

@sijans
Copy link

sijans commented May 31, 2023

Have you a jpg/png about NAND pinout to solder with? i get the image for raspberry pinout, missing the NAND pinout...

#13 (comment) should help you with the pinout.

@CRAWLER-V
Copy link

thanks.
now what i'm missing is what OS was mounted into raspberry? raspbian? a ubuntu/linux lite version?
And how i send commands to read/write nand? using a simple terminal console?

i'm talking about those commands:

Read full flash
rpi-tsop48-nand-b3 150 read_full 0 65536 mr33_full.dmp

Add padding for flashing
dd if=mr33_full.dmp bs=$((0x738000)) count=1 > newflash-mr33.bin
dd if=ubootmr332012.bin bs=132k count=5 >> newflash-mr33.bin

Erase blocks
rpi-tsop48-nand-b3 150 erase_blocks 56 5

Flash "new" bootloader
rpi-tsop48-nand-b3 150 write_full 3584 320 newflash-mr33.bin

Can i run those via terminal into a linux based OS for raspberry?

@sijans
Copy link

sijans commented Jun 1, 2023

You can use the official Linux Raspi OS, lite is sufficient as you only need the console. Compile rpi-tsop48-nand.cpp, the resulting executable is the program you use to read/write the NAND with the commands you posted.
See https://www.psx-place.com/threads/interested-in-a-rpi0-based-aio-chip-option.21558/#post-149868 for some more explanation.

@CRAWLER-V
Copy link

thanks, may i asking too much, can someone make a scheme of pinout to solder in the back of the mainboard? the pin on-chip are too too close together, i cannot solder there... but in the back there are electric pitches that could help to solder maybe.

@Leo-PL
Copy link

Leo-PL commented Jun 5, 2023

No luck here for you, most of the tracks end up under a BGA chip. I ended up desoldering chip with hot air, and using the cheap (~12$) ZIF socket adapter for TSOP48 that I had after previous project.

@exetico
Copy link

exetico commented Jun 5, 2023

Will this one fix the chip, if desoldered? 🤔
https://a.aliexpress.com/_mKf9hOG

@Leo-PL
Copy link

Leo-PL commented Jun 5, 2023

Should be fine.

@Leo-PL
Copy link

Leo-PL commented Jun 21, 2023

@Flole998 Yeah, this would definitely be the next step, but I'd like to prevent inexperienced people from softbricking their devices. Not to mention that converting MR33 isn't for the faint-hearted from the start.

I'm not even sure if it's still broken, maybe there was some kernel update that broke it and another one fixed it again, I wouldn't rule that out.

I had a chance yesterday to flash an image which had a bit above 4MB kernel, resulting in soft-brick of two units. Currently 23.05-rc1 and even 22.03.2 have kernels greater than that, 23.05 of course being bigger.

I wonder if it's possible to use lzma-loader on ipq40xx. but I'm pretty sure, that MR33 images need to be disabled for time being. @chunkeey how do you think?
Either way, I'll submit a patch to disable the image generation, and then investigate how to actually fix the issue.

Also, does anyone with MR74 know if it works? It seems to have different U-boot build, which also doesn't have the Cisco trap for tinkerers.

@Flole998
Copy link

I don't think it's related to size, I've removed stuff from the image so it's smaller but it still didn't boot.

@Leo-PL
Copy link

Leo-PL commented Jun 21, 2023

It is strictly the kernel size, not just the rest of image.

@Flole998
Copy link

I did reduce the kernel size, i believe i removed USB support and also emmc support.

@Leo-PL
Copy link

Leo-PL commented Jun 21, 2023

By any chance, do you have any boot log of such image? My devices froze right after "Starting kernel", showing no further activity.

Regardless of the cause, the issue is still there - I believe the images should be disabled until the root cause is at least known.

@Flole998
Copy link

I'm seeing exactly the same thing, no further output.

@Leo-PL
Copy link

Leo-PL commented Jun 21, 2023

Could you do tar tvf on your shrinkened sysupgrade image?
I can try to start analyzing the issue with more depth after the weekend - I need my units working until then.

@Flole998
Copy link

I no longer have those images, I've deleted them since I considered them broken

@sijans
Copy link

sijans commented Jun 21, 2023

What do you think is the limit for the kernel size?
I'm currently running an image with 4221132 bytes kernel size, even a snapshot release with 4442372 bytes was working fine a few weeks ago. But I had the issue with a 4200157 bytes kernel.

@Flole998
Copy link

That's basically consistent with what I was seeing, some builds were working, others weren't, the size doesn't seem to matter.

@Leo-PL
Copy link

Leo-PL commented Jun 21, 2023

@sijans do you know from which lines were the working and non-working kernels? i.e. 5.10 or 5.15? Maybe there is a regression blocking boot.

@sijans
Copy link

sijans commented Jun 21, 2023

Both 5.15. 5.15.111 was not working, 5.15.113 boots fine, but I need to check what is installed currently. Maybe even a newer version. Edit: It's a gluon build with 5.15.111 that runs fine again.

@Leo-PL
Copy link

Leo-PL commented Jun 25, 2023

Now I wonder how to use the Meraki's dual boot scheme with OpenWrt, to get some kind of recovery in case the boot fail. I'd be grateful for any documentation. Going to dig through U-boot sources to get some information on that, because debricking the unit is royal PITA.
Edit; Just checked the GPL dump for 2021.07 release and it has no fall-back mechanism present - just as if this all was an afterthought. Though, analyzing default U-boot environment from the binary itself may help.

@sijans
Copy link

sijans commented Jun 26, 2023

@Leo-PL You need access to the serial console and have an initramfs image in the second partition "part.old".

If you enter
STINKBUG # run set_ubi
STINKBUG # run bootkernel2
you can boot the second partition and then write a new image to the first partition with sysupgrade.

Default boot command is meraki_boot=run set_ubi; run bootkernel1; run bootkernel2, I think it will only fallback if bootkernel1 is really corrupt and not freezing.

@Leo-PL
Copy link

Leo-PL commented Jul 16, 2023

I made some progress investigating the booting issue here: openwrt/openwrt#12953
TL;DR the build system seems non-deterministic, I was able to get booting and non-booting images on the same commit, with minor differences inside the kernel binary, after unpacking. I can upload the images somewhere, for further analysis. I need to compare expanded kernel configurations before going with Ghidra.

@halmartin
Copy link

I had to flash an MR33 recently, and since python2 is no longer packaged for many distributions, I migrated ubootwrite.py to python3.

Fork is here: https://github.com/halmartin/mr33-openwrt-flash

Tested on Arch Linux with python 3.11.5, no warranty provided 😄

@joshiewtf
Copy link

Does anyone have a 2017 uboot dump? I want to look into a file and see whether i can find anything that can be exploited into writing, or at least getting shell access without bricking the board.

@halmartin
Copy link

halmartin commented Apr 20, 2024

Does anyone have a 2017 uboot dump? I want to look into a file and see whether i can find anything that can be exploited into writing, or at least getting shell access without bricking the board.

You don't need a dump, the source code for 2017.07 is available.

You can find the function that enables secure boot, bricking the device, right here: https://github.com/riptidewave93/meraki-uboot/blob/mr33-20190225/board/qca/arm/common/meraki_rel_boot.c#L107-L135

You can dig through the u-boot source code for vulns, but I hope you have a large stack of devices you don't mind bricking if you plan to test it...

@rileyg98
Copy link

rileyg98 commented Apr 21, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests