Reset ASPEED 2500 on Gigabyte possible? #51
Comments
|
Hello. Firstly, I'm sorry for your situation, it sounds like a bit of a pickle. Working backwards, I have a few points:
Really, the u-boot output you posted at the start is the meaty bit: This is interesting on a couple of fronts.
So a few thoughts:
The interest here is that LPC and eSPI are both buses that connect the BMC to the host, and that the host BIOS recovery state may in some way be interfering with the BMC start up (... which is pretty unfortunate if true). There are some experiments that might be worth trying, but I'd want more information on the state of the machine prior to making some guesses. |
|
+1 to all the above from @amboar, though a couple additional things:
What mechanism did you use for this firmware flashing? If it was something other than culvert I might suggest retrying it via culvert, because its firmware write path also reads the data back and checks it against the input, so if it was via another flashing mechanism that doesn't do similar checking and the flash write actually got borked somehow it could potentially have gone undetected... And on a related note, what version of culvert are you running? Until fairly recently there were some problems with it that could lead to flash corruption in some cases (see #45 for details), so I'd recommend using the latest commit if you're not already. |
|
Hey thanks for the quick follow up! I'm not home yet so I can't test for a few more hours, but here's what I can answer for now;
I'll run the commands you suggested later tonight and report back. Ty! |
It's not clear to me exactly what your power setup is, but if turning off your PDU removes AC supply to the PSU for the board then that gives me enough. A common design for boards with BMCs is that the BMCs are powered even if the host is not, so just double-checking here.
Okay, interesting. Honestly it sounds like a bit of a QA fail, and returning the board seems like a reasonable course of action. The firmware should behave better than that. |
|
Alright I just connected to the server and here's what I got when I read the address you suggested: And yeah, turning off the PDU does cut power to the PSU entirely and all the mobo's LEDs turn off.
I completely agree, but I see this as a good learning opportunity for me and I'm a bit impatient to try out my new server so if possible I'd rather avoid returning the mobo 😄. If there's anything I can do to help make sense of the current situation let me know. I manage Linux servers on a daily basis so while embedded systems are not my forte I hope to be able to make sense of this given a bit of guidance. |
|
Okay, so It's a bit curious. I wonder why the eSPI handhake message is reported given the SoC is strapped for LPC. It's also interesting that u-boot runs but claims not to be able to detect the flash, given u-boot needs to be loaded from the flash. A dump of the FMC registers might be interesting. Also it might be worth dumping the flash image ( |
|
Alright, I've pushed a real hack of a patch that will dump the FMC controller state for you @bielids: e1a919d Run it like: Currently it's hard-coded to read the FMC registers. Something to improve as I make the patch less of a hack. Example output (in this case I'm using the debug UART bridge, the trailing |
|
Thanks a lot for the patch! I'm having some issues building but I'm sure that it's due to something in my conda environment so let me look into that after my meetings and update you once I get it running. I'll do the QEMU test after that |
|
Alright I spent quite a while getting it to compile on PVE to no avail (devmem.h not found), which is weird since I first compiled it directly from the host. Anyways, I finally managed to get it to run by building it with Here's what I got in the end: How did you decode the value I gave you to end up with the output you sent here by the way? |
|
I was able to boot the FW I dumped with culvert by the way. I've never tried dumping firmwares and loading them in QEMU before so let me know if there's anything else I should try to get more info. Here's what I got: When I try using one of the two files that appears to be a kernel I just get |
The tool that I use to decode registers is bitfield, but that requires that you have the necessary configs set up to decode the registers. I write my own bitfield configs for the AST2500, but they're derived from the SoC datasheet, which cannot be freely distributed. I'll decode the FMC register values at some point, though it may not be until next week. Something that I should have mentioned was that the
This in itself is interesting. It suggests Can you paste the log (stderr) output of
Nice.
Some of the trick here is to choose a machine with the same SPI-NOR part. Given qemu boots it at all I guess you picked one with at least the same size flash part?
Try tacking the following onto the qemu commandline: |
Oh, yeah, sorry, the condition on 201 was an oversight after a few iterations of the patch, you were right to change it. It was also late here when I wrote the patch :) |
|
Another thought - what do you see on the BMC console if you run |
Ah OK, I'll need to power-cycle the box again then, I had to do a bit of troubleshooting due to line 201 so it's unlikely that it was in a "clean" state. The flash chip detected by culvert is a Macronix MXxxL51235F: Full output
Yeah exactly, I can't say that I know exactly what I'm doing but I noticed that Let me try adding options to qemu to see if it works and running the additional commands you suggested before I head out. It's getting late but this is the furthest I've gotten in like 2 weeks haha |
|
Here's what I get when I read the FMC controller state right after a reboot: root@pve:~# ./culvert.bin read controllerAnd after doing running Terminal output
I don't get much when running the |
|
By the way, I found this issue which mentions issues with the aspeed 2400 having issues reading from flash at boot due to incorrectly set registers ( Interestingly they tested it with 3 PNOR chips and only Is it in any way related? |
bielids
changed the title
Haha, yes, in so many ways. Regarding the people commenting on that issue, @legoater maintains the Linux driver for the Aspeed FMC and looks after the Aspeed QEMU models. @shenki is the upstream Linux kernel maintainer for BMC SoCs. In past lives we all worked together on IBM's Power systems firmware. That thread documents some troubles booting the Power8 host processor with an OpenBMC-based BMC firmware and the OpenPOWER host firmware stack. Part of the OpenPOWER host firmware stack is skiboot. However, I've been doing some poking. The AST2400 had a feature where SDRAM could be remapped to 0x0 in the physical address space of the SoC. Unfortunately the datasheet for the AST2500 claims that they dropped that feature. If they hadn't we could potentially do an in-memory boot for you (essentially we'd stop the ARM core, flip the remap bit, load a different u-boot at address 0 (which is now in RAM), perform a reset of the ARM core, and then clear the stop bit). However, without the memory remap capability we have to modify flash to do anything different. Are you comfortable with reflashing the BMC SPI-NOR? It might be interesting to experiment with a u-boot build from e.g. OpenBMC to see if we can get past the failure to initialise the flash subsystem. |
|
Oh wow, lots of interesting information to parse through here. Re. the in-mem boot that's a bummer, sounds like a low-risk way to troubleshoot the BMC.
Absolutely, I've got until tomorrow to return this board. This can be done from within the OS right (ie. no need to solder)? |
Yeah, no need to solder. We can reflash the BMC firmware via the PCIe bridge you're using currently, or over the BMC UART (without requiring a functioning BMC or host). I'm aware of at least a couple of large-ish cloud-ish vendors doing OpenBMC conversions this way using I had a look at the marketing photos of the board, and it looks like it has a socketed flash as well (handy if you have a flash programmer). Given that we have to modify flash we'd need to do a bit more digging to find the partition boundaries being used by the vendor image. If we're to do the experiment, we only want to re-write u-boot and not corrupt anything subsequent. We may be more constrained than the size of current OpenBMC u-boot builds (e.g. the Romulus image is ~366kB). Also before we go re-writing anything on the BMC flash, it would pay to |
|
Here's the data from the FW I dumped a few days ago using culvert: As for this:
Unfortunately I do not. I've got a few arduinos handy but I don't know if I'll have time tomorrow to setup something that could help. |
Mind pointing me in the right direction? I see that it's an option I can set but I don't quite get how I can set it. |
|
Not sure if that's useful but with ast2600-evb as my machine type I get this: I noticed that the fmc_model and spi_model for ast2600 in QEMU's code lines up with what I have on my machine, and so does the error... |
There's some more detail in the Aspeed-specific qemu documentation, which indicates you can specify https://www.qemu.org/docs/master/system/arm/aspeed.html#boot-options For instance, try: Regarding the flash type, For reference you can find your mapping from the JEDEC chip ID ( https://gitlab.com/qemu-project/qemu/-/blob/master/hw/block/m25p80.c?ref_type=heads#L243 |
|
|
|
What I wonder is how u-boot got into the state where there was something wrong with its flash driver, when the BMC had apparently booted okay prior to entering BIOS recovery mode. @bielids can you provide more details on which BMC and host firmware updates you applied in what sequence relative to changes to the BIOS recovery state? |
|
Hey, I didn't make any progress last night as I broke my initrd files and pxeboot stopped working on this host for some reason.
So basically I went into bios recovery mode in the hopes of getting more info out of my system as I had just received it (CPU & motherboard) but I couldn't get it to POST at all (I think that screw torque was the issue in the end). Here's a full timeline:
I spent a few nights troubleshooting post issues so I may be missing a few steps in between but that's the gist of it |
|
So I downloaded the 12.61.17 BMC firmware from Gigabyte. Extracting it, it seems they just provide the complete flash image, along with what is presumably their own copy of Aspeed's We can boot it directly in qemu: I wonder if you should try reflashing the image using |
|
Hey, the seller agreed to extend the return window until the end of the month. I truncated and then flashed the fw using culvert which actually got the BMC to a working state finally! With that being said, a few boots later and I was back to square 1 weirdly enough. |
|
Unfortunately I'm no longer able to detect the SPI flash even with culvert now: and here's the BMC console after a reset: The only thing that I did between the BMC working and now is setting an IPv4 address using the BIOS settings. 😕 Current culvert probe output: EDIT: after leaving the server turned off all night the BMC booted up without issues. I've left it off overnight many times before to no avail but this time it made a difference. ipmitool works (locally or via ipv4) |
|
Okay. Where would you like to take this? Should we close it for now? |
|
Closing this for now, re-open if you think there are things we can do to improve culvert for dealing with these circumstances. |
|
Hey hey, thanks a lot, we can leave this closed, I appreciate everything you folks have done to help. I was away from the computer for the past 2 weeks and unexpectedly did not have any kind of reception/connectivity during that time. So yesterday I did some memory tests and noticed that I would frequently get errors when running test #4 in memtest86+ on CPUs 23/51. I had been having random kernel panics so the idea that the CPU had some sort of issue causing this didn't seem out of the question. I checked sibling CPUs for core 23 and 51 came up. I did a few more tests using memtester and whenever core/thread 23/51 is used to write to memory I get errors. After disabling this core my system has been stable. I don't think that this caused my issues in the first place but I don't think that it helped either. Moving forward I'll leave that core disabled. Right now I have a zpool to recover from before I can further look into the IPMI issues I was having but once that's done (hopefully today) I'd be more than happy to do dumps or whatever else you think might be useful in figuring out what happened here in the first place. I would love to contribute to this project after all the work that you've put into this issue but I'm no C dev so there's only so much I can do. Let me know if you want anything from this system and I'll be glad to help. I understand if you're not interested, you've already spent a lot of time on this :) This tool saved me some many headaches and along the way I've learned so much about embedded systems. I'm now thinking of taking a course on u-boot and embedded systems in general just to get a better understanding of it all. Thanks again! |
No worries at all, I hope everything is okay for you.
Don't be concerned about that, there's certainly no expectation that you contribute back on my part! Sometimes it's interesting to dig into a problem and I enjoy helping others.
Yeah, largely I need whatever is to be done to be lead by you. Unfortunately I don't have the capacity to take on the issues with your system as a project, but I can tinker around the edges by improving the tools to help you out.
Awesome, I'm glad it was in some part interesting and not just 100% frustration at expensive hardware gone wrong. All the best with the future embedded adventures :) |
|
Hi! This thread is only thing in whole internet about AST2500 BMC recovery, so I hope I could chime in with my issue with ASP2500. If you know another place where it's more appropriate, I would be very grateful for pointing it out :) I did very stupid thing and tried to upgrade BMC firmware from 12.40.17 to 12.61.21 on Gigabyte MZ01-CE1 rev2 from linux running on "host" system via BMC's KVM session. Obviously (in hindsight) KVM session dropped at some point during firmware process. BMC_LED led switched off, BMC stopped responding to pings. I waited for 15 minutes, nothing happened. I reset power and BMC_LED now in steady on state. I've tried to reset CMOS by setting CLR_CMOS jumper and power cycling the board, but this didn't helped either (and I suspect that's why host cpu doesn't boot anymore) I've located BMC uart port and can see that BMC is stuck in u-boot: I've binwalk'ed firmware file tried to After a while BMC resets and drops back to uboot. I've also tried to boot from one of addresses listed in I have very basic understanding of u-boot so I have no further ideas how I can re-flash BMC. Unfortunately, I don't have VGA → HDMI adapter, it will take about a week to get one, so I don't know what happens when host is powered on. CPU fan start spinning at max PWM but nothing happens. I've made bootable USB flash drive with alpine linux that should autostart, enable network interface, get DHCP address and start with sshd, but it doesn't seem to boot. Another problem is that I'm on Apple-silicon Mac and don't have access to Intel hardware. I couldn't build the What gives me hope that culvert's readme says I suppose main question is it even possible to reflash via physical UART port if BMC in this state? If not, is there any other options? Any help is greatly appreciated! |
Unless the Gigabyte u-boot version that's currently on it disables the AST2500's on-by-default debug UART functionality (which I'd guess is unlikely, especially given that it's apparently built on a 3.x (!) kernel), yes, it should be possible to reflash the BMC firmware via the UART -- it is very slow, however. If you have a raw flash image for the desired firmware, you can do so by running something like: where [EDIT: looking back at previous discussion on this issue, I see from @amboar's comment that it looks like Gigabyte does in fact provide what's pretty much just a raw flash image.] For a 64MiB flash part a full reflash via the debug UART will likely take something on the order of dozens of hours to finish (perhaps a few days, I don't remember exactly how fast it generally runs offhand). There's some chance it might be possible to tftp in an alternate kernel/initramfs to boot (e.g. an OpenBMC evb-ast2500 build, which I'd guess would stand a decent chance of booting functionally enough for basic network & flash access to work) and do the full reflash from there, which would likely run quite a bit faster (~5-10 minutes if you can get it going successfully), but would be more experimental -- given the age of the u-boot you've got installed, you might need to also temporarily install OpenBMC's (newer) u-boot to be able to boot a FIT image. Note also that an x86 host isn't required to run culvert -- I've been successfully using it from a Raspberry Pi (ARM) for quite some time, for example. The host OS may be a bigger issue; I don't know offhand exactly how portable culvert and its various dependencies are expected to be, or if they should compile/work on macOS -- what problem(s) did you run into while trying? |
I don't want to derail this issue as there's a lot of interesting stuff happening in it, but for anyone looking in the future, I prefer that we use a discussion instead. @y8 - @zevweiss has already covered a fair bit here otherwise, so I'll let that run its course. |
|
First, I'd like to than you for you tremendous dedication and depth of your help! As Johannes said in mailing list it's very inspiring!
But the end I have built a static version on x86 cloud host and moved it to x86 emulator on my Mac and it worked :) Kudos to #46! However, when I tried to run I played with all kinds of default IPMI password that Gigabyte provide in their documentation, but no luck. According to strace culvert sent password value to UART and waited for response, which won't happen because it basically writes to u-boot shell.
Image file is slightly smaller than 64M (67,108,864)
I've considered this option, but couldn't find any docs on how this process works. Discussion you pointed to provides very detailed instructions on this, which I'll try later today! Thank you! Seems like booting in OpenBMC and flashing from there is viable option. I don't mind having it as BMC firmware if KVM, Power Management and Fan control (fan for EPYC 7551P is very loud at full speed) works on AST2500 :) I'll leave an option to reflash via culvert as a last resort, because we can have short blackouts here and I don't have backup power supply to power BMC while it being flashed. Once again, than you very much for your help! |
I see...so yeah, I guess dtc has some incompatibility with the Apple toolchain's assembler somehow? It might be worth filing a bug if you feel like it, though I'm not sure if the dtc devs regard macOS as a supported platform.
Ah, yeah...I suspect you may be the first to try to build culvert on arm64 -- something like this might fix it: diff --git src/mb.h src/mb.h
index 58bc83abc6f0..49229e71563b 100644
--- src/mb.h
+++ src/mb.h
@@ -10,7 +10,7 @@
#elif defined(__x86_64__)
#include "x86.h"
#define iob() mfence()
-#elif defined(__arm__)
+#elif defined(__arm__) || defined(__aarch64__)
/*
* HACK: Assumes we're running remotely or on the AST itself. If ARM is
* the host arch then we need to fix up the barriers
Oh dear -- I'm sorry, I completely forgot about that aspect, because I have that variable set automatically in my shell profile on systems where I use culvert. Unfortunately it needs to be set to a specific 64-byte string of gibberish from a specific page in the AST2500 datasheet -- a document that is (sadly) officially confidential. Aspeed's efforts at security-by-obscurity there haven't been (ahem) entirely successful (heck, it's embedded in their socflash binary and they don't seem to make any effort to restrict the distribution of that AFAICT), but posting it on github would be a little more bold than might be prudent for me at the moment. So uh...internet scavenger hunt to see if you can find it? (Apologies.)
As noted somewhere in the mailing list thread I think, it seems like Gigabyte's images are indeed slightly smaller than the flash chip, which seems a bit odd (I guess they just omit an unused tail end of it?), but should be easy to just Looking at those two files, however, reveals something slightly surprising (or maybe not, I dunno): The two are identical aside from a 128-byte footer present in the larger one (126121.bin). Given that, I'd guess the footer is probably some auxiliary metadata (checksums or the like) that isn't meant to actually go into the flash, so I'd probably use
Unfortunately I'm not aware of anyone porting OpenBMC to any Gigabyte hardware thus far (certainly not in mainline OpenBMC), so functionality would be pretty limited -- on most systems KVM support is pretty plug-and-play in my experience and would likely work, but host power and fan control generally require a fair amount of platform-specific information (specific GPIOs, temperature sensors, etc. etc.), so it's probably not going to be something you'll want to install and use day to day (unless you or someone else puts in the time & effort to reverse-engineer a working port...which would be cool, but also quite a bit of work).
Ah -- yeah, given that, aiming for the faster method is probably wise. If you can get an OpenBMC FIT image booted to an initrd shell, you can use the busybox tftp command from it to load the desired flash image in a tmpfs on the BMC, after which running
No problem! |
|
@y8 (and @zevweiss) Regarding |
I found datasheet a bit before @amboar kindly provided public link :D But it didn't work for some reason. I tried to manually input it over 1200b terminal connection but it didn't work either: no output after pasting password (I waited for ~5 minutes) From SDK User Guide it seems like BMC got 2 UART ports and only one has Debug mode enabled. Maybe JTAG_BMC is the wrong one and I need to use SOC_UART port that is not populated. I followed Johannes instructions and was able to chain-load patched u-boot, yay! But haven't had much luck with OpenBMC fit image: From But linux panics with Does it mean that there is something wrong with device tree in this image? I tried both I'm building |
|
Uh, I suppose this story abruptly ends here: seems like my USB UART ground wire was loose and I fried the BMC when I was powering PSU on :| Any tips on how to safely wire UART to mains powered motherboard to avoid incidents like this are very appreciated :) But anyway, now there is no serial output at all. I tried different UART adapter and still got radio silence. I will solder and check what is going on with SOC_UART later today and maybe manage to find VGA → HDMI adapter later this week to check if host system can boot, but it feels unlikely. If by chance it does, I can manage to live without BMC. A bit sad, but after all, it was my attempt to build a cheap-y homelab for machine learning. I got this motherboard for few hundred dollars on eBay. I bet seller will be more than happy to supply me with another one :D I will post update on how it went. Once again, I'd like to thank you: your help is priceless and I've learned a lot! I believe in the spirit of this project my main takeaway is first hand experience how easy it is to plant something in BMC and blurry (and already scary) picture about what implications it might have for host system Thanks again! |
Well I'll be...news to me! Though given that they apparently consider the password straight-up public now, is there any reason we shouldn't just include it in culvert instead of requiring it to be passed via an environment variable? |
The long file names and multiple levels of symlinks in the
Damn, well that's a shame. In my experience the serial adapters themselves are usually the easier thing to accidentally fry, so I was going to suggest trying another, but it sounds like you've already ruled that out. If it's any consolation, I once accidentally connected my serial adapter's GND to +12VDC (I could have sworn I checked the rail with a multimeter and confirmed it was ground, but apparently I mixed something up somewhere along the line)...it got quite warm, and never worked again. Though actually, now that I think about it I've also managed to cook the BMC side of the UART on one unit at some point, though I no longer remember how. Experiences like these are why I now have one of these posters on the wall in the corner of my office where the relevant hardware sits: |
|
Oh, and re:
The debug UART mechanism is in my experience a bit fragile and lockup-prone; if you haven't done so already I'd suggest (once you've got some working hardware) trying a full DC power cut/restore to get it back into a stable initial state before attempting to poke it with culvert. If it works it should do so fairly promptly (within a second); if it still doesn't after a power cycle I'm not sure what else to suggest other than double/triple checking the password (e.g. for |
|
A bit of good news: kind people from local user group where I was looking to lend some VGA-equipped hardware, suggest that if it starts the fans and enables ethernet controller, then it most likely can POST, but can't boot from my USB pen drive it might try to boot from PXE. And yes, it indeed does! I'm not sure why it's hit and miss after power on, sometimes it resets, sometimes I can see how negotiated link speed jumps from 10 to 1000M. But in a minute after it stabilizes on 1000M, it broadcasts DHCP requests! Also network interface on BMC also establishes link with router so hopefully only UART side is fried and I still can boot host system and flash it with with culvert :) I also tried both COM1 and COM2, but now luck here: according to gigabyte docs serial forwarding is disabled by default and unfortunately both ports were silent. I have double-checked my both UARt adapter by connecting RX to TX and they both properly echo input on 9600 and 115200 bauds. Now I need to figure out how to properly boot it over PXE given that I don't have ethernet on my mac (first time when wireless bite my arse) and my router is from ISP with all the fun stuff locked out. Worst of all, on average it takes about 12 minutes for motherboard to actually broadcast DHCP request. I wonder what really happens during this 12 minutes! So far, I managed to configure dnsmasq to respond to PXE requests only originated from from motherboard MAC to avoid messing up dhcp on router. I have dnsmasq announcing PXE options to MB but PXE is (according to internet and my fresh experience) complete PITA. I can't even find the proper combination of So this story is not over yet :) @zevweiss I'm totally sticking that warning sign on this motherboard! |
Note that I pinned the
Yep, I'm thinking the same. Just need to be clear in the commit message for the change what the source of the password is.
100% this (setting aside your other hardware concerns). The debug UART init sequence is fragile and will enter a bad state if someone even so much as looks at it funny. |
|
Hooray, I have working BMC! 🥳 I finally managed to boot over PXE (not an easy task) into linux with ssh and re-flash firmware However I had issues with culvert. It could probe the controller: But couldn't read the firmware (I wanted to dump it just in case things go sideways) (without verbose output it fails at So I decided to try to flash it with stock utility first. And it worked! After power cycle BMC booted, blinked at me and it seems like everything works (and much better than firmware from 2020) I will test host hardware to figure out is there any damage beyond UART and then, if needed, I can do whatever it might help to contribute to culvert! I have aarch64 around, so I will try patch suggested by @zevweiss. If you don't mind I can contribute GitHub Actions to produce static binaries for PR's and tag pushes for supported platforms. This might be handy! Can't thank you enough for your help on this journey! This experience is so inspiring and help me believe in FOSS community If we happen to meet drinks and dinner is on me :) Thanks again! |
|
Based on some of the commentary here I've pushed a dev/gigabyte-misc branch. It would be helpful if you could test it. It's not clear to me why it might have failed to identify the flash. However, from the logs I'm implying that culvert has chosen the P2A bridge to access the BMC. I expect gigaflash is doing the same. If you're up for a bit of experimentation my quick poking around suggests we can use
Thanks for the enthusiasm, but I prefer we don't distribute culvert binaries.
Oh, no need at all :) |
|
Here's what I've got running list of packages needed to build culvert on clean ubuntu, because I lost list I've made to make last build (maybe add them to README?)sudo apt update && sudo apt install -y build-essential meson bison swig flex cmake ccache pkg-config device-tree-compiler python3-dev libyaml-dev libftdi-devlibreadline-dev zlib1g-dev libssl-dev
|
|
So we do have this: but it looks like I missed this note towards the end of the documentation:
|
Hmm...this would of course be its own chunk of development effort, but could a userspace mmio interposer/tracer perhaps be rigged up via a combination of ptrace and userfaultfd? |
|
hmm, that's a bit of a nerd snipe... |
Hi, this may be a long shot, but I've got a Gigabyte MZ32-AR0 rev3 with a BMC that's not starting up (steady BMC_LED light, no firmware version displayed during POST, no devices detected by ipmi_si). I came across this utility and I was wondering if I could leverage this to possibly factory reset it and get it back to a working state. It looks like the BMC stopped working after I put the board in BIOS recovery mode and I've spent countless hours trying to get it back up. Connecting to the server's serial port gave me no more useful info and connecting to BMC_UART only got me this:
I've tried flashing a new FW (a supported one of course, from the mobo's support page) to no avail (it flashes "correctly" but I still can't detect the BMC, and the mgmt port doesn't appear to be in a proper state based on the flashing). I've tried connecting to the console using this tool but that didn't get me anywhere either.
At this point I'm running out of options and I've only got until Friday to return the board (which will take a few weeks) so I'm hoping for a hail mary at this point. I went through the issues and the little documentation I could find about this tool and didn't get anywhere. Do you have anything to suggest?
This is what I see when probing:
Reading the RAM seems to get me info similar to what I'm getting from u-boot
Thanks!
The text was updated successfully, but these errors were encountered: