nRF52833 DK (PCA10100) CircuitPython UF2 does not work on Adafruit_nRF52_Bootloader

[jpconstantineau]

CircuitPython version

adafruit-circuitpython-pca10100-en_US-7.0.0

Code/REPL

no code...

Behavior

Cannot flash CircuitPython on nRF52833 Dev Kit.

Bootloader was flashed with embedded JLink. Tried with latest release of bootloader (0.6.2) as well previous version (0.6.1)

Nothing happens is what I see...

When I drag/drop the UF2, it copies, the device reboots back into bootloader mode.

Description

No response

Additional information

I was first trying with a new nRF52833 board but it didn't work. Had to buy a nRF552833 Dev Kit rom Nordic to validate that the reference board didn't work... It actually shows the exact same problem as my board. Hence, the issue has been replicated on a board currently out for CP 7.0.0

[tannewt]

Does CircuitPython work if you flash it directly with JLink?

[jpconstantineau]

Flash the uf2? I don't have the tooling to get the hex file as I rely solely on GHA CI. I don't recall seeing the hex files anywhere.

[jpconstantineau]

I finally managed to get a hex compiled locally and flashed and the same happens. @petejohanson was able to flash his ZMK firmware on test hardware and got it running fine. As such, its not really a bootloader issue. ZMK uses Zephyr, which uses its own BLE stack (not the softdevice). This might have an impact but not quite sure.

Does CP automatically initialize/start the SD when starting up? Or does it wait for bleio to get called? Could this be related to the new BLE workflow?

Nordic doesn't have that version of the softdevice for support of the 833. That is mentioned in one of the PRs for the Arduino nRF52 board package...

@dhalbert indicated (on discord I seem to remember) that he has a 833DK. I have one too. Would dropping the BLE stuff altogether from the modules be a good test of SD compatibility (in the board makefile)? Or does it get called anyways?

[dhalbert]

Does CP automatically initialize/start the SD when starting up? Or does it wait for bleio to get called? Could this be related to the new BLE workflow?

The new workflow does use the SD from the start. You could prevent that by doing supervisor.disable_ble_workflow() in boot.py. We might use the SD for a few other things, but we don't enable it until we need it.

[dhalbert]

Does CircuitPython 6.3.0 work?

[jpconstantineau]

With this bootloader (from INFO_UF2.TXT)
UF2 Bootloader 0.6.2 lib/nrfx (v2.0.0) lib/tinyusb (0.10.1-293-gaf8e5a90) lib/uf2 (remotes/origin/configupdate-9-gadbb8c7)
Model: Nordic nRF52833 DK
Board-ID: nRF52833-pca10100-v1
SoftDevice: S140 version 6.1.1
Date: Sep 10 2021

Neither adafruit-circuitpython-pca10100-en_US-6.3.0.uf2 or adafruit-circuitpython-pca10100-en_US-6.0.0.uf2 work.

I drag/drop the UF2 onto the drive and it never gets to boot into CP properly - it just goes back into the bootloader mass storage drive...

[jpconstantineau]

Regarding supervisor.disable_ble_workflow: CP doesn't even get to boot to be able to place/edit the boot.py file.

[jpconstantineau]

Any thoughts on where I should poke next?
Is this an incompatibility between bootloader and CP? Is there a combination we know worked in the past?

[dhalbert]

The PCA10100 was added after 5.3.1, but is in some alphas and betas shortly after. So try the earliest builds here and see if they work: https://adafruit-circuit-python.s3.amazonaws.com/index.html?prefix=bin/pca10100/

Then also try earlier versions of the bootloader, from around that time. The PCA10100 was added Nov 11, 2019, but I don't know that it was tested at that time. On the other hand, presumably the people who added the board to CircuitPython originally presumably tested the firmware and probably used the UF2 bootloader.

[dhalbert]

@xobs Are you still using the PCA10100 or simmel? Do you have any ideas about why @jpconstantineau might be having problems? Thanks.

[xobs]

No, sorry. I haven't used the NRF52833 hardware in a very long time.

[jpconstantineau]

@xobs Did you use the Adafruit_nRF52_Bootloader? The first release the PCA10010 in is 0.3.0 - 2020.01.13
If so, do you think it would be that release? Timing looks about right...

[xobs]

I used https://github.com/simmel-project/bootloader

[jpconstantineau]

Thanks. That will give me something to compare with...

[jpconstantineau]

Quick update...

I tried a bunch of bootloader releases and CP releases:

Bootloaders:

• 0.3.0 Loaded fine
• 0.3.1 Could not load bootloader on hardware
• 0.3.2 Loaded fine
• 0.4.0 Bootloader drive did not mount
• 0.4.1 Bootloader drive did not mount
• 0.5.0 Bootloader drive did not mount
• 0.5.1 Bootloader drive did not mount
• 0.6.0 Bootloader drive did not mount
• 0.6.1 Loaded fine
• 0.6.2 Loaded fine

CircuitPython Builds for PCA10010:

• 5.4.0beta0
• 5.4.0beta1
• 6.0.0
• 6.1.0
• 6.2.0
• 6.3.0
• 7.0.0
• 2021-11-03 build (latest)

None of the bootloader/CP builds combinations made it into a running Circuitpython on the PCA10010.

[tannewt]

@jpconstantineau Do you have a jlink? You could use that to see how far code execution is getting.

[jpconstantineau]

I do have a jlink (it's included as part of the PCA10010). I never got into using it for debugging... I also have a Black Magic probe.

Sounds like I'll need to learn how to use them to figure out if it even gets into the code. From what I have seen on the nrf52840, when the code faults, it might reboot. In this case, I don't see it enter a boot loop at all. It's goes straight back to the bootloader as if nothing got flashed...

Do we have the right memory/flash mapped to the device? I know the ZMK guys were able to load their firmware on that same bootloader and it works... (Zephyr based) They had to figure out the memory map first (from what I hear...)

[scath999]

Had the same issue - glad I'm not alone.

When I drag and drop the .UF2 file onto the bootloader in windows, I'm told that the bootloader doesn't have enough space for the file.

adafruit-curcuitpython-pca10100-en_US-7.0.0.ut2 filesize: 458k (460k on disk)
Bootloader capacity :128k

Does this help?

(PS I'm brand new to firmware development - no idea what's relevant or not. Sorry if I'm muddying the water or asking a stupid question. Appreciate the patience).

[scath999]

PS - to be clear, I'm referring to the dev board, not the actual production board.

[scath999]

Ok. With @dhalbert's help, I'm now up to speed and at the same point as @jpconstantineau.

The dk board has a built in segger jlink, and I'm able to successfully executemake BOARD=pca10100 flash

This appears to complete properly, but when the device restarts there is no change - the board reboots and appears as a drive to the OS, but with the same volume label and contents as before.

I have left a message on the nordic dev boards asking for some insight here - perhaps there's some kind of signing issue that's causing the board to reject the build? (Shooting in the dark here).

Next steps: how can I use my setup to help you diagnose and fix what's going on? I have time to do this on my side, and am happy to volunteer. I've also joined the discord, so you can find me there.

Best,

S.

[scath999]

Update:

While trying to flash firmware.hex over to the board using nrfjprog, I get the following errors:

D:\CircuitPython\circuitpython\ports\nrf\build-pca10100>nrfjprog -f NRF52 --program firmware.hex -erasesector --verify --log Parsing image file. Verifying programming. ERROR: [ nRF52] - Data does not match in address range [0x00027000-0x00027100] (Flash) ERROR: [ nRF52] - Expected byte value 0x02 but read 0x00 at address 0x00027002. ERROR: [ nRF52] - Flash verification failed. ERROR: [ nRF52] - Failed while verifying file firmware.hex. ERROR: Write verify failed.

I'm assuming that I'm missing some steps and/or some parameters here. Sorry, I'm new.

@jpconstantineau Did you use the commandline to flash? If so, what did you use?
@dhalbert @tannewt Does this help?

[jpconstantineau]

I used the drag/drop feature of the JLink that's embedded with the PCA10100. I didn't use the command line.
What you provide as command line is indeed progress!

From the address, it seems to fall within the flash address space.
I don't know if the memory map is similar to the nrf52832/40 (at the bottom), but if it is, this would fall well within the application space (See here)

Where is the memory map for the nRF52833 set and how does it compare with the nRF52840?

[jpconstantineau]

Found a good nugget of information in the nrf makefile here

It does the flash in multiple steps, including an overwrite of a bit to bypass the validation check. I suspect this is something we don't do with the UF2 or the drag/drop of the Hex file; making the bootloader reject circuitpython. However, that doesn't explain the verify error @scath999 found.

@scath999 If you can figure out the parameters passed to nrfjprog and do it manually, perhaps this will help.

Note that the 7.x softdevice is also merged as part of the make. my hexmerge.py install is broken and I can't get it to build completely.

For the build, l use: make BOARD=pca10100 V=2
I don't know how to call the FLASHER portion of the make script.

[scath999]

@jpconstantineau

try make BOARD=pca10100 flash

[jpconstantineau]

Progress! Thanks! However, nrfjprog within wsl isn't found... I know I have it on the windows side... I'll try adding it to the path but I suspect I got to fix my hexmerge first.

[scath999]

@jpconstantineau

To install hexmerge, try pip3 install intelhex?

[jpconstantineau]

I have done that already. Something is actually broken... Somewhere...
I get this when I do my build:

Traceback (most recent call last):
  File "/usr/local/bin/hexmerge.py", line 11, in <module>
    load_entry_point('intelhex==2.3.0', 'console_scripts', 'hexmerge.py')()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 490, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2853, in load_entry_point
    raise ImportError("Entry point %r not found" % ((group, name),))
ImportError: Entry point ('console_scripts', 'hexmerge.py') not found

[scath999]

@jpconstantineau Good find. However, when I build the project, I find s140 v7.2 in the build directory and none others.

[jpconstantineau]

@scath999 Regarding virtual... I build on windows using WSL. I now have this repo to script the creation of the WSL environment, setup the dependencies, clone the repo, setup the python dependencies, then do a build of the pca10100.

That new broadcom firmware module is a beast! at 1 GiB and only 5% done. Takes forever to download...

Is WSL ultimately running in a virtual machine? I guess so... I am still stuck at trying to get a build done.

[jpconstantineau]

@scath999 Got the build complete. I suspect that .local/bin wasn't in my path and that was one of the issues I had.

From the hexmerge command, it looks like it's 7.0.1 SD that gets merged in the hex.

hexmerge.py -o build-pca10100/firmware.combined.hex build-pca10100/firmware.hex bluetooth/s140_nrf52_7.0.1/s140_nrf52_7.0.1_softdevice.hex

[scath999]

@jpconstantineau I see - nice. Congratz on getting the build going, and thanks for posting that.

Looks to me like this is a pretty strong candidate for the underlying reason why CircuitPython fails to deploy on this board. What do you think?

@tannewt @xobs @dhalbert Where can JP and I find this in the code? Hope I'm not disturbing your weekends ;)

Cheers!

[dhalbert]

circuitpython/ports/nrf/boards/common.template.ld is a template with values that are filled in from port- or board-specific values. Note SD_FLASH_SIZE in particular.

[scath999]

@jpconstantineau I have it working. Can you verify if this works?

Steps:
After cloning the repository and updating the submodules

1. Build mpy-cross as given in the current docs. (Note: there may be some manual editing required to get this to build on windows, but it can be done).
2. Move to /ports/nrf
3. Plug your USB cable into the J-Link port, located on the left side of the board above the battery
4. Power up the 833
5. Run make V=1 BOARD=pca10100 sd - this will create and flash the Soft Device using nrfjprog
6. Run make V=1 BOARD=pca10100 flash - this will create and flash the Firmware using nrfjprog
7. Power down the 833
8. Unplug your USB cable currently in the J-Link port on the left hand side of the board
9. Plug your USB into the MSD port, located at the bottom of the board
10. Power up the 833

Notice: A new device appears with the volume label "CIRCUITPY"

11. Open the volume named "CIRCUITPY"

Notice: The expected directory structure appears, including code.py

[dhalbert]

@scath999 Great! How was this different from what you were doing before that was not working?

[scath999]

@dhalbert I think @jpconstantineau and I were missing a step: I didn't do step 5, which accounts for the memory errors I was getting before.

The intuition that I was missing this was gathered from reading the Nordic Dev forums, but strengthened by the docs found here

All in all seems like user error. However, these instructions appear on the page for the 840 board - I never got that far in the instructions because of this: I figured these instructions didn't apply to the 833.

Do these steps appear somewhere in the docs that I missed?

[dhalbert]

There is ports/nrf/README.md, and there are some README.mds in some individual board directories. They don't mention flashing the SD specifically, because it would have been flashed when you flashed the bootloader (or should have been).

If you would like to submit a PR to update that README, that would be great. In the past I have thought of just deleting the individual board README's, because they are not kept up to date, but if there are nuggets of info in them that could be hoisted up to the general README (and then they would be deleted), that would also be great.

[jpconstantineau]

This may account for the "build" instructions, it doesn't account for the standard way of simply uploading the UF2 to the board bootloader which is what most users would do to setup their boards.

You appear to flash twice. One with the new softdevice, the other without. Would we need 2 different uf2 or a different bootloader (one with the other softdevice)

[jpconstantineau]

@dhalbert the nrf52 bootloader only has a uf2 with the old bootloader and not the 7.x one. Since 6.x isn't officially supported by Nordic on the 833, I suspect this should be resolved at the bootloader level.

I do seem to remember someone working and getting a PR on this.

[scath999]

@dhalbert Whatever I can do to help :) And thank you again for all your support - you and your team have been endlessly patient and kind. I think simply mentioning a more verbose set of steps in case dragging and dropping fails would suffice.

@jpconstantineau Did the steps I outlined above work for you? If so, would you agree that a more granular set of instructions might be helpful in the event that dragging & dropping doesn't work?

[jpconstantineau]

@scath999 I am in between switching PCs (darn black Friday sales...). It will take some time before I can test it out. However, the softdevice discrepancy does make a lot of sense. I think we will need to confirm that a bootloader update with the correct softdevice will resolve the issue of the CP UF2 drag/drop not working.

[scath999]

@jpconstantineau Understood.

@dhalbert I will wait for JP to confirm the steps work, or report that they don't, before submitting a PR. Good?

[dhalbert]

Sounds fine! Note https://github.com/adafruit/Adafruit_nRF52_Bootloader/blob/master/README.md. I believe when you load the bootloader with the flash Makefile target there, it also loads the SD. But that is not the case for CircuitPython.

[scath999]

Cheers!

[jpconstantineau]

I can confirm that running the following two commands make the PCA10100 run CP.

make V=1 BOARD=pca10100 sd
make V=1 BOARD=pca10100 flash

However, this doesn't resolve the issue that the CP UF2 doesn't work on the vanilla bootloader as it requires the embedded jlink that's on the PCA10100 to flash everything. This is not desirable for non dev-kit boards (like the BlueMicro833) as it doesn't have an embedded JLink and users wouldn't be able to load CP the normal way (drag/drop a UF2)

The first command calls this:

nrfjprog --eraseall -f nrf52
nrfjprog --program bluetooth/s140_nrf52_7.0.1/s140_nrf52_7.0.1_softdevice.hex -f nrf52
nrfjprog --program build-pca10100/firmware.hex --sectorerase -f nrf52
nrfjprog --reset -f nrf52

The second command calls this:

nrfjprog --program build-pca10100/firmware.hex --sectorerase -f nrf52
nrfjprog --erasepage 0x7F000 -f nrf52
nrfjprog --memwr 0x7F000 --val 0x00000001 -f nrf52
nrfjprog --reset -f nrf52

What's so different from the nRF52840 that it requires this special memory re-write? Can't we embed this in the hex and convert it to a UF2? (or will it make it into a super large UF2?)

Doesn't the --eraseall parameter erase even the bootloader?

I think we can work in the bootloader repo/project to take care of the first step (get SD to the right version). However, we need to understand why we need the second step... (especially the memwr...)

[jpconstantineau]

There is a PR already under way in the bootloader repo BOOT#231 that would help with the default softdevice but it's currently stuck with non-related hardware dependencies (USB VID/PID on a board that it also brings in). I left a comment asking if the two could be split in two separate PRs.

I haven't tested if those changes will resolve the issue here (or simplify what needs to be done to upload CP). However, I suspect it help simplify things...

[scath999]

Thanks for confirming @jpconstantineau.

I agree that it would be nice to simply drag and drop the UF2 onto an SOC, or even the nrf52833, but if I'm understanding the issue properly this is being curbed by available physical memory that the default bootloader is allowed.

Thoughts?

[jpconstantineau]

I just got my new custom board to compile and get flashed to my custom hardware (just did a PR for my BlueMicro833). It appears that CP doesnt need the second command to get CP running on the board. The first one (make V=1 BOARD=pca10100 sd) seems to flash to the board and it loads fine. With the exception that the bootloader seems to be gone and I can't enter in it anymore (doing the double-reset dance).

Looking at the nrfprog docs, do we need nrfjprog --eraseall -f nrf52 at all? This seems to get to erase the bootloader. Then we flash the softdevice and CP. However, couldn't we flash the SD with the following instead?

nrfjprog --program bluetooth/s140_nrf52_7.0.1/s140_nrf52_7.0.1_softdevice.hex --sectorerase -f nrf52

[scath999]

@jpconstantineau I honestly don't know - I'm really new to hardware development.

Have you had any luck using NRFConnect? (https://www.nordicsemi.com/Products/Development-tools/nRF-Connect-for-desktop/Download#infotabs)

That has been even easier than using nrfjprog.

My process with that app is to select the DK from the top left dropdown, Add the SD hex, Add the Firmware hex, then click "erase and write".

Not sure if this helps answer your question. At this point I'm not sure if you just want your device to run CPY, or if you want to find a way to get the Drag & Drop functionality working.

Can you clarify?

[jpconstantineau]

Nrfconnect: never used it to flash. I have stayed away from the SDK and since the Arduino nRF52 bootloader works great with the Arduino IDE, I kept to that. I use nrfconnect on my phone to diagnose connections/BLE services. Working vs flashing: it's actually both. I sell boards with the bootloader so that people can flash whatever they want. The original 840 works great with the Arduino framework, ZMK as well as CircuitPython. One can switch between any of these without the need of a jlink to flash a new "framework" or bootloader. ZMK uses the UF2 method for flashing their firmware to the board. With Arduino, one can get a UF2 but if you use the IDE, serial would be the way most people go. With Circuitpython, you load CP, then it makes available the repl via serial. With the chip shortage, I had to move to the 833... At first, I couldn't even get CP to run, simply because the UF2 mechanism of loading it doesn't work. As it was new hardware for me (the nrf52833), and that it doesn't have Arduino support as well, I couldn't validate if it was my hardware, the bootloader, CircuitPython or something else that didn't work. I got the PCA10100 to validate that it wasn't my hardware. However, even that didn't work (hinting that it wasn't my hardware but something more fundamental) In the meantime, the ZMK folks were able to validate that my hardware was running fine and that they could load their firmware on the 833. They don't use that softdevice as Zephyr uses its own BLE stack. I feel that the nrf52833 bootloader isn't quite up to what it should be as someone pointed out memory discrepancies as well as the fact that 6.x SD doesn't support the 833 but 7.x does. In the long run, it would be good to sort this out (bootloader and/or CP) so that we have a reference stack we can work with for developing new boards. Just getting CP to run without the bootloader (or what seems so) is a good step forward but even upgrading CP to a new release would need a jlink; something not desirable.

…

On Tue., Dec. 7, 2021, 23:29 scath999, ***@***.***> wrote: @jpconstantineau <https://github.com/jpconstantineau> I honestly don't know - I'm really new to hardware development. Have you had any luck using NRFConnect? ( https://www.nordicsemi.com/Products/Development-tools/nRF-Connect-for-desktop/Download#infotabs ) That has been even easier than using nrfjprog. My process with that app is to select the DK from the top left dropdown, Add the SD hex, Add the Firmware hex, then click "erase and write". Not sure if this helps answer your question. At this point I'm not sure if you just want your device to run CPY, or if you want to find a way to get the Drag & Drop functionality working. Can you clarify? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5498 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIYW6BMJUCBR533UI24VBLUP3UK3ANCNFSM5GR3WZYQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[jpconstantineau]

Quick update on this...

I was able to retain the bootloader if I bypassed the SD flash step and did it manually as such:

nrfjprog --family NRF52 --eraseall
nrfjprog --family NRF52 --program "bootloader" --chiperase --reset
nrfjprog --family NRF52 --program "softdevice" --sectorerase --reset

Followed by

make V=1 BOARD=bluemicro833 flash

this called

nrfjprog --program build-bluemicro833/firmware.hex --sectorerase -f nrf52
nrfjprog --erasepage 0x7F000 -f nrf52
nrfjprog --memwr 0x7F000 --val 0x00000001 -f nrf52
nrfjprog --reset -f nrf52

I also tried the above steps manually and removed the reset of the data at location 0x7F000. Without this step, I observed the exact same behavior as the drag/drop of the uf2 on the bootloader drive.
I did try the above steps with the bootloader with the correct memory settings but the same thing happens. (there is a bug in the bootloader and thought that the fix might resolve this as well - it didn't)

This location is related to bootloader settings. I found some information here:

https://devzone.nordicsemi.com/f/nordic-q-a/17438/bootloader-sdk12-wont-start-application

It appears we can generate a hex file with this information and merge it in.
It's getting late here. Hopefully someone can chime on these findings.

One thing we can update is the SD flash step to avoid a full erase of the chip and use a sector erase instead.

[jpconstantineau]

Some more updates...
After a massive sense of déjà vu, I dug back to the work I did years ago on the nRF52832 and found that if I perform the following steps:

nrfjprog --family NRF52 --eraseall
nrfjprog --family NRF52 --program "bootloader.hex" --chiperase --reset
nrfjprog --family NRF52 --program "softdevice.hex" --sectorerase --reset
nrfjprog --family NRF52 --program "app_valid_setting_apply_nRF52832.hex" --sectorerase --reset

I was able to drag/drop the CP UF2 file and get it loaded and running.

In a nutshell, here are the issues found
1 - NRF Makefile line 286 and 287 should be updated/removed to do --sectorerase instead of --eraseall (this only affects users building CP and flashing to their devices using jlink)
2 - bootloader hex and/or CP UF2/hex should include the bootloader settings with respect to App valid/crc check. (this affects all users setting up CP on their newly purchased hardware...)

I'll be checking up on how to do this second piece with mergehex and uf2conv and identify a path forward.

[jpconstantineau]

Good news everyone. I just tested changes on the bootloader side to make a build that includes the 7.2 Softdevice and there is no need to do anything special to load Circuitpython. Drag/drop of the CP UF2 as per other boards works perfectly fine.

I'll close this issue once all the bootloader changes are merged in and tested with merged binaries.

[jpconstantineau]

The recent updates to the bootloader and builds for the nRF52833 have resolved this.

If you encounter this issue, make sure you have the latest bootloader on your nRF52833 device as well as the latest CircuitPython build.

I suspect that the root cause of the problem is that the softdevice that was part of the bootloader was not at the version that was needed by Circuitpython. Now that this is resolved, I doubt we will see this again.

[tannewt]

Thank you @jpconstantineau