bmips: add build support for BCM3380 SoC (Netgear CG3100D)

[rikka0w0]

1. Add necessary build scripts with make menuconfig support to build images for BCM3380
   

2. Give you the option to choose the image packer between hcsmakeimage and Broadcom's aeolus ProgramStore. See BCM33xx image packer in Target Images option.

The hcsmakeimage uses its own lzma decompressor. For some reason, it is extremely slow. Luckily, Broadcom open-sourced the aeolus utility, which allows us to utilize the built-in lzma decompressor of the stock bootloader. The image produced by aeolus is smaller and loads faster.

What works

1. After applying these patches, you can build a bootable initramfs OpenWrt image. You need to open the case and connect the serial console to your PC (115200, 8n1). You need to interrupt the boot process and use tftp to upload and boot the initramfs image.
2. The initramfs OpenWrt image will give you an interactive console over the serial connection.
3. CPU cache and SMP work. Both cores can run at their full strength.
4. SPI, OpenWrt can now access the on-board flash. (Updated on 2025/01/13)
5. OpenWrt can now detect the on-board ethernet switch (BCM53115). (Updated on 2025/01/14)
6. Ethernet works (to some extent) with my new Unimac driver.

What does not work

Sort by priority. GPIO and pinmux may be easier to support.

1. As of 2025/Feb/09, the ethernet driver uses polling for Rx, which adds unnecessary latency and high CPU usage.
2. GPIO and pinmux. Don't know how to test it. Need to change a mux position in order to detect BCM53115.
3. USB. The SoC has a different USB peripheral compared to BCM63xx. The biggest difference is that BCM3380 seems to support USB-OTG.
4. PCIE. Dont have time to look at it. However, if we get the PCIE to work, having Wifi on CG3100D can be very straight forward, at the PCIE Wifi card already has mainline Linux support.
5. Cable model. Never expect this to work.

We can declare a victory if we can make the ethernet work...

More process can be found here:

https://gist.github.com/rikka0w0/4e4d5feb3a50a8b64224750140f859ef#file-cg3100-md

Boot log:
https://openwrt.org/inbox/toh/openwrt/netgear_cg3100d_v3

Any help is welcomed.

[danitool]

Hi @rikka0w0 :

2. GPIO and pinmux. Don't know how to test it. May related to the SPI problem.

Searching a bit in the source code from these boards:

static inline void GPIOSetOutput( unsigned int gpioBit, unsigned int value )
{
    uint32 mask;
    volatile unsigned int *pGPIO = (volatile unsigned int *)0xb4e00108;

    mask = ~(1 << gpioBit);
    *pGPIO = ((*pGPIO & mask) | (value << gpioBit));
}
#if defined(CONFIG_BCM93380)
    /* Chip select 0 */
    GPIOSetOutput( 14, 1 );

Looks like it enables the SPI chip select 0 pinmux. You could test this code somewhere, for example in setup.c at bmips directory. BTW it should be already enabled by the bootloader, it can be checked using devmem and see the state of the bits at the address 0x14e00108.

[rikka0w0]

Hi @danitool,

Thanks for pointing this out. However, I found that the code snippet you provided above only controls the GPIO output state, not the mux. The GPIO_BASE is 0xb4e00100, so volatile unsigned int *pGPIO = (volatile unsigned int *)0xb4e00108; points to the DataLo field.

This Netgear Open sourced firmware contains some Linux kernel code that has a complete register definition of the BCM3380 SoC. If you unzip the downloaded zip archive inside CG3100L_V1.0.4_Linux_src.tar.bz2.zip/shared/broadcom/include/bcm963xx, you can find the headers there.

Here are some register definition from bcm3380/gpio_regs.h

typedef union {
  struct {
    uint32 PmSelectMdio                   :2; 
    uint32 PmSelectGmiiTxd                :2; 
    uint32 PmSelectGmiiTx                 :2; 
    uint32 PmSelectGmiiRx                 :2; 
    uint32 PmSelectGmiiRxd                :2; 
    uint32 PmSelectReserved               :2; 
    uint32 PmSelectGpio3532               :2; 
    uint32 PmSelectGpio3130               :2; 
    uint32 PmSelectGpio2924               :2; 
    uint32 PmSelectGpio2320               :2; 
    uint32 PmSelectGpio1918               :2; 
    uint32 PmSelectGpio1716               :2; 
    uint32 PmSelectGpio1514               :2; 
    uint32 PmSelectGpio1312               :2; 
    uint32 PmSelectGpio1106               :2; 
    uint32 PmSelectGpio0500               :2; 
  } Bits;
  uint32 Reg32;
}  GpioPinMuxSel;

typedef union {
  struct {
    uint32 Reserved                       :10;
    uint32 PmSelectUsb1                   :2; 
    uint32 PmSelectUsb0                   :2; 
    uint32 PmSelectGmiiClk                :2; 
    uint32 PmSelectLed0700                :2; 
    uint32 PmSelectSpisUart1              :2; 
    uint32 PmSelectAgci0704               :2; 
    uint32 PmSelectAgci0300               :2; 
    uint32 PmSelectPass                   :2; 
    uint32 PmSelectHvg                    :2; 
    uint32 PmSelectBmu                    :2; 
    uint32 PmSelectTpMisc                 :2; 
  } Bits;
  uint32 Reg32;
}  GpioPinMuxSelHi;

typedef union {
  struct {
    uint32 Reserved                       :14;
    uint32 SpiSSlv                        :1; 
                                              
    uint32 SpiSlvRst                      :1; 
    uint32 Reserved2                      :6; 
    uint32 DlyDis                         :1; 
                                              
    uint32 Reserved3                      :4; 
    uint32 SerialAddrCfg                  :4; 
    uint32 SpiMode                        :1; 
  } Bits;
  uint32 Reg32;
}  GpioSpiConfig;

typedef struct {
  uint32                              DirLo;                   // (00）
  uint32                              DirHi;                   // (04)
  uint32                              DataLo;                  // (08)
  uint32                              DataHi;                  // (0C)
  uint32                              Reserved1;               // (10)
  GpioSpiConfig                       SpiConfig;               // (14)
  uint32                              Reserved2;               // (18)
  GpioVregConfig                      VregConfig;              // (1C)
  uint32                              Reserved3;               // (20)
  uint32                              Reserved4;               // (24)
  GpioTestControl                     Testcontrol;             // (28)
  uint8                               Pad0[0x14];              // (2C)
  GpioStrapBus                        StrapBus;                // (40)
  GpioStrapOverride                   StrapOverride;           // (44)
  uint8                               Pad1[0x4];               // (48)
  GpioRefPllStatus                    RefPllStatus;            // (4C)
  GpioOscCtl                          OscCtl;                  // (50)
  GpioXtalbufCtrl                     XtalbufCtrl;             // (54)
  GpioDiagMemCtl                      DiagMemCtl;              // (58)
  uint32                              DiagMemSize;             // (5C)
  uint32                              SrcAddr;                 // (60)
  uint32                              DestAddr;                // (64)
  GpioRingOscCtrl0                    RingOscCtrlSet0;         // (68)
  GpioRingOscCtrl1                    RingOscCtrlSet1;         // (6C)
  GpioRbusDiagSel                     RbusDiagSel;             // (70)
  uint32                              DiagCaptLastWrAddr;      // (74)
  GpioMipsDdrPllOverride              MipsDrrPllOverride;      // (78)
  GpioDieRevId                        Dierevid;                // (7C)
  GpioPinMuxSel                       PinMuxSel;               // (80)
  GpioPinMuxSelHi                     PinMuxSelHi;             
  GpioSpimasterControl                SpimasterControl;        
  GpioClkrstMisc                      ClkrstMisc;              
  uint32                              TransportOe;             
  GpioDiagCaptOvflDet                 DiagCaptOvflDet;         
  GpioDiagMemHbCount                  DiagMemHbCount;          
}  GpioRegs;

[danitool]

@rikka0w0
another snippet found in the same file (path: shared/opensource/flash/spiflash.c) from the OEM source code:

#if defined(CONFIG_BCM93380)
	*(unsigned long *)0xb4e00188 = (*(unsigned long *)0xb4e00188 | 1<<19) & ~(1<<16);

    SPI->spiClkCfg = ((2 << 3) | 7);
#endif

Accordingly to the registers you pointed out, it seems it clears the bit 16 and sets the bit 19 at the SpimasterControl , and then it configures the spiclk (probably the speed and also another bit).

[rikka0w0]

Hi, @danitool

I also found the same result last night.

Today, I read the value of all SPI-related registers found in gpio_regs.h and compared their value in the stock bootloader against U-Boot.

These are found in the stock bootloader:

0xb4e00114 = 0x00000809
GpioSpiConfig.SpiMode (bit 0) = 1
SerialAddrCfg = 4b0100
DlyDis = 0
Reserved2 = 6b000010

b4e00128 = 0x00000000
GpioTestControl = 0

0xb4e00140 = 0x807efbe7
GpioStrapBus.BootHsSpimB (bit 8) = 1

0xb4e00184 = 0x55555055
GpioPinMuxSelHi.PmSelectSpisUart1 = 2b01

0xb4e00188 = 0x00080000
GpioSpimasterControl.SpimModeOverride (bit 19) = 1
GpioSpimasterControl.HsSpimEn (bit 16) = 0

0xb4e0018c = 0x00000032
GpioClkrstMisc.SpimClkSel = 0

The only difference happens at 0xb4e00188. In U-Boot, it was all 0. I tried the following in the U-Boot console, and U-Boot can successfully detect the on-board SPI flash:

CG3100D # sf probe
jedec_spi_nor spi-flash@0: unrecognized JEDEC id bytes: ff, ff, ff
CG3100D # mw.l 0xb4e00188 0x0080000 1
CG3100D # sf probe
SF: Detected s25sl064p with page size 256 Bytes, erase size 64 KiB, total 8 MiB

Thanks!

[rikka0w0]

This could be helpful for the ethernet driver development:
https://github.com/RMerl/asuswrt-merlin.ng/blob/5c84a3449e787144a7579982b6198ad956cf1c08/release/src-rt-5.04axhnd.675x/bootloaders/u-boot-2019.07/drivers/net/bcmbca/phy/unimac_drv_impl1.c

https://github.com/RMerl/asuswrt-merlin.ng/blob/5c84a3449e787144a7579982b6198ad956cf1c08/release/src-rt-5.04axhnd.675x/bcmdrivers/opensource/phy/unimac_drv.h#L4

This header seems to match the register layout found in the Netgear Opensourced kernel:
https://github.com/SWRT-dev/asuswrt-bcm/blob/3cf9d517b5bf98115e2b5a5dd7d96f78d7d315a4/release/src-rt-5.04axhnd.675x/bcmdrivers/opensource/phy/unimac_drv.h
https://github.com/SWRT-dev/asuswrt-bcm/blob/3cf9d517b5bf98115e2b5a5dd7d96f78d7d315a4/release/src-rt-5.04axhnd.675x/bcmdrivers/opensource/phy/unimac_drv_impl1.c

[wigyori]

I guess this would work on the Cisco EPC3925 too? That's 3380 also - https://oldwiki.archive.openwrt.org/toh/cisco/epc3925

[rikka0w0]

I guess this would work on the Cisco EPC3925 too? That's 3380 also

Correct. EPC3925 and CG3100D should have very similar hardware design. They may even share the same schematic. I would expect little to none DTS modification to get OpenWrt (with this patch) working on EPC3925.

I just figure out how to do networking on BCM3380. The next step is to implement the ethernet driver in the Linux way...

[rikka0w0]

Okay, guys, I have exciting news to announce!
The ethernet stack is working! I can ping and ssh into the OpenWrt running on CG3100D from my PC!

However, several important features are missing in my driver, including:

1. An Rx FIFO
2. Flow control and back pressure support
3. Interrupt-driven Rx for a lower latency (currently, I'm using polling)

I will provide more information here, just in case someone with better Linux driver development skills than me wants to join and improve the ethernet driver.

The base address of each SoC component can be found here. Its parent folder contains the header files that contain the register definition.

The addressing and memory layout are explained here.

I found that we need three components to make the ethernet work:

1. The Unimac, can be further divided into the interface, the core, the MBDMA, and the Mib block (for statistics).
2. The Free Pool Manager (FPM).
3. The Message Processing Processor (MSP), contains many blocks, but we only need the "Ioproc" block here.

The Unimac core handles most of the MAC configuration stuff, including the MAC address and the maximum frame length. The Unimac interface provides access to the external world, e.g. the MDIO bus. The MBDMA block interacts with FPM and MSP to process incoming and outgoing packets. The FPM manages a circular buffer for packet data. The MSP seems to be a FIFO that holds some 2-byte message. Each message contains a token that describes the length of each packet and its location with in the FPM working memory.

Note that the CPU must access the MSP through the SMISB bus (e.g., use 0xb8801240 instead of 0x15801240 or 0xb5801240 for the Ioproc.In.IncomingMessageFifo.InMsgData register), I have no idea what is SMISB. When configuring Unimac, FPM, and MSP, physical addresses (e.g. 0x15801240) must be used.

The Rx process in the polling implementation is very straightforward. When a packet arrives at the Unimac:

1. [By the hardware] The Unimac requests a token from an FPM Pool with a given length (equal to the received packet size). The token contains the size and the next available space in the FPM working memory (unimac->pFpmMemPhysical and unimac->pFpmMem, allocated during FPM initialization).
2. [By the hardware] The Unimac places the received packet into the FPM working memory at the offset indicated by the token.
3. [By the hardware] The Unimac pushes a message that contains this token along with some kind of state into Ioproc.In.IncomingMessageFifo.InMsgData. The message looks like (found here):

typedef struct LanRxMsg
{
    uint32 msgHdr;
        #define LANRX_MAC_ID_MASK     0x03C00000
        #define LANRX_MAC_ID_SHIFT    22
        #define LANRX_QOS_MASK        0x000f0000
        #define LANRX_QOS_SHIFT       16
    uint32 token;
} LanRxMsg;

If the highest 6 bits are not all zero, the stock bootloader considers the message invalid. I don't know why.

4. The uiEthPoll function is called by a Linux kernel timer periodically. It constantly checks the Fifo not empty flag (the highest bit) of Ioproc.In.IncomingMessageFifo.InMsgSts. Each read to Ioproc.In.IncomingMessageFifo.InMsgData pops one byte from the queue. Two consecutive valid reads indicate a valid LanRxMsg. Then, the token can be obtained. The uiEthPoll now knows the length of the received packet and its location in the FPM working memory.
5. After processing the packet, the token is returned to the FPM by writing it to FpmBlock.FpmPool.Pool1AllocDealloc.

Hence, I think to implement an interrupt-based receiver, we need to handle some interrupt produced by the MSP (Ioproc.In.IncomingMessageFifo).

[rikka0w0]

I'm trying to get more information from the eCos image (CG3100L_V5.5.4_EU_src.tar.bz2 from https://www.downloads.netgear.com/files/GPL/CG3100L_V5.5.4_EU_V1.0.4_Linux_src.zip). My router model is CG3000-1STAUS, but the hardware is identical to CG3100D.

Unfortunately, CG3100L_V5.5.4_EU_src.tar.bz2 wont contain any closed source code:

The eCos license allows commercial users of eCos not to release the code they built on top of eCos so we will miss the actual BFC libraries that are closed source.

However, if we can compile an eCos binary from CG3100L_V5.5.4_EU_src.tar.bz2, it will help us to identify library functions and drastically save us time and effort. IDA's F.L.I.R.T can match functions by their signature.

We would need the .a (static libraries) files to create FLIRT signature file.

Building CG3100L_V5.5.4_EU_src.tar.bz2 is challenging because it relies on ancient software...

Edit: https://github.com/bcm33xx/CG3100L_V5.5.4_EU