aarch64 bcm2711_defconfig fails to get to early console

[hvenev]

aarch64 bcm2711_defconfig with CONFIG_BCM_VIDEOCORE disabled (see #3031) fails to get to early console with the kernel command line earlycon=pl011,0xfe201000 console=ttyAMA0,115200 root=/dev/mmcblk0p2 rootfstype=f2fs rootwait. The kernel7l.img provided in https://github.com/raspberrypi/firmware with the same kernel command line and firmware boots just fine (and indeed starts printing to the console earlier than when earlycon is not set).

[hvenev]

This happens both on current rpi-4.19.y (e8a66b4) with firmware from raspberrypi/firmware@bcf40b5 and raspberrypi-kernel_1.20190620-1 (883a1a5) with firmware from 1.20190620+1 (raspberrypi/firmware@88ca908).

[pelwell]

2711 v8 support is a work in progress. In particular, the v8 ARM stubs need GIC support.

[hvenev]

Isn't early console initialized before the GIC?

[pelwell]

Perhaps, but the L2 cache latencies need to be set before either of them.

[hvenev]

OK... How do I reproduce the results from cdb78ce? Is it enough to set total_mem=1024? Also, where are the L2 cache latencies set for 32-bit arm?

[agherzan]

@pelwell I see that raspberrypi/tools@7f4a937#diff-d9d684ceaf48229906f8a534b4cac276 includes gic support for v8 arm stubs. Isn't this included?

[pelwell]

It's not built into the firmware yet, but if you build it yourself and add armstub=... to config.txt it should work.

[agherzan]

That was what I suspected. Thanks. I"ll try that. Are there any other known issues even with this stub added?

[pelwell]

Not that I'm aware of, but as you can tell this isn't one of the use cases we tested pre-launch. The two related config.txt settings are arm_64bit and enable_gic: the former should be set automatically if kernel8.img is loaded, and the latter is set if the dtb includes an enabled GIC node.

[diegosueiro]

@agherzan and @pelwell ,

I can confirm that the 64-bit kernel boots having the following in the config.txt:

kernel=kernel8.img                                                              
arm_64bit=1
device_tree=bcm2711-rpi-4-b.dtb
enable_gic=1                                       
armstub=armstub8-gic.bin

I used Yocto and meta-raspberrypi (master branches) with some additions to have a raspberrypi4-64 machine and building the kernel and using the firmwares in the head of its branches (rpi-4.19.y and master).

The tools/armstubs I had to compile separately to produce the armstub8-gic.bin.

[agherzan]

Hi @diegosueiro. You'll have this in meta-raspberrypi master today. Also, I don't think you need the device tree set. Also I'd expect kernel8 to be picked up automatically too.

[diegosueiro]

@agherzan,

I was just extra cautious with the settings. 😉

[agherzan]

@diegosueiro have you limit the total memory as well?

[diegosueiro]

@agherzan, as far as I remember I haven't touch any memory configuration.

[agherzan]

@diegoherranz Do you happen to have a 1G board? Also, when you tested, did it mount the rootfs and boot the distro?

[diegosueiro]

@agherzan, yes I'm using the 1GB and I have the full system booted. But I'm using systemd instead of sysvinit.

I can send you my changes later.

[agherzan]

@pelwell Do you have any idea why kernel will hang at waiting for the rootfs?

[    2.246724] Waiting for root device /dev/mmcblk0p2...
[    2.260587] mmc1: queuing unknown CIS tuple 0x80 (2 bytes)
[    2.267794] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
[    2.275006] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
[    2.283493] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
[    2.290703] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
[    2.339258] random: fast init done
[    2.361344] mmc1: new high speed SDIO card at address 0001

[pelwell]

Is this with >1GB? The SDHCI controller only sees the bottom 1GB of RAM. There may be a way of telling Linux about this restriction, but at the time I played with the 64-bit kernel I couldn't make it work without artificially limiting the RAM.

[agherzan]

@pelwell It is indeed on a 4G. When you say that you only made it work with limiting the ram, did you use total_mem=1024? Or have you limited in linux as well?

[pelwell]

I used total_mem=1024, which is propagated to Linux via the Device Tree memory setting.

[agherzan]

That's what I assumed but sadly I get this with that setting on 1024.

[rfinnie]

That's what I assumed but sadly I get this with that setting on 1024.

Confirmed same on my Pi 4 2GB ("mmc1: queuing unknown CIS tuple 0x80"), with bcm2711_defconfig. total_mem=1024 doesn't seem to have an effect on MMC. I also tried putting the rootfs on a USB flash drive, which it does find:

[    2.092135] Waiting for root device UUID=ab214f01-fe36-4818-8450-ad7c795f0f42...
...
[    3.948868] scsi 0:0:0:0: Direct-Access     Samsung  Flash Drive      1100 PQ: 0 ANSI: 6
[    3.957703] sd 0:0:0:0: [sda] 125313283 512-byte logical blocks: (64.2 GB/59.8 GiB)
[    3.965639] sd 0:0:0:0: [sda] Write Protect is off
[    3.978096] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    5.398480]  sda: sda1
[    5.402732] sd 0:0:0:0: [sda] Attached SCSI removable disk

but doesn't seem to mount as root despite e.g. CONFIG_USB_STORAGE being included. I haven't debugged beyond that. Edit: passing root=/dev/sda1 works. Huh.

[sakaki-]

Thanks all for the hints in this thread. Have used it to bring the gentoo-on-rpi3-64bit userland up on an RPi4 (brief notes here).

Screenshot:

[pelwell]

Nice work, and thanks for sharing!

If anyone figures out how to persuade the SD interface to work in a 2GB or 4GB system before we get round to it then please share for the good of all Pi 4 users. The solution will be a software configuration change that enables software bounce buffers rather than finding a hardware switch - there is no magic bit that will create the missing address lines.

[github-usr-name]

@j-selby Based on lines 3558-3572 of sdhci.c, I believe that @pcwalton is correct; the quirk flags disable DMA and revert to PIO:

   3558     if ((host->quirks & SDHCI_QUIRK_BROKEN_DMA) &&
   3559         (host->flags & SDHCI_USE_SDMA)) {
   3560         DBG("Disabling DMA as it is marked broken\n");
   3561         host->flags &= ~SDHCI_USE_SDMA;
   3562     }
   3563 
   3564     if ((host->version >= SDHCI_SPEC_200) &&
   3565         (host->caps & SDHCI_CAN_DO_ADMA2))
   3566         host->flags |= SDHCI_USE_ADMA;
   3567 
   3568     if ((host->quirks & SDHCI_QUIRK_BROKEN_ADMA) &&
   3569         (host->flags & SDHCI_USE_ADMA)) {
   3570         DBG("Disabling ADMA as it is marked broken\n");
   3571         host->flags &= ~SDHCI_USE_ADMA;
   3572     }

[j-selby]

@github-usr-name my understanding is that it falls back anyway, effectively just hiding the error logging. Besides, disabling DMA is obviously not a long term fix.

[github-usr-name]

@j-selby We agree that disabling DMA [for the SDHC] is not a long term fix, but as a short-term workaround it's better than nothing IMO and results in a less broken system. Limiting the DMA address space is clearly the right solution, but I lack the knowledge to do that.

Respectfully, your understanding doesn't match the code - refer to the snippet above. When the quirk is in effect the kernel never tries to use DMA; without it, DMA is attempted for each and every access before falling back to PIO. As a result, when sdhci.debug_quirks=96 is added to cmdline.txt, the machine boots & operates far quicker. Undesirable? Yes. But also better than without the quirk. My 2Gb system boots in ~6seconds c.f. ~2mins without it.

[j-selby]

@github-usr-name well, obviously there are performance improvements from not attempting an invalid operation continuously - I never said that that wasn't the case. There are artificial delays added with the console spamming going on. My point was that a lot of the solutions posed already are all achieving the same thing fundamentally.

[sakaki-]

Slightly OT (apologies), but I have just released a pair of weekly autobuilds for the bcm2711_defconfig kernel for the RPi4 (GitHub projects here and here). The first provides a 'vanilla' defconfig build; the second is slightly tweaked (to turn on KVM etc). Although intended for use in my 64-bit Gentoo image, the created release tarballs should be serviceable for any appropriate 32-bit or 64-bit userland (arch etc).

Each kernel release tarball currently provides the following files:

• /boot/kernel8-p4.img (this is the bootable 64-bit kernel);
• /boot/COPYING.linux (the kernel's license file);
• /boot/config-p4 (the configuration used to build the kernel);
• /boot/System-p4.map (the kernel's symbol table);
• /boot/bcm2711-rpi-4-b.dtb (the device tree blob; currently only one);
• /boot/armstub8-gic.bin (stubs required for the GIC);
• /boot/overlays/... (the device tree blob overlays);
• /lib/modules/<kernel release name>/... (the module set for the kernel).

Full details on the project pages linked above.

NB: these prebuilt kernels are provided as a convenience only, in the hope they will be useful, but without warranty etc. Use at your own risk! Given that the releases in these projects are created automatically, there is no guarantee that any given kernel will boot correctly. You need up-to-date boot firmware on your target image also, for these to work.

Best, sakaki

PS equivalent autobuilds for the bcmrpi3_defconfig kernel may be found here and here, for those interested.

[hvenev]

This should have been closed ages ago.