Ampere Altra Developer Platform

[geerlingguy]

Basic information

(This thing is... kinda more than a 'board' — but I still want data somewhere, and this is as good a place as any!)

• Board URL (official): https://amperecomputing.com/systems/altra/kraken-comhpc-WS
• Board purchased from: (delivered by Ampere)
• Board purchase date: April 3, 2023
• Board specs (as tested): 96-core / 64 GB RAM / 128 GB SSD
• Board price (as tested): Approx $5000

Linux/system information

# output of `neofetch`
jgeerling@ampere-altra:~$ neofetch
            .-/+oossssoo+/-.               jgeerling@ampere-altra 
        `:+ssssssssssssssssss+:`           ---------------------- 
      -+ssssssssssssssssssyyssss+-         OS: Ubuntu 22.04.2 LTS aarch64 
    .ossssssssssssssssssdMMMNysssso.       Host: Ampere Altra Developer Platform ES2 
   /ssssssssssshdmmNNmmyNMMMMhssssss/      Kernel: 5.19.0-40-generic 
  +ssssssssshmydMMMMMMMNddddyssssssss+     Uptime: 5 mins 
 /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/    Packages: 1565 (dpkg), 11 (snap) 
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Shell: bash 5.1.16 
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Resolution: 1920x1080 
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   Terminal: /dev/pts/1 
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   CPU: (96) @ 2.800GHz 
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   GPU: 0004:02:00.0 ASPEED Technology, Inc. ASPEED Graphics Family 
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Memory: 2102MiB / 63897MiB 
 /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/
  +sssssssssdmydMMMMMMMMddddyssssssss+                             
   /ssssssssssshdmNNNNmyNMMMMhssssss/                              
    .ossssssssssssssssssdMMMNysssso.
      -+sssssssssssssssssyyyssss+-
        `:+ssssssssssssssssss+:`
            .-/+oossssoo+/-.

# output of `uname -a`
Linux ampere-altra 5.19.0-40-generic #41~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Mar 31 16:02:33 UTC 2 aarch64 aarch64 aarch64 GNU/Linux

Benchmark results

CPU

Configured with 96 GB RAM (6 x 16GB DDR4 ECC Registered DIMMs):

• Geekbench: (800 single / 35818 multi - https://browser.geekbench.com/v5/cpu/21323770)
• 1,188.3 Gflops (geerlingguy/top500-benchmark - result)

Power

• Idle power draw (at wall): 64.5 W (85W with 96 GB RAM)
• Maximum simulated power draw (stress-ng --matrix 0): 220 W (242W with 96 GB RAM)
• During Geekbench multicore benchmark: 156 W (178W with 96 GB RAM)
• During top500 HPL benchmark: 296 W (4.01 Gflops/W)

Disk

Transcend 128GB PCIe Gen 3 x4 NVMe SSD (TS128GMTE652T)

Benchmark	Result
fio 1M sequential read	1245 MB/s
iozone 1M random read	1058 MB/s
iozone 1M random write	665 MB/s
iozone 4K random read	72.99 MB/s
iozone 4K random write	246.88 MB/s

curl https://raw.githubusercontent.com/geerlingguy/pi-cluster/master/benchmarks/disk-benchmark.sh | sudo bash

Run benchmark on any attached storage device (e.g. eMMC, microSD, NVMe, SATA) and add results under an additional heading. Download the script with curl -o disk-benchmark.sh [URL_HERE] and run sudo DEVICE_UNDER_TEST=/dev/sda DEVICE_MOUNT_PATH=/mnt/sda1 ./disk-benchmark.sh (assuming the device is sda).

Also consider running PiBenchmarks.com script.

PiBenchmarks.com result: TODO - should be on https://pibenchmarks.com/latest/ soon

     Category                  Test                      Result      
HDParm                    Disk Read                 1533.06 MB/s             
HDParm                    Cached Disk Read          776.86 MB/s              
DD                        Disk Write                407 MB/s                 
FIO                       4k random read            94377 IOPS (377511 KB/s) 
FIO                       4k random write           74202 IOPS (296811 KB/s) 
IOZone                    4k read                   243709 KB/s              
IOZone                    4k write                  198612 KB/s              
IOZone                    4k random read            70575 KB/s               
IOZone                    4k random write           231884 KB/s              

                          Score: 45797  

Network

(Everything runs as expected... this thing's a bonafide server!)

GPU

• TODO: Haven't determined standardized benchmark yet. See Issue #2.

Memory

tinymembench results:

Click to expand memory benchmark result

tinymembench v0.4.10 (simple benchmark for memory throughput and latency)

==========================================================================
== Memory bandwidth tests                                               ==
==                                                                      ==
== Note 1: 1MB = 1000000 bytes                                          ==
== Note 2: Results for 'copy' tests show how many bytes can be          ==
==         copied per second (adding together read and writen           ==
==         bytes would have provided twice higher numbers)              ==
== Note 3: 2-pass copy means that we are using a small temporary buffer ==
==         to first fetch data into it, and only then write it to the   ==
==         destination (source -> L1 cache, L1 cache -> destination)    ==
== Note 4: If sample standard deviation exceeds 0.1%, it is shown in    ==
==         brackets                                                     ==
==========================================================================

 C copy backwards                                     :   9424.0 MB/s
 C copy backwards (32 byte blocks)                    :   9387.8 MB/s
 C copy backwards (64 byte blocks)                    :   9390.8 MB/s
 C copy                                               :   9366.1 MB/s
 C copy prefetched (32 bytes step)                    :   9984.4 MB/s
 C copy prefetched (64 bytes step)                    :   9984.1 MB/s
 C 2-pass copy                                        :   6391.4 MB/s
 C 2-pass copy prefetched (32 bytes step)             :   7237.8 MB/s
 C 2-pass copy prefetched (64 bytes step)             :   7489.6 MB/s
 C fill                                               :  43884.4 MB/s
 C fill (shuffle within 16 byte blocks)               :  43885.4 MB/s
 C fill (shuffle within 32 byte blocks)               :  43884.2 MB/s
 C fill (shuffle within 64 byte blocks)               :  43877.5 MB/s
 NEON 64x2 COPY                                       :   9961.9 MB/s
 NEON 64x2x4 COPY                                     :  10091.6 MB/s
 NEON 64x1x4_x2 COPY                                  :   8171.5 MB/s
 NEON 64x2 COPY prefetch x2                           :  11822.9 MB/s
 NEON 64x2x4 COPY prefetch x1                         :  12123.8 MB/s
 NEON 64x2 COPY prefetch x1                           :  11836.5 MB/s
 NEON 64x2x4 COPY prefetch x1                         :  12122.3 MB/s
 ---
 standard memcpy                                      :   9894.0 MB/s
 standard memset                                      :  44745.2 MB/s
 ---
 NEON LDP/STP copy                                    :   9958.0 MB/s
 NEON LDP/STP copy pldl2strm (32 bytes step)          :  11415.6 MB/s
 NEON LDP/STP copy pldl2strm (64 bytes step)          :  11420.5 MB/s
 NEON LDP/STP copy pldl1keep (32 bytes step)          :  11475.2 MB/s
 NEON LDP/STP copy pldl1keep (64 bytes step)          :  11452.9 MB/s
 NEON LD1/ST1 copy                                    :  10094.8 MB/s
 NEON STP fill                                        :  44744.7 MB/s
 NEON STNP fill                                       :  44745.2 MB/s
 ARM LDP/STP copy                                     :  10136.4 MB/s
 ARM STP fill                                         :  44731.7 MB/s
 ARM STNP fill                                        :  44730.0 MB/s

==========================================================================
== Memory latency test                                                  ==
==                                                                      ==
== Average time is measured for random memory accesses in the buffers   ==
== of different sizes. The larger is the buffer, the more significant   ==
== are relative contributions of TLB, L1/L2 cache misses and SDRAM      ==
== accesses. For extremely large buffer sizes we are expecting to see   ==
== page table walk with several requests to SDRAM for almost every      ==
== memory access (though 64MiB is not nearly large enough to experience ==
== this effect to its fullest).                                         ==
==                                                                      ==
== Note 1: All the numbers are representing extra time, which needs to  ==
==         be added to L1 cache latency. The cycle timings for L1 cache ==
==         latency can be usually found in the processor documentation. ==
== Note 2: Dual random read means that we are simultaneously performing ==
==         two independent memory accesses at a time. In the case if    ==
==         the memory subsystem can't handle multiple outstanding       ==
==         requests, dual random read has the same timings as two       ==
==         single reads performed one after another.                    ==
==========================================================================

block size : single random read / dual random read, [MADV_NOHUGEPAGE]
      1024 :    0.0 ns          /     0.0 ns 
      2048 :    0.0 ns          /     0.0 ns 
      4096 :    0.0 ns          /     0.0 ns 
      8192 :    0.0 ns          /     0.0 ns 
     16384 :    0.0 ns          /     0.0 ns 
     32768 :    0.0 ns          /     0.0 ns 
     65536 :    0.0 ns          /     0.0 ns 
    131072 :    1.3 ns          /     1.8 ns 
    262144 :    2.3 ns          /     2.9 ns 
    524288 :    3.2 ns          /     3.9 ns 
   1048576 :    3.6 ns          /     4.2 ns 
   2097152 :   22.9 ns          /    33.0 ns 
   4194304 :   32.6 ns          /    40.9 ns 
   8388608 :   38.1 ns          /    43.5 ns 
  16777216 :   43.2 ns          /    48.6 ns 
  33554432 :   86.2 ns          /   112.2 ns 
  67108864 :  109.3 ns          /   135.2 ns 

block size : single random read / dual random read, [MADV_HUGEPAGE]
      1024 :    0.0 ns          /     0.0 ns 
      2048 :    0.0 ns          /     0.0 ns 
      4096 :    0.0 ns          /     0.0 ns 
      8192 :    0.0 ns          /     0.0 ns 
     16384 :    0.0 ns          /     0.0 ns 
     32768 :    0.0 ns          /     0.0 ns 
     65536 :    0.0 ns          /     0.0 ns 
    131072 :    1.3 ns          /     1.8 ns 
    262144 :    1.9 ns          /     2.3 ns 
    524288 :    2.2 ns          /     2.5 ns 
   1048576 :    2.6 ns          /     2.8 ns 
   2097152 :   21.6 ns          /    31.6 ns 
   4194304 :   31.1 ns          /    39.4 ns 
   8388608 :   35.8 ns          /    41.7 ns 
  16777216 :   38.5 ns          /    43.0 ns 
  33554432 :   79.9 ns          /   104.9 ns 
  67108864 :  101.1 ns          /   125.4 ns 

[geerlingguy]

Also tested with Phoronix:

• pts/build-linux-kernel: https://openbenchmarking.org/result/2304170-NE-LINUXCOMP70 (116.59 seconds)
• pts/phpbench: https://openbenchmarking.org/result/2304174-NE-AMPEREALT14 (498149)
• pts/compress-7zip: https://openbenchmarking.org/result/2304174-NE-AMPEREALT36 (161132 / 337214)

[geerlingguy]

Tried to install Windows 11 and it's been quite the saga. Haven't been able to get an ISO that boots yet:

• Tried https://uupdump.net, multiple configurations and using canary, dev, and beta releases, best I could get is it booting to the windows logo with an Exception appearing on screen and locking up
• Tried https://uup.rg-adguard.net, same thing
• Tried downloading Windows Developer Kit 2023 recovery image (had to dig up my serial number), copying that on top of a Recovery Media usb stick, that didn't work (it wouldn't boot at all... let me select the drive in BIOS but would immediately kick back to BIOS)
• Tried installing Parallels Desktop for Mac and having it install Windows 11 — it did, creating an ISO that looked like a similar layout to the ones from UUPDump, and it actually booted for like 3 seconds longer, and instead of an exception printed to screen, I got the sad face blue screen of death
• I've also tried more incantations with UUPDump, and older versions, like the one linked from the UTM Docs, but those didn't work either.

I know at least Windows 11 Pro can be run on these chips:

• Evidence: Techcrunch article mentions it running virtualized in Azure on Ampere
• Evidence: @JoeSpeeds shows it running Windows 11 in this Twitter post
• I-Pi Development Kits YouTube video - "All you have to do is—it's, uh, USB bootable, you turn the power on..." — yeah but how did you get that USB bootable drive? :D

I hate knowing something is possible but not being able to do it lol.

Edit: After a ton of searching I also found this article: Ampere ALTRA – Industry leading ARM64 Server — it says you can use qemu-image to convert a vhdx to a raw disk image file (qemu-img convert -O raw Windows11_InsiderPreview_Client_ARM64_en-us_25201.VHDX Windows11_InsiderPreview_Client_ARM64_en-us_25201.raw), so I may see if that's possible. If I can get that VHDX file... grr...

[geerlingguy]

On my Mac, I downloaded a copy of the VMDK file kindly provided by DB Tech. Then I installed qemu with brew install qemu, and ran:

qemu-img convert -O raw Windows11_InsiderPreview_Client_ARM64_en-us_25324.VHDX Windows11_InsiderPreview_Client_ARM64_en-us_25324.raw

I formatted my Corsair USB stick (128 GB) as ExFAT and copied over the .raw file (it's like 70 GB!).

I booted up Ubuntu on the Ampere Altra, and am copying the .raw file contents over to an unused NVMe SSD:

sudo dd if=/media/jgeerling/Untitled/Windows11_InsiderPreview_Client_ARM64_en-us_25324.raw of=/dev/nvme0n1

Note: on this system, the second drive has Ubuntu installed (/dev/nvme1n1). If you're coming from Google and trying to copy and paste... make sure you don't blow away an installation by copying over it on nvme0n1 lol.

The copy took about 24 minutes (yowza!) at 48.7 MB/sec. Rebooting now...

[geerlingguy]

It sounds like according to some, scaling beyond 32 or 64 cores might break with Geekbench (possibly due to some internal configuration), so another way to game that system is to lock an instance of Geekbench 5 to each core individually, e.g.:

for ((i=0; i<60; ++i)); do numactl -C $i ./geekbench5 --cpu & >> "gkbch"$i 2>&1; done

The core count could be more efficient at a higher or lower number (accounting for overhead) than the 60 above. Also, install numactl with sudo apt install -y numactl.

The 30,000 score does seem low even accounting for overhead losses, compared to an 800 or so single core.

Running it individually on 60 cores, I ended up locking up the machine a bit, but it did seem to be progressing nicely through all the single-core tests. I'll try again later.

[geerlingguy]

Drat, the Windows boot results in a sad face blue screen of death with ACPI_BIOS_ERROR.

[geerlingguy]

To get an Nvidia Quadro RTX 8000 working, I ran sudo apt install libglvnd-dev pkg-config, then installed Nvidia's latest aarch64 driver for the Quadro RTX 8000 (running the .run file with sudo).

I can get full resolution video output through DisplayPort on the card, and Ubuntu (and neofetch) sees the card, alongside lshw. But:

$ nvidia-smi
No devices were found

$ sudo dmesg | tail
[  276.229559] NVRM: GPU 000d:01:00.0: RmInitAdapter failed! (0x24:0x65:1423)
[  276.229665] NVRM: GPU 000d:01:00.0: rm_init_adapter failed, device minor number 0
[  280.597471] NVRM: GPU 000d:01:00.0: RmInitAdapter failed! (0x24:0x65:1423)
[  280.597541] NVRM: GPU 000d:01:00.0: rm_init_adapter failed, device minor number 0
[  466.107317] audit: type=1107 audit(1681788653.426:93): pid=1779 uid=102 auid=4294967295 ses=4294967295 subj=unconfined msg='apparmor="DENIED" operation="dbus_signal"  bus="system" path="/org/freedesktop/login1" interface="org.freedesktop.login1.Manager" member="SessionNew" name=":1.12" mask="receive" pid=7171 label="snap.firefox.firefox" peer_pid=1858 peer_label="unconfined"
                exe="/usr/bin/dbus-daemon" sauid=102 hostname=? addr=? terminal=?'
[  473.623874] NVRM: GPU 000d:01:00.0: RmInitAdapter failed! (0x24:0x65:1423)
[  473.623963] NVRM: GPU 000d:01:00.0: rm_init_adapter failed, device minor number 0
[  477.995897] NVRM: GPU 000d:01:00.0: RmInitAdapter failed! (0x24:0x65:1423)
[  477.995979] NVRM: GPU 000d:01:00.0: rm_init_adapter failed, device minor number 0

In the Nvidia dev forums, I found this post which states:

The nvidia driver doesn’t like your board.
...
This is a known and recurring bug in the nvidia driver, the known working versions list is based on reports from other users.

So... looks like it's off to the races to find a working driver!

I also opened an issue on the Ampere Developers site: GPU Support for Ampere Altra?.

[geerlingguy]

Er... apparently I still had the VGA plugged into my monitor, so the DisplayPort output wasn't actually working. D'oh! Well, we'll see what others say. I might have better luck compiling Linux 6.2 with AMD's latest driver support and my RX 6700 XT. We'll see!

[istori1]

I have been enabling aarach64 support of NVIDIA on Flathub and some apps. When you get the NVIDIA GPU running there are couple of apps you can run.

flathub/org.freedesktop.Platform.GL.nvidia#148
flathub/org.freedesktop.Sdk.Extension.llvm15#10
flathub/org.kde.kdenlive#207

https://github.com/flathub/org.jellyfin.JellyfinServer

https://github.com/flathub/com.dec05eba.gpu_screen_recorder

https://github.com/LizardByte/Sunshine/blob/master/packaging/linux/flatpak/dev.lizardbyte.sunshine.yml#L195

[ThomasKaiser]

The 30,000 score does seem low even accounting for overhead losses, compared to an 800 or so single core.

Here someone tests an 80 core Ampere Altra machine, obviously with more reliable power monitoring and generating higher GB5 scores than your machine (882/444285): https://youtu.be/m6-juFXR9c0?t=560

All those 80 core benchmarks show a higher single-threaded score than your machine but multi differs drastically (33000 - 46500): https://browser.geekbench.com/search?page=1&q=MP32-AR1-SW-HZ-001

Why do these scores differ? Obviously nobody knows and nobody cares (at least all scores exceeding 40000 have 256 GB RAM in common)

[ThomasKaiser]

BTW: could you please share output from this command:

cat /sys/devices/system/cpu/cpu*/topology/physical_package_id | sort | uniq -c

I'm curious since a dual socket Ampere Altra Max system showed different package IDs for cpu0 - cpu8 but then identical package ID for all the remaining 247 cores. Wondering whether here it's the same (affects sbc-bench -j mode that is designed to help explaining why Geekbench scores differ)

[geerlingguy]

Why do these scores differ? Obviously nobody knows and nobody cares (at least all scores exceeding 40000 have 256 GB RAM in common)

Note that I was told by the folks at Ampere I should fill all six RAM slots in the machine for better scores, as the NUMA layout on the Ampere requires all channels to be full for best performance. Whether that affects Geekbench scores in particular remains to be seen, but judging by STH's testing on Epyc and Ampere, it seems that's a reasonable outcome.

I'll soon be ordering two more 16GB sticks of RAM to match what's already installed. Right now I have the Ampere workstation back in a box because I'm trying to wrap up another project that's taking up the majority of the space in my little workshop :( — I hope to get it back out in mid May.

[ThomasKaiser]

as the NUMA layout on the Ampere requires all channels to be full for best performance

Oh, now with you mentioning NUMA I realize that there's no Ampere Altra with 96 cores so your box combines an AADP-64 in one CPU socket with an AADP-32 in the other? Unfortunately the 'info' collected by neofetch lacks most interesting bits.

Anway: once you unpack your box again I would really love to see sbc-bench -G (executing Geekbench in monitored environment) or sbc-bench -j and after the initial measurements then executing Geekbench in another Terminal. Since the tool has been designed to also provide hints as why scores differ (could be as simple as 'excessive swapping' since Geekbench needs an insane amount of memory per thread)

[geerlingguy]

This chip is the M96-28, which is a 96-core part running at 2.8 GHz. I will try to get the machine up and running this week if I get a chance, and I'll give you more info. But might not be able to until may again.

[geerlingguy]

It looks like their EDK2 fork just got a fix for the Windows BSOD I was running into (apparently it should've already worked on the 32/64/80 core Altra parts, but it needed fixing on Altra Max (96/128 core parts). Waiting for a new version of the firmware to be available and I'll test it out again!