Repository navigation
AD103 x2 (RTX 4080 SUPER, 10de:2702) - auto-boot from internal NVMe, with real LLM prefill/decode before/after #104
Replies: 4 comments
我在我的另一个工具 https://github.com/Skyworks-Neo/nvoc 中的cli-stressor-cuda-rs中也有int8路径,那里的压力测试调用cublas动态链接库执行,使用gemm kernel之后自动路由到的是纯FP32拼接,所以他和纯FP32一样,没有提升 |
您要考虑到,这是为了大多数用户安全性的设计。专业用户显然了解任何一个fat32,只要能在UEFI DXE环境下被索引到块设备,就能作为文件系统被识别,进而运行里面的可执行EFI程序启动。但对于大部分普通用户而言,他们的认知就是“我系统引导分区又炸掉了,你这个东西让我电脑起不来了就是你的问题”,所以我们的策略是: 还有的情况是,用户内部的ESP分区大小不够,这个时候就需要缩容,以您的设备为例,NTFS的缩容是比较简单的。但真实情况下,用户可能会有双系统,多系统,跨盘引导项,根本不支持冷热缩容的Linux神秘高级文件系统(btrfs,zfs等),我们作为一个工具,强行囊括所有用户的所有存储设备配置情形并提供缩容方案,不现实且具有较大数据安全风险——数据无价! FAQ补充说明:完全认可你的补充,会加入readme,同时将上述策略也公开提示。 |
|
Recorded — thank you for this exceptional report (ed871b7). What landed in the docs:
One ask: you offered the full 76 KB |
|
Thanks — and congratulations on the Evidence attached for https://gist.github.com/Scorp1o117/dc87b47ce27b00e084374598bbc43236 13 files, 392 KB total:
Raw and unedited. I scanned the log for hostnames, serials, MACs and IPs before publishing — it is firmware/PCI registers and loader state only. On INT8: your answer closes the loop. Locked INT8 at 92.6 TF sitting below locked FP16 at 179.7 TF never read like an INT8 tensor-core number. Noted. On the no-automation-script policy: agreed, and your reasoning is the better one — I was thinking about whether the technique works, you are thinking about who ends up holding the bag when it does not. The btrfs/zfs point is fair; my target happened to be plain NTFS, which is the easy case. 数据无价 is the right default. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
AD103 ×2 · Dual RTX 4080 SUPER (
10de:2702) — auto-boot from internal NVMe, with real LLM prefill/decode numbersSecond AD103 desktop data point, and as far as I can tell the first one that reports real large-model inference throughput before and after, not just GEMM microbenchmarks.
Hardware / platform
10de:2702, 80 SM, 32 GiB each95.03.44.00.02llama-server(Qwen3-VL-Embedding-2B) resident on the same cards during every measurementTool v2.1.8, assets regenerated locally with
make-usbroot.ps1 -Family ad102, all SHA-256 verified. Seven of the eight artifacts matched the hashes published indocs/ada28e0-uefi-report.mdbyte-for-byte; onlyBOOTX64.EFIdiffers, and it matches the releaseSHA256SUMS.txt(f9f3665c…), as expected for the newer build.Deployment: internal NVMe partition, no USB, fully automatic
Worked through a few dead ends before landing here — posting them because I suspect others will hit the same ones.
What did not work
bcdedit /create /application osloaderproduces a Windows Boot Loader entry whosedeviceispartition=S:— a drive letter that does not survive a reboot — and firmware never enumerates it.bcdedit /create /application bootsectorproduces a Real-mode Boot Sector entry, and thedevicefield silently stays unset.{fwbootmgr} displayorder, and neither appears in the F11 one-time boot menu.What worked
Shrink an internal NVMe data volume by 2 GB, create a FAT32 partition, and lay it out exactly like the USB stick:
Then in BIOS, move the partition (shown as
UEFI OS) to the top of the boot priority.Note: the
.binassets and the arm marker go in the volume root, not next toBOOTX64.EFI.loader/src/tx.rsreads them withget_image_file_system(image_handle())+open_volume()+\name— the root of whichever volume the image booted from. Putting them next to the binary would not be found.Result: every boot auto-runs the transaction with zero keystrokes, then chainloads
\EFI\Microsoft\Boot\bootmgfw.efi. Cold boot cost is ~17–22 s.Transaction result (both cards, one pass)
Both cards:
FEAT_PLM 0xffffff8f → 0xffffffff, selectors written and surviving FLR.tu10x_*reads0xbadf5040on AD103 as expected.Windows came up 2 s after the log closed — drivers 617.42,
CM_PROB_NONE, no Code 43, no Xid.Measured compute (cuBLAS dense GEMM, this machine)
Peak of 3 runs × 25 iterations after 3 warmups, per card. Values are TFLOPS.
Two observations that might be useful to the matrix:
1. FP16/FP16 accumulate is unchanged. It was already at full speed when locked (179.7 TF), so it acts as a control: the ~1.8× on the other three paths cannot be an environment or clock artefact.
2. INT8 is unchanged too, and it is slower than FP16. Locked INT8 (92.6 TF) sits below locked FP16 (179.7 TF), the opposite of the usual "INT8 = 2× FP16" expectation; unlocking moves it by 1%. I have not investigated the kernel-level reason and am only reporting the observation — if anyone knows whether cuBLAS routes
CUBLAS_COMPUTE_32Ioff the path this tool frees, I would like to hear it.The operating point barely moved, which is the interesting part: locked vs unlocked full-load SM clock was 1.86–2.17 GHz both times, power 243–285 W (320 W limit), 55–60 °C. So the gain is not bought with clocks or watts — consistent with the project's "not an overclock" framing.
Real LLM inference: what the unlock is actually worth
This is the part I could not find community numbers for. GEMM microbenchmarks say 1.75×; an actual model is a different story.
Setup: Qwen3.8-Flash-Next IQ3_S (GSQ-RCO abliterated, 2 shards, 78 GB), Strata engine 0.1.55, context 262144,
--kv int8,--kv-resident 32768, MTP--spec 4, expert cache auto, both GPUs, layer split auto. Expert weights ≈46.84 GiB in RAM + 12.5k experts / 22.8 GiB in VRAM.Protocol: temperature 0, seed 42, thinking disabled, warmup + 3 rounds × 2 independent runs (n=6 per metric), all samples
cache_n=0, frozen 150 KB input document, byte-identical bench script on both sides.The 132 K result is the cleanest: locked runs spanned 4,956–4,994 (0.8% spread), unlocked spanned 5,822–5,989 — the two groups do not overlap at all.
Takeaway: ~1.19× on prefill and ~1.05× on decode from a 1.75× GEMM ceiling. The gap is expected — expert routing, KV traffic, PCIe and CPU-side scheduling do not benefit from the emission-rate unlock — but it is the number a user actually gets, and it is a fair bit smaller than the microbenchmark suggests.
Also worth noting for anyone benchmarking this model on Ada: 44 K prefill is consistently slower than 132 K prefill (1.26–1.29× ratio), both before and after unlock. So it is a property of the runtime at that chunk size, not something the unlock introduced.
Summary for the matrix
10de:2702(4080 SUPER) works: two cards, one pass, both verified end to end.loader.log(76 KB) and the raw benchmark JSON if the maintainers want them.Two questions for the maintainers, if I may:
bcdeditcannot create the firmware entry would save the next person a few hours.AD103 ×2 · 双 RTX 4080 SUPER(
10de:2702)— 内置 NVMe 全自动启动 + 真实 LLM prefill/decode 实测第二个 AD103 桌面数据点;据我所知,也是第一个同时给出真实大模型推理吞吐前后对比、而不只是 GEMM 微基准的。
硬件与平台
10de:2702,80 SM,各 32 GiB95.03.44.00.02llama-server(Qwen3-VL-Embedding-2B)工具版本 v2.1.8,资产用
make-usbroot.ps1 -Family ad102本地再生,SHA-256 全部校验。八个产物里有七个与docs/ada28e0-uefi-report.md公布的哈希逐字节一致;只有BOOTX64.EFI不同——那是新版本,与发布包SHA256SUMS.txt(f9f3665c…)相符。部署方式:内置 NVMe 分区,不用 U 盘,全自动
走了几条死路才到这里,一并写出来——我猜别人也会踩。
走不通的做法
bcdedit /create /application osloader会生成一个 Windows Boot Loader 条目,device落成partition=S:(一个重启就不存在的盘符),固件根本不会枚举它。bcdedit /create /application bootsector会生成 Real-mode Boot Sector,device字段直接为空,/set device静默失败。{fwbootmgr} displayorder,也都不出现在 F11 一次性启动菜单里。走通的做法
在一块内部 NVMe 数据盘尾部收缩出 2 GB,建 FAT32 分区,布局与 U 盘完全一致:
然后在 BIOS 里把这个分区(显示为
UEFI OS)调到启动顺序第一位。注意:
.bin资产和 arm 标记要放卷根,不是BOOTX64.EFI旁边。loader/src/tx.rs用的是get_image_file_system(image_handle())+open_volume()+\name,也就是「镜像启动所在那卷的根目录」。放同目录是找不到的。效果:每次开机零按键自动跑事务,然后链回
\EFI\Microsoft\Boot\bootmgfw.efi。冷启动额外耗时约 17–22 秒。事务结果(两卡,单次遍历)
两张卡都是
FEAT_PLM 0xffffff8f → 0xffffffff,选择器写入并经受住 FLR。AD103 上tu10x_*读作0xbadf5040属预期。日志关闭 2 秒后 Windows 启动——驱动 617.42,设备管理器
CM_PROB_NONE,无 Code 43,无 Xid。算力实测(cuBLAS dense GEMM,本机)
每卡取 3 轮 × 25 次迭代的峰值,预热 3 次。单位 TFLOPS。
两点或许对矩阵有用:
1. FP16/FP16 累加完全没变。 它在未解锁时就已经跑满(179.7 TF),因此可以当作对照组:另外三条路径的约 1.8× 增益不可能是环境或时钟造成的伪影。
2. INT8 也没变,而且比 FP16 更慢。 未解锁时 INT8(92.6 TF)低于未解锁的 FP16(179.7 TF),这与通常「INT8 = 2× FP16」的预期相反;解锁只让它动了 1%。我没有深究内核层面的原因,只报告现象——如果有人知道 cuBLAS 的
CUBLAS_COMPUTE_32I是否根本不走本工具释放的那条通路,我很想听听。运行点几乎没动,这才是关键:解锁前后满载 SM 时钟都是 1.86–2.17 GHz,功耗 243–285 W(上限 320 W),温度 55–60 °C。所以收益不是靠频率或瓦数换来的,与项目「不是超频」的定位一致。
真实 LLM 推理:解锁到底值多少
这是我在社区里找不到数据的那部分。GEMM 微基准说 1.75×,真实模型是另一回事。
配置:Qwen3.8-Flash-Next IQ3_S(GSQ-RCO abliterated,2 分片,78 GB),Strata 引擎 0.1.55,上下文 262144,
--kv int8,--kv-resident 32768,MTP--spec 4,专家缓存 auto,双卡,层切分 auto。专家权重约 46.84 GiB 在内存 + 12.5k 专家 / 22.8 GiB 在显存。协议:temperature 0、seed 42、关闭思考,预热 + 3 轮 × 2 次独立运行(每指标 n=6),全部样本
cache_n=0,冻结 150 KB 输入文档,两侧使用逐字节相同的测量脚本。132K 那组最干净:未解锁 6 次落在 4,956–4,994(极差 0.8%),解锁后落在 5,822–5,989,两组完全不重叠。
结论:1.75× 的 GEMM 天花板,到 prefill 只剩约 1.19×、decode 只剩约 1.05×。差距符合预期——专家路由、KV 访存、PCIe、CPU 侧调度都不吃发射率解锁——但这是用户真正能拿到的数字,比微基准小不少。
另有一个给在 Ada 上测这个模型的人的提醒:44K prefill 一直比 132K 慢(比值 1.26–1.29×),解锁前后都一样。所以这是该运行时在那个 chunk 规模下的固有特性,不是解锁引入的。
给矩阵的结论
10de:2702(4080 SUPER)可用:两卡、单次遍历、全链读回验证通过。loader.log(76 KB)或原始基准 JSON,我很乐意提供。如果有机会,想向维护者提两个问题:
bcdedit造不出固件启动项」,能替下一个人省下几个小时。All reactions