ttkmd-2.10.0

Tenstorrent PCIe Driver Version 2.10.0 Release Notes

This release adds exclusive-open (O_EXCL) arbitration of the device character file so tools like tt-flash can fence off a device for the duration of a flash-and-reset sequence, introduces a new ioctl that exports a TLB window as a dma-buf for peer-to-peer PCIe DMA (e.g. from an RDMA NIC into Blackhole memory), and tears down userspace MMIO mappings when a device is removed.

Warning

An exported window lets another device DMA into the chip. If the chip's NOC hangs while such DMA is in flight, outcomes observed in testing range from silently dropped writes (the initiating NIC sees posted writes complete and reports no error) up to an uncorrectable PCIe error escalating to a host machine-check and reboot, depending on how the endpoint misbehaves on the bus. The driver does not detect a hung NOC and does not intervene.

---

New Features

Exclusive Device Open (O_EXCL)

open() of a device file now behaves as a reader/writer lock arbitrated at open time, so a tool can guarantee no other process is using the device before it begins and that no new openers can interpose once it has started. O_EXCL is the writer, plain opens are readers, and O_NONBLOCK selects trylock behavior:

• open(O_RDWR | O_EXCL) blocks until the device is idle, then takes exclusive ownership.
• open(O_RDWR | O_EXCL | O_NONBLOCK) returns -EAGAIN if the device is not immediately idle.
• open(O_RDWR) blocks while an O_EXCL fd is held, then proceeds once it is released.
• open(O_RDWR | O_NONBLOCK) returns -EAGAIN while an O_EXCL fd is held.

The resetter's fd now stays valid across the reset_gen bumps performed by CONFIG_WRITE, USER_RESET, ASIC_RESET, and ASIC_DMC_RESET, so an entire flash-then-reset sequence (ending in POST_RESET) can run from a single O_EXCL fd without a close/reopen window that would drop and re-acquire exclusivity. Other fds still observe the bump and continue to fail with -ENODEV.

A blocking O_EXCL writer can be starved by a steady stream of plain opens, since it only wakes on the transition to an idle device; callers that cannot tolerate starvation should use O_NONBLOCK with retry. Readers are not starved — they wake deterministically when the single O_EXCL holder releases.

Export TLB Window as a dma-buf (Experimental Peer-to-Peer DMA)

A new ioctl TENSTORRENT_IOCTL_EXPORT_TLB_DMABUF wraps a window previously allocated with TENSTORRENT_IOCTL_ALLOCATE_TLB in a dma-buf and returns a file descriptor for it. The fd can be handed to another device's driver — e.g. an RDMA NIC via ibv_reg_dmabuf_mr() — so that device can perform peer-to-peer PCIe DMA into and out of the window, which routes onto the NOC according to the window's CONFIGURE_TLB setup. (Requires Linux 5.8 or later; the ioctl returns -EOPNOTSUPP on older kernels, where the rest of the driver continues to build and run.)

• The exported region is device BAR, not host memory. The importer must be capable of peer-to-peer DMA, and the PCIe topology between the two devices must support routing traffic directly between them. The driver does not validate the path.
• A live export pins its window. The window stays out of the allocation pool until both the owning fd and every export have released it, so it cannot be re-allocated to a different client while an importer still holds a mapping.
• Destructive resets are refused with -EBUSY while any export is live. Resetting the chip under in-flight peer-to-peer writes has been observed to make the endpoint emit a fatal Malformed TLP; on platforms with firmware-first error handling (APEI/GHES) this escalates to an unconditional kernel panic that requires a power cycle, and Linux-side AER settings cannot prevent it. There is deliberately no override. To unblock the reset — including on a wedged chip — tear down the importer first: closing the NIC's memory registration releases the dma-buf. (RESTORE_STATE and POST_RESET, the re-init halves of a reset sequence, are not blocked.)
• The -EBUSY guard covers only the driver's own reset ioctl. Resets the driver cannot intercept — one initiated out-of-band from the BMC, or the firmware auto-reset watchdog firing on a hung device — are not blocked and carry the full hazard. In particular, a NOC hang while an export is live can trip the watchdog into an unguarded reset-under-DMA with no host software involved.
• Suspend and removal cannot be refused, so on those paths the driver revokes exported mappings instead, after which further maps fail. Revocation only works on importers that support on-demand paging (e.g. mlx5); on a pin-only importer it is inert — the NIC refuses the invalidate callback and the mapping stays live. The application is responsible for stopping all DMA into a window before the device is removed, suspended, or reset by any path the driver does not control.

Note

This is an experimental capability under active development. We welcome bug reports as teams integrate against real hardware.

A new test/demo tool, tools/rdma/nic_bh_p2p_dma, exercises RDMA NIC ↔ Blackhole peer-to-peer DMA over a fabric end to end.

---

Bug Fixes

• Userspace mappings outliving device removal. When the driver lets go of a device — including a software-initiated removal via the sysfs remove file — any existing userspace mmap()s of it are now torn down, so an application holding a mapping faults on access instead of issuing stray MMIO to a device the driver no longer manages.
• Blackhole DMA misdirected to the wrong PCIe outbound window. The streaming DMA mask used for TENSTORRENT_IOCTL_PIN_PAGES was set to 64 bits, which let the IOMMU return an IOVA wider than 58 bits. On Blackhole a pinned buffer's address can be used directly as a NOC address, where the upper 6 bits of a 64-bit NOC-outbound address select the PCIe outbound TLB window; an address at or above 2^58 selected the wrong window and silently misdirected the DMA. The streaming mask is now capped at 58 bits. Wormhole's coherent 32-bit ALLOCATE_DMA_BUF behavior is unchanged. On an affected host still running 2.10.0-rc1 or older, loading the driver with dma_address_bits=58 is an equivalent workaround.
• TLB windows and outbound iATU regions are now reclaimed from reset-invalidated fds. A reset that bumps the reset generation invalidates every other open fd, and previously an invalidated fd only returned its TLB windows and outbound iATU regions to the pool at close(). A process that wedged and never exited held its resources across every reset, and enough such processes could exhaust the pool. Resets now reclaim these resources from the fds they invalidate, and the stale fd's eventual close() cannot free a resource that a post-reset fd has since claimed. The resetter's own fd keeps its resources, and resource locks are deliberately untouched — they are pure software state and may intentionally span a reset.

---

Compatibility Notes

Linux 7.1 Build Support

The driver now builds against Linux 7.1.

O_EXCL Now Has Meaning on Device Files

Applications that do not pass O_EXCL are unaffected unless a tool (such as tt-flash) is actively holding the device exclusively, in which case their open() blocks until the tool finishes, or returns -EAGAIN if they passed O_NONBLOCK.

Reset-Issuing fd No Longer Self-Invalidates

The reset flavors that bump the reset generation (CONFIG_WRITE, USER_RESET, ASIC_RESET, ASIC_DMC_RESET) previously invalidated the issuing fd along with every other open fd, forcing the caller to close and reopen before issuing POST_RESET. The issuing fd now survives the bump and can complete the sequence directly. All other pre-reset fds still become invalid and fail with -ENODEV.