On newer cores, we need to explicitly set the subsystems to reset via
the watchdog. Logic adapted from bcmdhd.

Signed-off-by: Hector Martin <marcan@marcan.st>

Currently the pcie code multiplexes all register accesses through a
single window. This isn't very efficient, and it creates race conditions
when we access registers from multiple paths (e.g. in the interrupt
handler). Since the chip has a dedicated window for the PCIe core
registers, we can use that instead, avoid all the gratuitous window
switching, and fix the IRQ race issues.

Signed-off-by: Hector Martin <marcan@marcan.st>

Newer firmwares notify the host of boot completion via an MSI, so let's
make sure that is initialized before booting the firmware.

Signed-off-by: Hector Martin <marcan@marcan.st>

With secure boot, the vector is not accessible and trying to write it
triggers PCIe errors. Skip it in that case.

Signed-off-by: Hector Martin <marcan@marcan.st>

Make sure the firmware can't get any early notifications by masking all
IRQs explicitly before loading the firmware.

Signed-off-by: Hector Martin <marcan@marcan.st>

This is essentially identical to SCAN_V2 with an extra field where we
had a padding byte, so don't bother duplicating the entire structure.
Just add the field and the logic to set the version properly.

Signed-off-by: Hector Martin <marcan@marcan.st>

This extended command supports bit set/clear operations, but we just use
it like the old full mask set command.

Signed-off-by: Hector Martin <marcan@marcan.st>

The structures are compatible and just add fields, so we can just treat
it as always v112. If we start using new fields, that will have to be
gated on the version.

Signed-off-by: Hector Martin <marcan@marcan.st>

New firmware wants extra fields, hopefully old firmware ignores them.

Signed-off-by: Hector Martin <marcan@marcan.st>

Signed-off-by: Hector Martin <marcan@marcan.st>

Fix access point mode by bringing firmware into appropriate state before setting up the device.

Signed-off-by: Patrick Blass <patrickblass.dev@gmail.com>

This patch fixes the firmware memory map structure to be complete.
Along the way, we fix a failure to align the heap memory start address,
which causes failures with the newest apple wifi firmware.

With this patch, we can load the latest (sonoma 14.0 as of right now) apple wifi firmware.

Signed-off-by: Daniel Berlin <dberlin@dberlin.org>

This patch adds support for 6G chanspecs, as part of adding 6G and
802.11ax support.

Signed-off-by: Daniel Berlin <dberlin@dberlin.org>

1. Correct VHT MCS settings to support as many tx/rx streams as chip
   does.

2. Correct VHT capabilities to support what all chips do.

3. Correct max AMPDU capabilities for VHT.

4. Support LDPC and STBC in VHT where available.

Signed-off-by: Daniel Berlin <dberlin@dberlin.org>

…tructure.

Signed-off-by: Daniel Berlin <dberlin@dberlin.org>

This patch adds support for using GCMP/etc during offload
where supported by the firmware.

Signed-off-by: Daniel Berlin <dberlin@dberlin.org>

We may just have had to clone a packet, and not actually
have run out of headroom.  Only issue warning about headroom
when we actually ran out of headroom.
This removes useless spam about needing 0 more bytes of headroom.

Signed-off-by: Daniel Berlin <dberlin@dberlin.org>

This patch adds support for handling the scan flags that come from the
802.11 stack.  This enables the stack to control whether we are doing
high/low power scans, as well as other options.

Signed-off-by: Daniel Berlin <dberlin@dberlin.org>

This patch adds support for 6G bands, along with HE capabilities,
as they are required to register 6G bands with wiphy.
This in turn, enables 802.11ax support for the other bands.

Scanning is not updated in this patch, so the bands are unused
except to be able to process what the firmware tells us.

Existing code is updated to handle all the bands rather than just 2g and
5g channels.

Signed-off-by: Daniel Berlin <dberlin@dberlin.org>

Currently, we ignore the default country in the reg notifier.
We also register a custom regulatory domain, which is set
as the default.
As a result, the chip is likely to be set to the correct country,
but the regulatory domain will not match it.

When the regulatory notifier is then called, we see the countries
are the same and do not change anything, even though the domain
is wrong.

This patch forces us to reset the bands on the first country change
even if the chip is already set to that country.

We also restore the original band info before reconstructing channel
info, as the new regdom power limits may be higher than what is
currently set.

Signed-off-by: Daniel Berlin <dberlin@dberlin.org>

Signed-off-by: Daniel Berlin <dberlin@dberlin.org>

Signed-off-by: Daniel Berlin <dberlin@dberlin.org>

This patch structurizes PNF scan handling, adding support for
netinfo v3 and PNO v3 structures.

This in turn, enables the chip to tell us about 6G scan results,
as the results contain chanspecs and not just channels.

Signed-off-by: Daniel Berlin <dberlin@dberlin.org>

Signed-off-by: Daniel Berlin <dberlin@dberlin.org>

To support new join parameter versions, we move to using a function
pointer structure that knows how to deal with the different versions
of structures

Drive-by fix: Always count the assoc_params length even if no bssid is
provided. It doesn't make sense to truncate it off, since we need to set
the bssid to the broadcast addr anyway in that case.

Signed-off-by: Hector Martin <marcan@marcan.st>
Signed-off-by: Daniel Berlin <dberlin@dberlin.org>

handle the interface versions we find.

Signed-off-by: Daniel Berlin <dberlin@dberlin.org>

This adds support for more authentication types during roaming offload,
enabling the firmware to handle roaming for ~all authentication types.

Signed-off-by: Daniel Berlin <dberlin@dberlin.org>

Signed-off-by: Daniel Berlin <dberlin@dberlin.org>

We try to retrieve a newer sta_info structure that has
both rx and tx ratespecs, but if we don't get the
structure we are expecting we fall back to tx rate info only.

Signed-off-by: Daniel Berlin <dberlin@dberlin.org>

Signed-off-by: Asahi Lina <lina@asahilina.net>

of_platform_device_destroy() can trigger several NULL pointer
dereference which have been elusive so far.
Comment this for now since the oopses causes the shutdown to hang.
Since dcp can not be reloaded this leaks the platform device on shutdown
and reboot.

Signed-off-by: Janne Grunau <j@jannau.net>

Avoids out of order messages and already unmapped buffers while tracing
with hv/trace_dcp.py.

Signed-off-by: Janne Grunau <j@jannau.net>

This callback is occasionally seen around (failed) modesets. There seems
to be no need to handle it so just trace it.

Signed-off-by: Janne Grunau <j@jannau.net>

Signed-off-by: Janne Grunau <j@jannau.net>

This reverts commit ab69434.

This mirrors what macOS does and should make reconnections more
reliable.

Signed-off-by: Janne Grunau <j@jannau.net>

Seems to make disconnect / reconnect more reliable and almost fixes
suspend/resume. The drm device tries to modeset too early on resume
which leaves the screen blank.
This should reduce power consumption after disconnecting the HDMI port.

Signed-off-by: Janne Grunau <j@jannau.net>

Makes connects more reliable.

Signed-off-by: Janne Grunau <j@jannau.net>

DCP notices the disconnects on its own and the parallel handling just
results in confusion (both on DRM and developer side).

Signed-off-by: Janne Grunau <j@jannau.net>

Signed-off-by: Janne Grunau <j@jannau.net>

Triggering modesets from drm_connector_helper_funcs.atomic_check is more
in line with DRM/KMS' design and allows returning errors from failed
modesets.
Ignore hotplug callbacks from DCP during modeset. DCP always does
disconnected -> connected on (at least the initial) modeset. Shield drm
helpers from this.
This improves reliability with externel (dptx based) displays.

Signed-off-by: Janne Grunau <j@jannau.net>

Signed-off-by: Janne Grunau <j@jannau.net>

Required for the DP2HDMI only atc3 port on 14/16 inch Macbook Pros.

Fixes: 9ca3958 ("phy: apple: atc: Support DisplayPort only operation")
Signed-off-by: Janne Grunau <j@jannau.net>

Signed-off-by: Janne Grunau <j@jannau.net>

Signed-off-by: Asahi Lina <lina@asahilina.net>

We want to use caller_location to uniquely identify callsites, to
automatically create lockdep classes without macros. The location
filename in local code uses the relative path passed to the compiler,
but if that code is generic and instantiated from another crate, the
path becomes absolute.

To make this work and keep the paths consistent, always pass an absolute
path to the compiler. Then the Location path is always identical
regardless of how the code is being compiled.

Signed-off-by: Asahi Lina <lina@asahilina.net>

This allows Rust code to use the Hasher interface with the kernel
siphash implementation.

Signed-off-by: Asahi Lina <lina@asahilina.net>

Lock classes aren't used without lockdep. The C side declares the key
as an empty struct in that case, but let's make it an explicit ZST in
Rust, implemented in a separate module. This allows us to more easily
guarantee that the lockdep code will be trivially optimized out without
CONFIG_LOCKDEP, including LockClassKey arguments that are passed around.

Depending on whether CONFIG_LOCKDEP is enabled or not, we then import
the real lockdep implementation or the dummy one.

Signed-off-by: Asahi Lina <lina@asahilina.net>

We want to be able to handle dynamic lock class creation and using
pointers to things that aren't a real lock_class_key as lock classes.
Doing this by casting around Rust references is difficult without
accidentally invoking UB.

Instead, switch LockClassKey to being a raw pointer wrapper around a
lock_class_key, which means there is no UB possible on the Rust side
just by creating and consuming these objects. The C code also should
never actually dereference lock classes, only use their address
(possibly with an offset).

We still provide a dummy ZST version of this wrapper, to be used when
lockdep is disabled.

Signed-off-by: Asahi Lina <lina@asahilina.net>

Using macros to create lock classes all over the place is unergonomic,
and makes it impossible to add new features that require lock classes to
code such as Arc<> without changing all callers.

Rust has the ability to track the caller's identity by file/line/column
number, and we can use that to dynamically generate lock classes
instead.

Signed-off-by: Asahi Lina <lina@asahilina.net>

Use the new automagic lock class code to remove the lock class and name
parameters from Lock::new() and Lock::pin_init(). The old functions
are renamed to new_with_class() and pin_init_with_class() respectively.

Signed-off-by: Asahi Lina <lina@asahilina.net>

Signed-off-by: Asahi Lina <lina@asahilina.net>

This allows Rust code to explicitly integrate types with lockdep.

Signed-off-by: Asahi Lina <lina@asahilina.net>

Signed-off-by: Asahi Lina <lina@asahilina.net>

This module is intended to contain functions related to kernel
timekeeping and time. Initially, this just wraps ktime_get() and
ktime_get_boottime() and returns them as core::time::Duration instances.

Signed-off-by: Asahi Lina <lina@asahilina.net>

The XArray is an abstract data type which behaves like a very large
array of pointers. Add a Rust abstraction for this data type.

The initial implementation uses explicit locking on get operations and
returns a guard which blocks mutation, ensuring that the referenced
object remains alive. To avoid excessive serialization, users are
expected to use an inner type that can be efficiently cloned (such as
Arc<T>), and eagerly clone and drop the guard to unblock other users
after a lookup.

Future variants may support using RCU instead to avoid mutex locking.

This abstraction also introduces a reservation mechanism, which can be
used by alloc-capable XArrays to reserve a free slot without immediately
filling it, and then do so at a later time. If the reservation is
dropped without being filled, the slot is freed again for other users,
which eliminates the need for explicit cleanup code.

Signed-off-by: Asahi Lina <lina@asahilina.net>
---

Hi everyone!

This abstraction is part of the set of dependencies for the drm/asahi
Apple M1/M2 GPU driver.

The branch at [1] contains the full series of patches rebased on
upstream leading to the complete driver, for reference on how it is
intended to be used.

Thank you everyone who helped review this on GitHub [2]! I hope I didn't
forget any CCs...

Note that I dropped the convenience `Deref` impl for `Guard`, since I
couldn't figure out how to do it safely. Suggestions welcome, or we can
leave it for a future improvement ^^

[1] https://github.com/AsahiLinux/linux/tree/gpu/rebase-20230224
[2] Rust-for-Linux#952

Changes in v3:
- Updated to the error v2/v3 series API.
- Renamed `err` to `ret` for consistency with the other instance.
- Link to v2: https://lore.kernel.org/r/20230224-rust-xarray-v2-1-4eeb0134944c@asahilina.net
Changes in v2:
- Added Pin requirement for all XArray operations, to close a
  soundness hole due to the lock in the XArray (locks are not safe to
  move while locked). Creation does not require pinning in place, since
  the lock cannot be acquired at that point.
- Added safety note to Drop impl about why we don't need to do the lock
  unlock dance to ensure soundness in case of a dropped lock guard.
- Downstream drm/asahi driver was also rebased on this version to prove
  it works (previously it was still on a pre-v1 version).
- This still depends on the Error series (v1). v2 of that will need a
  trivial rename of Error::from_kernel_errno -> Error::from_errno. If
  this version of XArray ends up looking good, I'll send a trivial v4 of
  XArray with the rename, after sending the v2 of the Error series.
- Link to v1: https://lore.kernel.org/r/20230224-rust-xarray-v1-1-80f0904ce5d3@asahilina.net

Some traits exposed by the kernel crate may not be intended to be
implemented by downstream modules. Add a Sealed trait to allow avoiding
this using the sealed trait pattern.

Signed-off-by: Asahi Lina <lina@asahilina.net>

Add a RawDevice trait which can be implemented by any type representing
a device class (such as a PlatformDevice), and a Device type which
represents an owned reference to a generic struct device.

Lina: Rewrote commit message, dropped the Amba bits, and squashed in
simple changes to the core RawDevice/Device code from latter commits
in rust-for-linux/rust. Also include the rust_helper_dev_get_drvdata
helper which will be needed by consumers later on.

Co-developed-by: Miguel Ojeda <ojeda@kernel.org>
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
Signed-off-by: Wedson Almeida Filho <wedsonaf@google.com>
Signed-off-by: Asahi Lina <lina@asahilina.net>

The io_pgtable subsystem implements page table management for various
IOMMU page table formats. This abstraction allows Rust drivers for
devices with an embedded MMU to use this shared code.

Signed-off-by: Asahi Lina <lina@asahilina.net>

RTKit is Apple's proprietary real-time operating system framework, used
across many subdevices on Apple Silicon platforms including NVMe, system
management, GPU, etc. Add Rust abstractions for this subsystem, so that
it can be used by upcoming Rust drivers.

Note: Although ARM64 support is not yet merged, this can be built on amd64
with CONFIG_COMPILE_TEST=y.

Signed-off-by: Asahi Lina <lina@asahilina.net>

Add Rust counterparts to these C macros. `container_of` is useful for C
struct subtyping, to recover the original pointer to the container
structure. `offset_of` is useful for struct-relative addressing.

Lina: Rewrote commit message and squashed in later changes from
rust-for-linux/linux, attributed below.

Co-authored-by: Miguel Ojeda <ojeda@kernel.org>
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
Co-authored:by: Léo Lanteri Thauvin <leseulartichaut@gmail.com>
Signed-off-by: Léo Lanteri Thauvin <leseulartichaut@gmail.com>
Co-authored-by: Wedson Almeida Filho <wedsonaf@google.com>
Signed-off-by: Wedson Almeida Filho <wedsonaf@google.com>
Signed-off-by: Asahi Lina <lina@asahilina.net>

Signed-off-by: Asahi Lina <lina@asahilina.net>

Commit reference: 3dfc5eb

Commit reference: 3dfc5eb

Commit reference: 3dfc5eb

Computed based on the PAGE_SHIFT macro from C.

// Co-authored-by: Adam Bratschi-Kaye <ark.email@gmail.com>
// Co-authored-by: Miguel Ojeda <ojeda@kernel.org>
// Signed-off-by: Asahi Lina <lina@asahilina.net>

Needed by the rust-for-linux/rust module_param module.

Signed-off-by: Asahi Lina <lina@asahilina.net>

Commit reference: 3dfc5eb

This is the same behavior as kstrtol/kstrtoul, and allows simple
`echo 0 > /sys/module/foo/parameters/bar` commands to work.

Signed-off-by: Asahi Lina <lina@asahilina.net>

Co-developed-by: Wedson Almeida Filho <wedsonaf@google.com>
Signed-off-by: Wedson Almeida Filho <wedsonaf@google.com>
Signed-off-by: Asahi Lina <lina@asahilina.net>

Commit reference: 3dfc5eb

Commit reference: 3dfc5eb

Signed-off-by: Asahi Lina <lina@asahilina.net>

Commit reference: 3dfc5eb

Commit reference: 3dfc5eb

Commit reference: 3dfc5eb

Commit reference: 3dfc5eb

This abstraction enables Rust drivers to walk Device Tree nodes and
query their properties.

Signed-off-by: Asahi Lina <lina@asahilina.net>

…ation

In order for modpost to work and correctly generate module aliases from
device ID tables, it needs those tables to exist as global symbols with
a specific name. Additionally, modpost checks the size of the symbol, so
it cannot contain trailing data.

To support this, split IdArrayIds out of IdArray. The former contains
just the IDs. Then split out the device table definition macro from the
macro that defines the device table for a given bus driver, and add
another macro to declare a device table as a module device table.
Drivers can now define their ID table once, and then specify that it
should be used for both the driver and the module:

// Generic OF Device ID table.
kernel::define_of_id_table! {ASAHI_ID_TABLE, &'static hw::HwConfig, [
    (of::DeviceId::Compatible(b"apple,agx-t8103"), Some(&hw::t8103::HWCONFIG)),
    (of::DeviceId::Compatible(b"apple,agx-t8112"), Some(&hw::t8112::HWCONFIG)),
    // ...
]}

/// Platform Driver implementation for `AsahiDriver`.
impl platform::Driver for AsahiDriver {
    /// Data associated with each hardware ID.
    type IdInfo = &'static hw::HwConfig;

    // Assign the above OF ID table to this driver.
    kernel::driver_of_id_table!(ASAHI_ID_TABLE);

    // ...
}

// Export the OF ID table as a module ID table, to make modpost/autoloading work.
kernel::module_of_id_table!(MOD_TABLE, ASAHI_ID_TABLE);

Signed-off-by: Asahi Lina <lina@asahilina.net>

This patch adds a logic similar to `devm_platform_ioremap_resource`
function adding:
  - `IoResource` enumerated type that groups the `IORESOURCE_*` macros.
  - `get_resource()` method that is a binding of `platform_get_resource`
  - `ioremap_resource` that is newly written method similar to
    `devm_platform_ioremap_resource`.

Lina: Removed `bit` dependency and rebased

Co-developed-by: Asahi Lina <lina@asahilina.net>
Signed-off-by: Maciej Falkowski <m.falkowski@samsung.com>

Allows drivers to configure the DMA masks for a device. Implemented
here, not in device, because it requires a mutable platform device
reference this way (device::Device is a safely clonable reference).

Signed-off-by: Asahi Lina <lina@asahilina.net>

Apple SoCs require non-posted mappings for MMIO, and this is
automatically handled by devm_ioremap_resource() and friends via a
resource flag. Implement the same logic in kernel::io_mem, so it can
work the same way.

Signed-off-by: Asahi Lina <lina@asahilina.net>

Commit reference: 3dfc5eb

Commit reference: 3dfc5eb

TODO: This isn't abstracted properly yet

Signed-off-by: Asahi Lina <lina@asahilina.net>

Signed-off-by: Asahi Lina <lina@asahilina.net>

Commit reference: 3dfc5eb

Apple Silicon SoCs (M1, M2, etc.) have a GPU with an ARM64 firmware
coprocessor. The firmware and the GPU share page tables in the standard
ARM64 format (the firmware literally sets the base as its TTBR0/1
registers). TTBR0 covers the low half of the address space and is
intended to be per-GPU-VM (GPU user mappings and kernel-managed
buffers), while TTBR1 covers the upper half and is global (firmware
code, data, management structures shared with the AP, and a few
GPU-accessible data structures).

In typical Apple fashion, the permissions are interpreted differently
from traditional ARM PTEs. By default, firmware mappings use Apple SPRR
permission remapping. The firmware only uses that for its own
code/data/MMIO mappings, and those pages are not accessible by the GPU
hardware. We never need to touch/manage these mappings, so this patch
does not support them.

When a specific bit is set in the PTEs, permissions switch to a
different scheme which supports various combinations of firmware/GPU
access. This is the mode intended to be used by AP GPU drivers, and what
we implement here.

The prot bits are interpreted as follows:

- IOMMU_READ and IOMMU_WRITE have the usual meaning.

- IOMMU_PRIV creates firmware-only mappings (no GPU access)
- IOMMU_NOEXEC creates GPU-only structures (no FW access)
- Otherwise structures are accessible by both GPU and FW

- IOMMU_MMIO creates Device mappings for firmware
- IOMMU_CACHE creates Normal-NC mappings for firmware (cache-coherent
  from the point of view of the AP, but slower)
- Otherwise creates Normal mappings for firmware (this requires manual
  cache management on the firmware side, as it is not coherent with the
  SoC fabric)

GPU-only mappings (textures/etc) are expected to use IOMMU_CACHE and are
seemingly coherent with the CPU (or otherwise the firmware/GPU already
issue the required cache management operations when correctly
configured).

There is a GPU-RO/FW-RW mode, but it is not currently implemented (it
doesn't seem to be very useful for the driver). There seems to be no
real noexec control (i.e. for shaders) on the GPU side. All of these
mappings are implicitly noexec for the firmware.

Drivers are expected to fully manage per-user (TTBR0) page tables, but
ownership of shared kernel (TTBR1) page tables is shared between the
firmware and the AP OS. We handle this by simply using a smaller IAS to
drop down one level of page tables, so the driver can install a PTE in
the top-level (firmware-initialized) page table directly and just add an
offset to the VAs passed into the io_pgtable code. This avoids having to
have any special handling for this here. The firmware-relevant data
structures are small, so we do not expect to ever require more VA space
than one top-level PTE covers (IAS=36 for the next level, 64 GiB).

Only 16K page mode is supported. The coprocessor MMU supports huge pages
as usual for ARM64, but the GPU MMU does not, so we do not enable them.

Signed-off-by: Asahi Lina <lina@asahilina.net>

This is ugly, we need a better way of expressing this.

Signed-off-by: Asahi Lina <lina@asahilina.net>

Signed-off-by: Asahi Lina <lina@asahilina.net>

drm_sched_fini() currently leaves any pending jobs dangling, which
causes segfaults and other badness when job completion fences are
signaled after the scheduler is torn down.

Explicitly detach all jobs from their completion callbacks and free
them. This makes it possible to write a sensible safe abstraction for
drm_sched, without having to externally duplicate the tracking of
in-flight jobs.

This shouldn't regress any existing drivers, since calling
drm_sched_fini() with any pending jobs is broken and this change should
be a no-op if there are no pending jobs.

Signed-off-by: Asahi Lina <lina@asahilina.net>

A signaled scheduler fence can outlive its scheduler, since fences are
independencly reference counted. Therefore, we can't reference the
scheduler in the get_timeline_name() implementation.

Fixes oopses on `cat /sys/kernel/debug/dma_buf/bufinfo` when shared
dma-bufs reference fences from GPU schedulers that no longer exist.

Signed-off-by: Asahi Lina <lina@asahilina.net>

DMA fences are the internal synchronization primitive used for DMA
operations like GPU rendering, video en/decoding, etc. Add an
abstraction to allow Rust drivers to interact with this subsystem.

Note: This uses a raw spinlock living next to the fence, since we do
not interact with it other than for initialization.
TODO: Expose this to the user at some point with a safe abstraction.

Signed-off-by: Asahi Lina <lina@asahilina.net>

This is just for basic usage in the DRM shmem abstractions for implied
locking, not intended as a full DMA Reservation abstraction yet.

Signed-off-by: Asahi Lina <lina@asahilina.net>

DRM drivers need to be able to declare which driver-specific ioctls they
support. This abstraction adds the required types and a helper macro to
generate the ioctl definition inside the DRM driver.

Note that this macro is not usable until further bits of the
abstraction are in place (but it will not fail to compile on its own, if
not called).

Signed-off-by: Asahi Lina <lina@asahilina.net>

Add the initial abstractions for DRM drivers and devices. These go
together in one commit since they are fairly tightly coupled types.

A few things have been stubbed out, to be implemented as further bits of
the DRM subsystem are introduced.

Signed-off-by: Asahi Lina <lina@asahilina.net>

A DRM File is the DRM counterpart to a kernel file structure,
representing an open DRM file descriptor. Add a Rust abstraction to
allow drivers to implement their own File types that implement the
DriverFile trait.

Signed-off-by: Asahi Lina <lina@asahilina.net>