Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

riscv: ASID-related and UP-related TLB flush enhancements #790

Open
wants to merge 13 commits into
base: for-next_base
Choose a base branch
from
Open

riscv: ASID-related and UP-related TLB flush enhancements #790

wants to merge 13 commits into from

Conversation

bjoto
Copy link

@bjoto bjoto commented Mar 27, 2024

Pull request for series with
subject: riscv: ASID-related and UP-related TLB flush enhancements
version: 6
url: https://patchwork.kernel.org/project/linux-riscv/list/?series=838693

@bjoto
Copy link
Author

bjoto commented Mar 27, 2024

Upstream branch: 4cece76
series: https://patchwork.kernel.org/project/linux-riscv/list/?series=838693
version: 6

@bjoto
Copy link
Author

bjoto commented Mar 27, 2024

Upstream branch: 4cece76
series: https://patchwork.kernel.org/project/linux-riscv/list/?series=838693
version: 6

@bjoto bjoto closed this Apr 2, 2024
@bjoto bjoto reopened this Apr 2, 2024
@bjoto
Copy link
Author

bjoto commented Apr 2, 2024

Upstream branch: 4cece76
series: https://patchwork.kernel.org/project/linux-riscv/list/?series=838693
version: 6

@bjoto
Copy link
Author

bjoto commented Apr 10, 2024

Upstream branch: 542124f
series: https://patchwork.kernel.org/project/linux-riscv/list/?series=838693
version: 6

@bjoto
Copy link
Author

bjoto commented Apr 10, 2024

Upstream branch: ba5ea59
series: https://patchwork.kernel.org/project/linux-riscv/list/?series=838693
version: 6

@SiFiveHolland
riscv: Flush the instruction cache during SMP bringup
3b4851f 
Instruction cache flush IPIs are sent only to CPUs in cpu_online_mask,
so they will not target a CPU until it calls set_cpu_online() earlier in
smp_callin(). As a result, if instruction memory is modified between the
CPU coming out of reset and that point, then its instruction cache may
contain stale data. Therefore, the instruction cache must be flushed
after the set_cpu_online() synchronization point.

Fixes: 08f051e ("RISC-V: Flush I$ when making a dirty page executable")
Reviewed-by: Alexandre Ghiti <alexghiti@rivosinc.com>
Signed-off-by: Samuel Holland <samuel.holland@sifive.com>
Signed-off-by: Björn Töpel <bjorn@rivosinc.com>
@SiFiveHolland
riscv: Factor out page table TLB synchronization
f59a7ca 
The logic is the same for all page table levels. See commit 69be3fb
("riscv: enable MMU_GATHER_RCU_TABLE_FREE for SMP && MMU").

Signed-off-by: Samuel Holland <samuel.holland@sifive.com>
Reviewed-by: Alexandre Ghiti <alexghiti@rivosinc.com>
Signed-off-by: Björn Töpel <bjorn@rivosinc.com>
@SiFiveHolland
riscv: Use IPIs for remote cache/TLB flushes by default
71b6ff5 
An IPI backend is always required in an SMP configuration, but an SBI
implementation is not. For example, SBI will be unavailable when the
kernel runs in M mode. For this reason, consider IPI delivery of cache
and TLB flushes to be the base case, and any other implementation (such
as the SBI remote fence extension) to be an optimization.

Generally, if IPIs can be delivered without firmware assistance, they
are assumed to be faster than SBI calls due to the SBI context switch
overhead. However, when SBI is used as the IPI backend, then the context
switch cost must be paid anyway, and performing the cache/TLB flush
directly in the SBI implementation is more efficient than injecting an
interrupt to S-mode. This is the only existing scenario where
riscv_ipi_set_virq_range() is called with use_for_rfence set to false.

sbi_ipi_init() already checks riscv_ipi_have_virq_range(), so it only
calls riscv_ipi_set_virq_range() when no other IPI device is available.
This allows moving the static key and dropping the use_for_rfence
parameter. This decouples the static key from the irqchip driver probe
order.

Furthermore, the static branch only makes sense when CONFIG_RISCV_SBI is
enabled. Optherwise, IPIs must be used. Add a fallback definition of
riscv_use_sbi_for_rfence() which handles this case and removes the need
to check CONFIG_RISCV_SBI elsewhere, such as in cacheflush.c.

Reviewed-by: Anup Patel <anup@brainfault.org>
Signed-off-by: Samuel Holland <samuel.holland@sifive.com>
Reviewed-by: Alexandre Ghiti <alexghiti@rivosinc.com>
Signed-off-by: Björn Töpel <bjorn@rivosinc.com>
@SiFiveHolland
riscv: mm: Broadcast kernel TLB flushes only when needed
f0bdffd 
__flush_tlb_range() avoids broadcasting TLB flushes when an mm context
is only active on the local CPU. Apply this same optimization to TLB
flushes of kernel memory when only one CPU is online. This check can be
constant-folded when SMP is disabled.

Reviewed-by: Alexandre Ghiti <alexghiti@rivosinc.com>
Signed-off-by: Samuel Holland <samuel.holland@sifive.com>
Signed-off-by: Björn Töpel <bjorn@rivosinc.com>
@SiFiveHolland
riscv: Only send remote fences when some other CPU is online
ffe47b6 
If no other CPU is online, a local cache or TLB flush is sufficient.
These checks can be constant-folded when SMP is disabled.

Signed-off-by: Samuel Holland <samuel.holland@sifive.com>
Reviewed-by: Alexandre Ghiti <alexghiti@rivosinc.com>
Signed-off-by: Björn Töpel <bjorn@rivosinc.com>
@SiFiveHolland
riscv: mm: Combine the SMP and UP TLB flush code
f83a025 
In SMP configurations, all TLB flushing narrower than flush_tlb_all()
goes through __flush_tlb_range(). Do the same in UP configurations.

This allows UP configurations to take advantage of recent improvements
to the code in tlbflush.c, such as support for huge pages and flushing
multiple-page ranges.

Reviewed-by: Alexandre Ghiti <alexghiti@rivosinc.com>
Signed-off-by: Samuel Holland <samuel.holland@sifive.com>
Reviewed-by: Yunhui Cui <cuiyunhui@bytedance.com>
Signed-off-by: Björn Töpel <bjorn@rivosinc.com>
@SiFiveHolland
riscv: Apply SiFive CIP-1200 workaround to single-ASID sfence.vma
591ae53 
commit 3f1e782 ("riscv: add ASID-based tlbflushing methods") added
calls to the sfence.vma instruction with rs2 != x0. These single-ASID
instruction variants are also affected by SiFive errata CIP-1200.

Until now, the errata workaround was not needed for the single-ASID
sfence.vma variants, because they were only used when the ASID allocator
was enabled, and the affected SiFive platforms do not support multiple
ASIDs. However, we are going to start using those sfence.vma variants
regardless of ASID support, so now we need alternatives covering them.

Signed-off-by: Samuel Holland <samuel.holland@sifive.com>
Signed-off-by: Björn Töpel <bjorn@rivosinc.com>
@SiFiveHolland
riscv: Avoid TLB flush loops when affected by SiFive CIP-1200
3117cdc 
Implementations affected by SiFive errata CIP-1200 have a bug which
forces the kernel to always use the global variant of the sfence.vma
instruction. When affected by this errata, do not attempt to flush a
range of addresses; each iteration of the loop would actually flush the
whole TLB instead. Instead, minimize the overall number of sfence.vma
instructions.

Signed-off-by: Samuel Holland <samuel.holland@sifive.com>
Reviewed-by: Yunhui Cui <cuiyunhui@bytedance.com>
Signed-off-by: Björn Töpel <bjorn@rivosinc.com>
@SiFiveHolland
riscv: mm: Introduce cntx2asid/cntx2version helper macros
c13d35a 
When using the ASID allocator, the MM context ID contains two values:
the ASID in the lower bits, and the allocator version number in the
remaining bits. Use macros to make this separation more obvious.

Reviewed-by: Alexandre Ghiti <alexghiti@rivosinc.com>
Signed-off-by: Samuel Holland <samuel.holland@sifive.com>
Signed-off-by: Björn Töpel <bjorn@rivosinc.com>
@SiFiveHolland
riscv: mm: Use a fixed layout for the MM context ID
70a4225 
Currently, the size of the ASID field in the MM context ID dynamically
depends on the number of hardware-supported ASID bits. This requires
reading a global variable to extract either field from the context ID.
Instead, allocate the maximum possible number of bits to the ASID field,
so the layout of the context ID is known at compile-time.

Reviewed-by: Alexandre Ghiti <alexghiti@rivosinc.com>
Signed-off-by: Samuel Holland <samuel.holland@sifive.com>
Signed-off-by: Björn Töpel <bjorn@rivosinc.com>
@SiFiveHolland
riscv: mm: Make asid_bits a local variable
d984d2c 
This variable is only used inside asids_init().

Reviewed-by: Alexandre Ghiti <alexghiti@rivosinc.com>
Signed-off-by: Samuel Holland <samuel.holland@sifive.com>
Signed-off-by: Björn Töpel <bjorn@rivosinc.com>
@bjoto
Copy link
Author

bjoto commented Apr 28, 2024

Upstream branch: 0a16a17
series: https://patchwork.kernel.org/project/linux-riscv/list/?series=838693
version: 6

@SiFiveHolland
riscv: mm: Preserve global TLB entries when switching contexts
f236986 
If the CPU does not support multiple ASIDs, all MM contexts use ASID 0.
In this case, it is still beneficial to flush the TLB by ASID, as the
single-ASID variant of the sfence.vma instruction preserves TLB entries
for global (kernel) pages.

This optimization is recommended by the RISC-V privileged specification:

  If the implementation does not provide ASIDs, or software chooses
  to always use ASID 0, then after every satp write, software should
  execute SFENCE.VMA with rs1=x0. In the common case that no global
  translations have been modified, rs2 should be set to a register
  other than x0 but which contains the value zero, so that global
  translations are not flushed.

It is not possible to apply this optimization when using the ASID
allocator, because that code must flush the TLB for all ASIDs at once
when incrementing the version number.

Reviewed-by: Alexandre Ghiti <alexghiti@rivosinc.com>
Signed-off-by: Samuel Holland <samuel.holland@sifive.com>
Signed-off-by: Björn Töpel <bjorn@rivosinc.com>
@SiFiveHolland
riscv: mm: Always use an ASID to flush mm contexts
f430839 
Even if multiple ASIDs are not supported, using the single-ASID variant
of the sfence.vma instruction preserves TLB entries for global (kernel)
pages. So it is always more efficient to use the single-ASID code path.

Reviewed-by: Alexandre Ghiti <alexghiti@rivosinc.com>
Signed-off-by: Samuel Holland <samuel.holland@sifive.com>
Signed-off-by: Björn Töpel <bjorn@rivosinc.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
for-next new V6
Projects
None yet
Milestone
No milestone
2 participants
@bjoto @SiFiveHolland