Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MIPS: eBPF: refactor code, add MIPS32 JIT #1875

Closed
wants to merge 17 commits into from
Closed

MIPS: eBPF: refactor code, add MIPS32 JIT #1875

wants to merge 17 commits into from

Conversation

kernel-patches-bot
Copy link

Pull request for series with
subject: MIPS: eBPF: refactor code, add MIPS32 JIT
version: 2
url: https://patchwork.kernel.org/project/netdevbpf/list/?series=557579

Nobody and others added 17 commits October 5, 2021 01:44
adding ci files
e00acb7
MIPS: eBPF: support BPF_TAIL_CALL in JIT static analysis
c248f4e 
Add support in reg_val_propagate_range() for BPF_TAIL_CALL, fixing many
kernel log WARNINGs ("Unhandled BPF_JMP case") seen during JIT testing.
Treat BPF_TAIL_CALL like a NOP, falling through as if the tail call failed.

Fixes: b6bd53f ("MIPS: Add missing file for eBPF JIT.")
Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
MIPS: eBPF: mask 32-bit index for tail calls
cfc3dec 
The program array index for tail-calls should be 32-bit, so zero-extend to
sanitize the value. This fixes failures seen for test_verifier test:

  852/p runtime/jit: pass > 32bit index to tail_call FAIL retval 2 != 42

Fixes: b6bd53f ("MIPS: Add missing file for eBPF JIT.")
Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
MIPS: eBPF: fix BPF_ALU|ARSH handling in JIT static analysis
b72d845 
Update reg_val_propagate_range() to add the missing case for BPF_ALU|ARSH,
otherwise zero-extension breaks for this opcode.

Resolves failures for test_verifier tests:
  963/p arsh32 reg zero extend check FAIL retval -1 != 0
  964/u arsh32 imm zero extend check FAIL retval -1 != 0
  964/p arsh32 imm zero extend check FAIL retval -1 != 0

Fixes: b6bd53f ("MIPS: Add missing file for eBPF JIT.")
Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
MIPS: eBPF: support BPF_JMP32 in JIT static analysis
346c0b7 
While the MIPS64 JIT rejects programs with JMP32 insns, it still performs
initial static analysis. Add support in reg_val_propagate_range() for
BPF_JMP32, fixing kernel log WARNINGs ("Unhandled BPF_JMP case") seen
during JIT testing. Handle code BPF_JMP32 the same as BPF_JMP.

Fixes: 092ed09 ("bpf: verifier support JMP32")
Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
MIPS: eBPF: fix system hang with verifier dead-code patching
7ca7bd9 
Commit 2a5418a changed verifier dead code handling from patching with
NOPs to using a loop trap made with BPF_JMP_IMM(BPF_JA, 0, 0, -1). This
confuses the JIT static analysis, which follows the loop assuming the
verifier passed safe code, and results in a system hang and RCU stall.
Update reg_val_propagate_range() to fall through these trap insns.

Trigger the bug using test_verifier "check known subreg with unknown reg".

Fixes: 2a5418a ("bpf: improve dead code sanitizing")
Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
MIPS: eBPF: fix JIT static analysis hang with bounded loops
f935968 
Support for bounded loops allowed the verifier to output backward jumps
such as BPF_JMP_A(-4). These trap the JIT's static analysis in a loop,
resulting in a system hang and eventual RCU stall.

Fix by updating reg_val_propagate_range() to skip backward jumps when in
fallthrough mode and if the jump target has been visited already.

Trigger the bug using the test_verifier test "bounded loop that jumps out
rather than in".

Fixes: 2589726 ("bpf: introduce bounded loops")
Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
MIPS: eBPF: fix MOD64 insn on R6 ISA
13739d3 
The BPF_ALU64 | BPF_MOD implementation is broken on MIPS64R6 due to use of
a 32-bit "modu" insn, as shown by the test_verifier failures:

  455/p MOD64 overflow, check 1 FAIL retval 0 != 1 (run 1/1)
  456/p MOD64 overflow, check 2 FAIL retval 0 != 1 (run 1/1)

Resolve by using the 64-bit "dmodu" instead.

Fixes: 6c2c8a1 ("MIPS: eBPF: Provide eBPF support for MIPS64R6")
Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
MIPS: eBPF: support long jump for BPF_JMP|EXIT
d3c5632 
Existing JIT code supports only short (18-bit) branches for BPF EXIT, and
results in some tests from module test_bpf not being jited. Update code
to fall back to long (28-bit) jumps if short branches are insufficient.

Before:
  test_bpf: #296 BPF_MAXINSNS: exec all MSH jited:0 1556004 PASS
  test_bpf: #297 BPF_MAXINSNS: ld_abs+get_processor_id jited:0 824957 PASS
  test_bpf: Summary: 378 PASSED, 0 FAILED, [364/366 JIT'ed]

After:
  test_bpf: #296 BPF_MAXINSNS: exec all MSH jited:1 221998 PASS
  test_bpf: #297 BPF_MAXINSNS: ld_abs+get_processor_id jited:1 490507 PASS
  test_bpf: Summary: 378 PASSED, 0 FAILED, [366/366 JIT'ed]

Fixes: b6bd53f ("MIPS: Add missing file for eBPF JIT.")
Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
MIPS: eBPF: drop src_reg restriction in BPF_LD|BPF_DW|BPF_IMM
3d1f634 
Stop enforcing (insn->src_reg == 0) and allow special verifier insns such
as "BPF_LD_MAP_FD(BPF_REG_2, 0)", used to refer to a process-local map fd
and introduced in 0246e64 ("bpf: handle pseudo BPF_LD_IMM64 insn").

This is consistent with other JITs such as riscv32 and also used by
test_verifier (e.g. "runtime/jit: tail_call within bounds, prog once").

Fixes: b6bd53f ("MIPS: Add missing file for eBPF JIT.")
Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
MIPS: eBPF: add core support for 32/64-bit systems
d6c86ab 
Update register definitions and flags for both 32/64-bit operation. Add a
common register lookup table, modifying ebpf_to_mips_reg() to use this,
and update enum and literals for ebpf_to_mips_reg() to be more consistent
and less confusing. Add is64bit() and isbigend() common helper functions.

On MIPS32, BPF registers are held in register pairs defined by the base
register. Word-size and endian-aware helper macros select 32-bit registers
from a pair and generate 32-bit word memory storage offsets. The BPF TCC
is stored to the stack due to register pressure.

Update bpf_int_jit_compile() to properly enable BPF2BPF calls, by adding
support for the extra pass needed to fix up function target addresses.
Also provide bpf line info by calling bpf_prog_fill_jited_linfo().
Modify build_int_prologue() and build_int_epilogue() to handle MIPS32
registers and any adjustments needed during program entry/exit/transfer
when transitioning between the native N64/O32 ABI and the BPF 64-bit ABI.
Also ensure ABI-consistent stack alignment and use the verifier-provided
stack depth during setup, saving considerable stack space.

Update emit_const_to_reg() to work across MIPS64 and MIPS32 systems and
optimize gen_imm_to_reg() to only set the lower halfword if needed.

Rework emit_bpf_tail_call() to also support MIPS32 usage and add common
helpers, emit_bpf_call() and emit_push_args(), handling TCC and ABI
variations on MIPS32/MIPS64. Add tail_call_present() and update tailcall
handling to support mixing BPF2BPF subprograms and tailcalls.

Add sign and zero-extension helpers usable with verifier zext insertion,
gen_zext_insn() and gen_sext_insn().

Add common functions emit_caller_save() and emit_caller_restore(), which
push and pop all caller-saved BPF registers to the stack, for use with
JIT-internal kernel calls such as those needed for BPF insns unsupported
by native system ISA opcodes. Since these calls would be hidden from any
BPF C compiler, which would normally spill needed registers during a call,
the JIT must handle save/restore itself.

Adopt a dedicated BPF FP (in MIPS_R_S8), and relax FP usage within insns.
This reduces ad-hoc code doing $sp manipulation with temp registers, and
allows wider usage of BPF FP for comparison and arithmetic. For example,
the following tests from test_verifier are now jited but not previously:

  939/p store PTR_TO_STACK in R10 to array map using BPF_B
  981/p unpriv: cmp pointer with pointer
  984/p unpriv: indirectly pass pointer on stack to helper function
  985/p unpriv: mangle pointer on stack 1
  986/p unpriv: mangle pointer on stack 2
  1001/p unpriv: partial copy of pointer
  1097/p xadd/w check whether src/dst got mangled, 1
  1098/p xadd/w check whether src/dst got mangled, 2

Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
bpf: allow tailcalls in subprograms for MIPS64/MIPS32
ddc4e26 
The BPF core/verifier is hard-coded to permit mixing bpf2bpf and tail
calls for only x86-64. Change the logic to instead rely on a new weak
function 'bool bpf_jit_supports_subprog_tailcalls(void)', which a capable
JIT backend can override.

Update the x86-64 eBPF JIT to reflect this, and also enable the feature
for the MIPS64/MIPS32 JIT.

Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
MIPS: eBPF: refactor common 32/64-bit functions and headers
d63c2e0 
Move core functions and headers to ebpf_jit_core.c and ebpf_jit.h, and
relocate the MIPS64 specific build_one_insn() to ebpf_jit_comp64.c.

Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
MIPS: eBPF64: support BPF_JMP32 conditionals
03b48af 
Simplify BPF_JGT/JLE/JSGT/JSLE and related code by dropping unneeded usage
of MIPS_R_T8/T9 registers and jump-around branches, and extra comparison
arithmetic. Also reorganize var declarations and add 'bpf_class' helper
constant.

Implement BPF_JMP32 branches using sign or zero-extended temporaries as
needed for comparisons. This enables JITing of many more BPF programs, and
provides greater test coverage by e.g. 'test_verifier'.

Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
MIPS: eBPF64: implement all BPF_ATOMIC ops
ba09a96 
Reorganize code for BPF_ATOMIC and BPF_MEM, and add the atomic ops AND,
OR, XOR, XCHG and CMPXCHG, with support for BPF_FETCH.

Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
MIPS: uasm: Enable muhu opcode for MIPS R6
48db765 
Enable the 'muhu' instruction, complementing the existing 'mulu', needed
to implement a MIPS32 BPF JIT.

Also fix a typo in the existing definition of 'dmulu'.

Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
MIPS: eBPF: add MIPS32 JIT
3d49046 
Add a new variant of build_one_insn() supporting MIPS32, leveraging the
previously added common functions, and disable static analysis as unneeded
on MIPS32. Also define bpf_jit_needs_zext() to request verifier zext
insertion. Handle these zext insns, and add conditional zext for all ALU32
and LDX word-size operations for cases where the verifier is unable to do
so (e.g. test_bpf bypasses verifier).

Aside from mapping 64-bit BPF registers to pairs of 32-bit MIPS registers,
notable changes from the MIPS64 version of build_one_insn() include:

  BPF_ALU{,64} | {DIV,MOD} | BPF_K: drop divide-by-zero guard as the insns
  underlying do not raise exceptions.

  BPF_JMP | JSET | BPF_K: drop bbit insns only usable on MIPS64 Octeon.

In addition to BPF_ADD, implement atomic insns AND, OR, XOR, XCHG and
CMPXCHG, together with support for BPF_FETCH, in mode BPF_W.

The MIPS32 ISA does not include 64-bit div/mod or atomic opcodes. Add the
emit_bpf_divmod64() and emit_bpf_atomic64() functions, which use built-in
kernel functions to implement the following BPF insns:

  BPF_STX   | BPF_DW  | BPF_ATOMIC
    (AND, OR, XOR, XCHG, CMPXCHG, FETCH)
  BPF_ALU64 | BPF_DIV | BPF_X
  BPF_ALU64 | BPF_DIV | BPF_K
  BPF_ALU64 | BPF_MOD | BPF_X
  BPF_ALU64 | BPF_MOD | BPF_K

Test and development primarily used LTS kernel 5.10.x and then 5.13.x,
running under QEMU. Test suites included the 'test_bpf' module and the
'test_verifier' program from kselftests. Testing with 'test_progs' from
kselftests was not possible in general since cross-compilation depends on
libbpf/bpftool, which does not support cross-endian builds (see also [1]).

The matrix of test configurations executed for this series covers:
MIPSWORD={64-bit,32-bit} x MIPSISA={R2,R6} x JIT={off,on,hardened}

On MIPS32BE and MIPS32LE there was general parity between the results of
interpreter vs. JIT-backed tests with respect to the numbers of PASSED,
SKIPPED, and FAILED tests.

A sample of results on QEMU/MIPS32LE from kernel 5.13.x:

  root@OpenWrt:~# sysctl net.core.bpf_jit_enable=1
  root@OpenWrt:~# modprobe test_bpf
  ...
  test_bpf: Summary: 378 PASSED, 0 FAILED, [366/366 JIT'ed]
  root@OpenWrt:~# ./test_verifier 0 884
  ...
  Summary: 1231 PASSED, 0 SKIPPED, 20 FAILED
  root@OpenWrt:~# ./test_verifier 886 1184
  ...
  Summary: 459 PASSED, 1 SKIPPED, 2 FAILED
  root@OpenWrt:~# ./test_progs -n 105,106
  ...
  105 subprogs:OK
  106/1 tailcall_1:OK
  106/2 tailcall_2:OK
  106/3 tailcall_3:OK
  106/4 tailcall_4:OK
  106/5 tailcall_5:OK
  106/6 tailcall_bpf2bpf_1:OK
  106/7 tailcall_bpf2bpf_2:OK
  106/8 tailcall_bpf2bpf_3:OK
  106/9 tailcall_bpf2bpf_4:OK
  106 tailcalls:OK
  Summary: 2/9 PASSED, 0 SKIPPED, 0 FAILED

Link: [1] https://lore.kernel.org/bpf/CAEf4BzZCnP3oB81w4BDL4TCmvO3vPw8MucOTbVnjbW8UuCtejw@mail.gmail.com/

Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
@kernel-patches-bot
Copy link
Author

Master branch: 0693b27
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=557579
version: 2

@kernel-patches-bot
Copy link
Author

At least one diff in series https://patchwork.kernel.org/project/netdevbpf/list/?series=557579 expired. Closing PR.

@kernel-patches-bot kernel-patches-bot deleted the series/513719=>bpf-next branch October 7, 2021 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
bpf-next rfc V2
Projects
None yet
Milestone
No milestone
1 participant
@kernel-patches-bot