Call convention support for x86-64 kernel interface (syscall) %rdi, %…

[therealdreg]

…rsi, %rdx, %r10, %r8, %r9

Your checklist for this pull request

• I've read the guidelines for contributing to this repository
• I made sure to follow the project's coding style
• I've added tests that prove my fix is effective or that my feature works (if possible)
• I've updated the documentation and the radare2 book with the relevant information (if needed)

Detailed description

The current amd64 call convention in r2 supported is User-level call convention: %rdi, %rsi, %rdx, %rcx, %r8, %r9

I add the amd64 kernel interface call convention (amd64syscall): %rdi, %rsi, %rdx, %r10, %r8, %r9

Test plan

Its impossible test it yet, reason: libr/core/cmd_anal.c

// TODO: move this to r_syscall
//TODO replace the hardcoded CC with the sdb ones

Just debug a x86-64-syscall-code with 4 args or more and run "as". Example with wait4 (4 args):

    mov rcx, 0 ; rcx is not an arg in this syscall-case
    mov rdi, 1
    mov rsi, 2
    mov rdx, 3
    mov r10, 4
    mov r8, 5
    mov r9, 6
    mov rax, 61 ; wait4(rdi, rsi, rdx, r10)
    syscall

Closing issues

[therealdreg]

@trufae, @thestr4ng3r, @Maijin Its impossible test it yet, reason: libr/core/cmd_anal.c

// TODO: move this to r_syscall
//TODO replace the hardcoded CC with the sdb ones

[trufae]

@David-Reguera-Garcia-Dreg i have pushed a PR implementing what you need to use this cc :) can you check it out and see if it works as expected? I'm fine to merge your PR like this. it doesnt hurts to have one more cc defined, but the CI must pass green.

thanks!

[therealdreg]

@David-Reguera-Garcia-Dreg i have pushed a PR implementing what you need to use this cc :) can you check it out and see if it works as expected? I'm fine to merge your PR like this. it doesnt hurts to have one more cc defined, but the CI must pass green.

thanks!

Thx for your PR @trufae , but I tried your #17960 and dont works as expected:

:> afc=amd64syscall
:> afc=
amd64syscall
:> afck
..
amd64syscall=cc
...
cc.amd64syscall.arg0=rdi
cc.amd64syscall.arg1=rsi
cc.amd64syscall.arg2=rdx
cc.amd64syscall.arg3=r10
cc.amd64syscall.arg4=r8
cc.amd64syscall.arg5=r9
cc.amd64syscall.ret=rax
...
default.cc=amd64syscall
...
:> dr
rax = 0x0000003d
rbx = 0x00000000
rcx = 0x00401007
rdx = 0x00000000
rsi = 0x00000000
rdi = 0x0000106b
r8 = 0x00000000
r9 = 0x00000000
r10 = 0x00000000
r11 = 0x00000302
r12 = 0x00000000
r13 = 0x00000000
r14 = 0x00000000
r15 = 0x00000000
rip = 0x0040102b
rbp = 0x00000000
rflags = 0x00000202
rsp = 0x7ffccb4a9a70
:> as rax
61 = wait4 (0x0000106b, 0x00000000, 0x00000000, 0x00401007)

-

dreg@fr33project:~/radare2# grep -R R_ANAL_CC_TYPE_STDCALL
grep: warning: libr/bin/d/dll/dll: recursive directory loop

dreg@fr33project:~/radare2# grep -R amd64syscall
libr/anal/d/cc-x86-64.sdb.txt:amd64syscall=cc
libr/anal/d/cc-x86-64.sdb.txt:cc.amd64syscall.arg0=rdi
libr/anal/d/cc-x86-64.sdb.txt:cc.amd64syscall.arg1=rsi
libr/anal/d/cc-x86-64.sdb.txt:cc.amd64syscall.arg2=rdx
libr/anal/d/cc-x86-64.sdb.txt:cc.amd64syscall.arg3=r10
libr/anal/d/cc-x86-64.sdb.txt:cc.amd64syscall.arg4=r8
libr/anal/d/cc-x86-64.sdb.txt:cc.amd64syscall.arg5=r9
libr/anal/d/cc-x86-64.sdb.txt:cc.amd64syscall.ret=rax
Binary file libr/anal/d/cc-x86-64.sdb matches
grep: warning: libr/bin/d/dll/dll: recursive directory loop

As you can see it is still using the old amd64 call convention using %rcx instead of %r10 for the fourth argument

[trufae]

Thats not how it works. See anal.cc and anal.syscc (you are interested in the last one)

[trufae]

See the failing tests please https://github.com/radareorg/radare2/pull/17954/checks?check_run_id=1435913663

[trufae]

Can you rebase?

[therealdreg]

Can you rebase?

done

[trufae]

Can you update the tests please? Use r2r -i

[trufae]

cant commit to your PR. so fixed the test in here #18074