[GDB][native] Running GDB with a debugee crashes #48

shahab-vahedi · 2021-04-23T14:36:06Z

Runing gdb alone is fine.

$ gdb -q
(gdb)

But running it with a debugee crashes the simulator (both QEMU and nSIM):

# gdb -q /bin/ls

Oops
Path: /usr/bin/gdb
CPU: 0 PID: 109 Comm: gdb Not tainted 5.6.0 foss-for-synopsys-dwc-arc-processors/binutils-gdb#3
Insn could not be fetched
ECR: 0x00040000 EFA: 0x00000000 ERET: 0x00000000
STAT32: 0x80080802 [IE K     ]  BTA: 0x202ef1e8
BLK: EV_Trap+0xc4/0xc8
 SP: 0x9f1c7f80  FP: 0x00000028
LPS: 0x202bca74 LPE: 0x202bca92 LPC: 0x00000000
r00: 0xffffff9c r01: 0x5f971718 r02: 0x00000000
r03: 0x00000200 r04: 0x2034c5f4 r05: 0x00000001
r06: 0x2034c0d0 r07: 0x00000044 r08: 0x000001b7
r09: 0x00000000 r10: 0x00000000 r11: 0x202ef1e8
r12: 0x80811e64 r13: 0x80080802 r14: 0x00000000
r15: 0x5f971718 r16: 0xffffff9c r17: 0x000003ff
r18: 0x00000000 r19: 0x202bca74 r20: 0x202bca92
r21: 0x80811e64 r22: 0x202ef1e8 r23: 0x00000000
r24: 0x00000000 r25: 0x000001b7

Stack Trace:

nSIM flags

$ nsimdrv -prop=nsim_isa_family=av2hs                                 \
          -prop=nsim_isa_core=3                                       \
          -prop=chipid=0xffff                                         \
          -prop=nsim_isa_atomic_option=1                              \
          -prop=nsim_isa_ll64_option=1                                \
          -prop=nsim_mmu=4                                            \
          -prop=mmu_pagesize=8192                                     \
          -prop=mmu_super_pagesize=2097152                            \
          -prop=mmu_stlb_entries=16                                   \
          -prop=mmu_ntlb_ways=4                                       \
          -prop=mmu_ntlb_sets=128                                     \
          -prop=icache=32768,64,4,0                                   \
          -prop=dcache=16384,64,2,0                                   \
          -prop=nsim_isa_shift_option=3                               \
          -prop=nsim_isa_swap_option=1                                \
          -prop=nsim_isa_bitscan_option=1                             \
          -prop=nsim_isa_sat=1                                        \
          -prop=nsim_isa_div_rem_option=2                             \
          -prop=nsim_isa_mpy_option=9                                 \
          -prop=nsim_isa_enable_timer_0=1                             \
          -prop=nsim_isa_enable_timer_1=1                             \
          -prop=nsim_isa_number_of_interrupts=32                      \
          -prop=nsim_isa_number_of_external_interrupts=32             \
          -prop=isa_counters=1                                        \
          -prop=nsim_isa_pct_counters=8                               \
          -prop=nsim_isa_pct_size=48                                  \
          -prop=nsim_isa_pct_interrupt=1                              \
          -prop=nsim_mem-dev=uart0,kind=dwuart,base=0xf0000000,irq=24 \
          -prop=nsim_isa_aps_feature=1                                \
          -prop=nsim_isa_num_actionpoints=4                           \
          -prop=nsim_isa_rtc_option=1                                 \
          -prop=nsim_isa_ad_option=1                                  \
          -prop=nsim_isa_fpud_div_option=1                            \
          -prop=nsim_isa_fpu_mac_option=1                             \
          -prop=nsim_fast=1 vmlinux

QEMU flags

$ FTP=hostfwd=tcp::10021-:21
$ SSH=hostfwd=tcp::10022-:22
$ TLN=hostfwd=tcp::10023-:23
$ DBG=hostfwd=tcp::$(1337)-:$(1337)

$ qemu-system-arc -M virt                                            \
                  -nographic                                         \
                  -no-reboot                                         \
                  -cpu archs                                         \
                  -netdev user,id=net0,$(FTP),$(SSH),$(TLN),$(DBG)   \
                  -device virtio-net-device,netdev=net0              \
                  -append nokaslr                                    \
                  -kernel vmlinux

environment

toolchain: arc-2021.03 (glibc)

binutils:  arc-2021.03         0d6fe26e921
gcc:       arc-2021.03         72d713caf61
gdb:       arc-2021.03-gdb     3ffb584639c  
newlib:    arc-2021.03         9f32ccbdc
toolchain:                     4c5aa3d
glibc:     arc-glibc-master    9826b03b74
uclibc-ng: upstream            tag v1.0.37
linux: 5.1 from abrodkin's repo (64690b80b  https://github.com/abrodkin/linux.git) with a patch added as attachment at the end of this topic

$ build-all.sh --source-dir    ${SRCDIR}                            \
               --build-dir     ${BLDDIR}                            \
               --install-dir   ${INSTDIR}                           \
               --cpu           archs                                \
               --no-multilib                                        \
               --no-uclibc                                          \
               --glibc                                              \
               --target-cflags "-Og -g3 -fvar-tracking-assignments" \
               --jobs          8                                    \
               --no-pdf                                             \
               --no-auto-checkout                                   \
               --no-auto-pull                                       \
               --no-external-download                               \
               --no-strip                                           \
               --native-gdb                                         \
               --system-expat                                       \
               --no-elf32-strip-target-libs                         \
               --no-native                                          \
               --no-optsize-newlib                                  \
               --no-optsize-libstdc++

tested with a linux image built with the toolchain and glibc
linux:     5.6 release (obtained by buildroot)
busybox: 1.32

vmlinuxaa.zip
vmlinuxab.zip
vmlinuxac.zip

$ cat vmlinuxa* > vmlinux.tar.xz
$ tar xf vmlinux.tar.xz

abrodkin_linux_patch.zip

The text was updated successfully, but these errors were encountered:

abrodkin · 2021-04-23T16:03:35Z

@shahab-vahedi may we move this one in https://github.com/foss-for-synopsys-dwc-arc-processors/toolchain/issues for better tracking as that's what we already have in existing tools but not in a WIP branch?

vineetgarc · 2021-04-23T16:12:35Z

So there are 2 problems here

For some reason userspace passes syscall num 0x000001b7 (439) to kernel while the last valid syscall is 438 (there are NR_syscalls 439 syscall 0 to 438). syscall handler has a off-by-one error where it checks > NR_syscalls - it needs to check >= 439
cmp r8, NR_syscalls
mov.hi r0, -ENOSYS

That would fix the kernel crash and kill the offending user process.

The main problem is userspace invoking this syscall - so we need to see the binary (or have native objdump in rootfs) to do that.

vineetgarc · 2021-04-23T16:15:02Z

@shahab-vahedi can u please upload the rootfs for this issue here.

vineetgarc · 2021-04-23T16:18:18Z

I tried NFS mounting host from qemu, but thats being rejected due to invalid port numbers from ARC side (not sure of this is a qemu problem or something else).

Apr 23 09:06:50 rpc.mountd[1322]: refused mount request from xxx: illegal port 33880
long story short I can't objdump the target libs on target or host, and hence need access to them externally (as stated above)

cupertinomiranda · 2021-04-23T16:32:00Z

@shahab-vahedi may we move this one in https://github.com/foss-for-synopsys-dwc-arc-processors/toolchain/issues for better tracking as that's what we already have in existing tools but not in a WIP branch?

My personal opinion is that if @shahab-vahedi thinks this is a gdb issue, let it stay under his "umbrella" until that is no longer the case, otherwise everything should be in the toolchain.

cupertinomiranda · 2021-04-23T16:32:40Z

Sorry pressed close by mistake

shahab-vahedi · 2021-04-23T17:32:52Z

@shahab-vahedi may we move this one in https://github.com/foss-for-synopsys-dwc-arc-processors/toolchain/issues for better tracking as that's what we already have in existing tools but not in a WIP branch?

My personal opinion is that if @shahab-vahedi thinks this is a gdb issue, let it stay under his "umbrella" until that is no longer the case, otherwise everything should be in the toolchain.

Thanks @cupertinomiranda . That was exactly my goal. When it is solved, I will move it.

shahab-vahedi · 2021-04-23T19:20:04Z

Changing the code in linux-5.6/arch/arc/kernel/entry.S:

80811e48:	208c 9dc6           	cmp	r8,439
80811e4c:	20ca 0f8d ffff ffda 	mov.hi	r0,0xffffffda
80811e54:	0010 000d           	bhi	16	;80811e64 <EV_Trap+0xc4>

to

80811e48:	208c 9dc6           	cmp	r8,439
80811e4c:	20ca 0f8a ffff ffda 	mov.ge	r0,0xffffffda
80811e54:	0010 000a           	bge	16	;80811e64 <EV_Trap+0xc4>

Solves the issue. Thank you @vineetgarc for all the support.

vineetgarc · 2021-04-23T19:38:27Z

Just for completeness whats going on here is

There are 2 faccesat syscall in linux kernel.
#define __NR_faccessat 48 # legacy
#define __NR_faccessat2 439 # added in v5.8
glibc 2.33 is "5.10 kernel capable" meaning it has "fast paths" for 5.10 but falls back if the kernel lacks capabilities (this is generic glibc for all arches). so glibc faccessat() lib call first tries to invoke 439 (and if kernel returns -ENOSYS it would fall back to legacy 48).
ARC Linux (v5.6) being used for this exercise has max syscalls supported 0 to 438, so when it sees syscall 439 it needs to return -ENOSYS so glibc falls back to slow path syscall 48.
But due to off by one kernel bug, it proceeds to handle syscall 439 which causes a null pointer deref and hence the crash.

We have NR_syscall syscalls from [0 .. NR_syscall-1]. However the check for invalid syscall number is "> NR_syscall". This off-by-one error erronesously allows "NR_syscall" itself as valid and when passed causes out-of-bounds syscall-call table access leading to crash. This problem showed up when testing glibc 2.33 (v5.10 kernel capable, includng faccessat2 syscall (439) against a v5.6 kernel with NR_syscalls=439 (0 to 438). Due to the bug, 439 was not returned with -ENOSYS but processed leading to a crash. Link: foss-for-synopsys-dwc-arc-processors#48 Reported-by: Shahab Vahedi <shahab@synopsys.com> Signed-off-by: Vineet Gupta <vgupta@synopsys.com>

vineetgarc · 2021-04-23T19:57:42Z

Patch posted to lkml, will be sent Linus way soon !
http://lists.infradead.org/pipermail/linux-snps-arc/2021-April/005008.html

We have NR_syscall syscalls from [0 .. NR_syscall-1]. However the check for invalid syscall number is "> NR_syscall". This off-by-one error erronesously allows "NR_syscall" itself as valid and when passed causes out-of-bounds syscall-call table access leading to crash. This problem showed up when testing glibc 2.33 (v5.10 kernel capable, includng faccessat2 syscall (439) against a v5.6 kernel with NR_syscalls=439 (0 to 438). Due to the bug, 439 was not returned with -ENOSYS but processed leading to a crash. Link: #48 Reported-by: Shahab Vahedi <shahab@synopsys.com> Signed-off-by: Vineet Gupta <vgupta@synopsys.com>

We have NR_syscall syscalls from [0 .. NR_syscall-1]. However the check for invalid syscall number is "> NR_syscall". This off-by-one error erronesously allows "NR_syscall" itself as valid and when passed causes out-of-bounds syscall-call table access leading to crash. This problem showed up when testing glibc 2.33 (v5.10 kernel capable, includng faccessat2 syscall (439) against a v5.6 kernel with NR_syscalls=439 (0 to 438). Due to the bug, 439 was not returned with -ENOSYS but processed leading to a crash. Link: foss-for-synopsys-dwc-arc-processors#48 Reported-by: Shahab Vahedi <shahab@synopsys.com> Cc: <stable@vger.kernel.org> Signed-off-by: Vineet Gupta <vgupta@synopsys.com>

vineetgarc · 2021-05-04T20:09:06Z

@shahab-vahedi is this still an issue ?

shahab-vahedi · 2021-05-04T20:52:48Z

Not from my side. You know when the patch will land. So you will be a good judge when to close this.

We have NR_syscall syscalls from [0 .. NR_syscall-1]. However the check for invalid syscall number is "> NR_syscall" as opposed to >=. This off-by-one error erronesously allows "NR_syscall" to be treated as valid syscall causeing out-of-bounds access into syscall-call table ensuing a crash (holes within syscall table have a invalid-entry handler but this is beyond the array implementing the table). This problem showed up on v5.6 kernel when testing glibc 2.33 (v5.10 kernel capable, includng faccessat2 syscall 439). The v5.6 kernel has NR_syscalls=439 (0 to 438). Due to the bug, 439 passed by glibc was not handled as -ENOSYS but processed leading to a crash. Link: foss-for-synopsys-dwc-arc-processors#48 Reported-by: Shahab Vahedi <shahab@synopsys.com> Cc: <stable@vger.kernel.org> Signed-off-by: Vineet Gupta <vgupta@synopsys.com>

vineetgarc · 2021-05-17T21:10:25Z

Fix merged in v5.13-rc2 3433adc

commit 3433adc upstream. We have NR_syscall syscalls from [0 .. NR_syscall-1]. However the check for invalid syscall number is "> NR_syscall" as opposed to >=. This off-by-one error erronesously allows "NR_syscall" to be treated as valid syscall causeing out-of-bounds access into syscall-call table ensuing a crash (holes within syscall table have a invalid-entry handler but this is beyond the array implementing the table). This problem showed up on v5.6 kernel when testing glibc 2.33 (v5.10 kernel capable, includng faccessat2 syscall 439). The v5.6 kernel has NR_syscalls=439 (0 to 438). Due to the bug, 439 passed by glibc was not handled as -ENOSYS but processed leading to a crash. Link: foss-for-synopsys-dwc-arc-processors#48 Reported-by: Shahab Vahedi <shahab@synopsys.com> Cc: <stable@vger.kernel.org> Signed-off-by: Vineet Gupta <vgupta@synopsys.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

commit 3433adc upstream. We have NR_syscall syscalls from [0 .. NR_syscall-1]. However the check for invalid syscall number is "> NR_syscall" as opposed to >=. This off-by-one error erronesously allows "NR_syscall" to be treated as valid syscall causeing out-of-bounds access into syscall-call table ensuing a crash (holes within syscall table have a invalid-entry handler but this is beyond the array implementing the table). This problem showed up on v5.6 kernel when testing glibc 2.33 (v5.10 kernel capable, includng faccessat2 syscall 439). The v5.6 kernel has NR_syscalls=439 (0 to 438). Due to the bug, 439 passed by glibc was not handled as -ENOSYS but processed leading to a crash. Link: foss-for-synopsys-dwc-arc-processors/linux#48 Reported-by: Shahab Vahedi <shahab@synopsys.com> Cc: <stable@vger.kernel.org> Signed-off-by: Vineet Gupta <vgupta@synopsys.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

commit 3433adc8bd09fc9f29b8baddf33b4ecd1ecd2cdc upstream. We have NR_syscall syscalls from [0 .. NR_syscall-1]. However the check for invalid syscall number is "> NR_syscall" as opposed to >=. This off-by-one error erronesously allows "NR_syscall" to be treated as valid syscall causeing out-of-bounds access into syscall-call table ensuing a crash (holes within syscall table have a invalid-entry handler but this is beyond the array implementing the table). This problem showed up on v5.6 kernel when testing glibc 2.33 (v5.10 kernel capable, includng faccessat2 syscall 439). The v5.6 kernel has NR_syscalls=439 (0 to 438). Due to the bug, 439 passed by glibc was not handled as -ENOSYS but processed leading to a crash. Link: foss-for-synopsys-dwc-arc-processors/linux#48 Reported-by: Shahab Vahedi <shahab@synopsys.com> Cc: <stable@vger.kernel.org> Signed-off-by: Vineet Gupta <vgupta@synopsys.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

commit 3433adc8bd09fc9f29b8baddf33b4ecd1ecd2cdc upstream. We have NR_syscall syscalls from [0 .. NR_syscall-1]. However the check for invalid syscall number is "> NR_syscall" as opposed to >=. This off-by-one error erronesously allows "NR_syscall" to be treated as valid syscall causeing out-of-bounds access into syscall-call table ensuing a crash (holes within syscall table have a invalid-entry handler but this is beyond the array implementing the table). This problem showed up on v5.6 kernel when testing glibc 2.33 (v5.10 kernel capable, includng faccessat2 syscall 439). The v5.6 kernel has NR_syscalls=439 (0 to 438). Due to the bug, 439 passed by glibc was not handled as -ENOSYS but processed leading to a crash. Link: foss-for-synopsys-dwc-arc-processors/linux#48 Reported-by: Shahab Vahedi <shahab@synopsys.com> Cc: <stable@vger.kernel.org> Signed-off-by: Vineet Gupta <vgupta@synopsys.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: randomhydrosol <randomhydrosol@glassrom.org>

commit 3433adc8bd09fc9f29b8baddf33b4ecd1ecd2cdc upstream. We have NR_syscall syscalls from [0 .. NR_syscall-1]. However the check for invalid syscall number is "> NR_syscall" as opposed to >=. This off-by-one error erronesously allows "NR_syscall" to be treated as valid syscall causeing out-of-bounds access into syscall-call table ensuing a crash (holes within syscall table have a invalid-entry handler but this is beyond the array implementing the table). This problem showed up on v5.6 kernel when testing glibc 2.33 (v5.10 kernel capable, includng faccessat2 syscall 439). The v5.6 kernel has NR_syscalls=439 (0 to 438). Due to the bug, 439 passed by glibc was not handled as -ENOSYS but processed leading to a crash. Link: foss-for-synopsys-dwc-arc-processors/linux#48 Reported-by: Shahab Vahedi <shahab@synopsys.com> Cc: <stable@vger.kernel.org> Signed-off-by: Vineet Gupta <vgupta@synopsys.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

cupertinomiranda closed this as completed Apr 23, 2021

cupertinomiranda reopened this Apr 23, 2021

vineetgarc transferred this issue from foss-for-synopsys-dwc-arc-processors/binutils-gdb Apr 23, 2021

vineetgarc transferred this issue from foss-for-synopsys-dwc-arc-processors/toolchain Apr 23, 2021

shahab-vahedi closed this as completed Apr 28, 2021

shahab-vahedi reopened this Apr 29, 2021

vineetgarc closed this as completed May 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GDB][native] Running GDB with a debugee crashes #48

[GDB][native] Running GDB with a debugee crashes #48

shahab-vahedi commented Apr 23, 2021

abrodkin commented Apr 23, 2021

vineetgarc commented Apr 23, 2021 •

edited

Loading

vineetgarc commented Apr 23, 2021

vineetgarc commented Apr 23, 2021

cupertinomiranda commented Apr 23, 2021

cupertinomiranda commented Apr 23, 2021

shahab-vahedi commented Apr 23, 2021

shahab-vahedi commented Apr 23, 2021 •

edited

Loading

vineetgarc commented Apr 23, 2021

vineetgarc commented Apr 23, 2021

vineetgarc commented May 4, 2021

shahab-vahedi commented May 4, 2021

vineetgarc commented May 17, 2021

[GDB][native] Running GDB with a debugee crashes #48

[GDB][native] Running GDB with a debugee crashes #48

Comments

shahab-vahedi commented Apr 23, 2021

abrodkin commented Apr 23, 2021

vineetgarc commented Apr 23, 2021 • edited Loading

vineetgarc commented Apr 23, 2021

vineetgarc commented Apr 23, 2021

cupertinomiranda commented Apr 23, 2021

cupertinomiranda commented Apr 23, 2021

shahab-vahedi commented Apr 23, 2021

shahab-vahedi commented Apr 23, 2021 • edited Loading

vineetgarc commented Apr 23, 2021

vineetgarc commented Apr 23, 2021

vineetgarc commented May 4, 2021

shahab-vahedi commented May 4, 2021

vineetgarc commented May 17, 2021

vineetgarc commented Apr 23, 2021 •

edited

Loading

shahab-vahedi commented Apr 23, 2021 •

edited

Loading