Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segmentation fault with Gitea on a Raspi 4 #3271

Closed
LuciferSam86 opened this issue Oct 6, 2019 · 26 comments

Comments

@LuciferSam86
Copy link

@LuciferSam86 LuciferSam86 commented Oct 6, 2019

Describe the bug
When I launch gitea I got "segmentation fault"

To reproduce

  1. Download gitea binary 1.9.3 from here : https://dl.gitea.io/gitea/1.9.3/gitea-1.9.3-linux-arm-6
  2. execute gitea web
  3. Get "segmentation fault"

Expected behaviour
Gitea should executed and deploy his webserver.

Actual behaviour
I get segmentation fault

System
Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:

  • Which model of Raspberry Pi? 4B with 4 GB of RAM
  • Which OS and version (cat /etc/rpi-issue)?
Raspberry Pi reference 2019-09-26
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 80d486687ea77d31fc3fc13cf3a2f8b464e129be, stage2
  • Which firmware version (vcgencmd version)?
Sep 24 2019 17:34:30
Copyright (c) 2012 Broadcom
version cd3add54955f8fa065b414d8fc07c525e7ddffc8 (clean) (release) (start)
  • Which kernel version (uname -a)?
    Linux raspberrypi 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux

Logs
The only log I have is from his stack trace using dbg

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0118d598 in brk ()
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /mnt/bigdata/stuff/gitea.
Use `info auto-load python-scripts [REGEXP]' to list them.
(gdb) where
#0  0x0118d598 in brk ()
#1  0x01167c14 in sbrk ()
#2  0x01140378 in __libc_setup_tls ()
#3  0x0113ff2c in __libc_start_main ()
#4  0x0001022c in _start ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Additional context
This answer from the ticket I opened at gitea github say if you use an older version of raspi kernel it works without problem:
go-gitea/gitea#8373 (comment)

EDIT : https://github.molgen.mpg.de/git-mirror/glibc/blob/20003c49884422da7ffbc459cdeee768a6fee07b/csu/libc-tls.c#L105

This is the function where the segfault happens.

@pelwell

This comment has been minimized.

Copy link
Contributor

@pelwell pelwell commented Oct 6, 2019

To save others following the link:

On my pi running kernel version: 4.19.66-v7+ gitea is working without problems. The other two pi´s - where gitea gave the "segmentation fault" - are running kernel version: 4.19.75-v7+.

@UweHeber

This comment has been minimized.

Copy link

@UweHeber UweHeber commented Oct 9, 2019

Thanks for tracking this this issue. Have the same problem with gitea 1.9.2 to 1.9.4 (arm v6) on my RaspberryPI 4

$ cat /proc/device-tree/model
Raspberry Pi 4 Model B Rev 1.1p

$ uname -a
Linux raspberrypi 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux

$ cat /proc/version
Linux version 4.19.75-v7l+ (dom@buildbot) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611)) #1270 SMP Tue Sep 24 18:51:41 BST 2019

@petergil

This comment has been minimized.

Copy link

@petergil petergil commented Oct 10, 2019

I can confirm that this also affects my Raspberry Pi 3 Model B Rev 1.2 so not restricted to rpi4.

@LuciferSam86

This comment has been minimized.

Copy link
Author

@LuciferSam86 LuciferSam86 commented Oct 10, 2019

Another interesting stuff , when I execute gitea under dbg , it works and it can deploy its webserver. At least on archlinux for raspi 4. But outside dbg I got segfault . Why?

@ciph

This comment has been minimized.

Copy link

@ciph ciph commented Oct 10, 2019

I have the same on my Raspberry Pi 3 B+.

Describe the bug
When I launch gitea I got "segmentation fault".

To reproduce
Download gitea binary 1.7.6, 1.8.0, 1.9.4 from here : https://dl.gitea.io/gitea/1.9.4/gitea-1.9.4-linux-arm-6
execute gitea web
Get "segmentation fault"

Expected behaviour
Gitea should execute and run the server

Actual behaviour
I get segmentation fault and the server does not start

System
Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:

  • Which model of Raspberry Pi? Pi3B+
  • Which OS and version (cat /etc/rpi-issue)? Raspberry Pi reference 2019-07-10 Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 175dfb027ffabd4b8d5080097af0e51ed9a4a56c, stage2
  • Which firmware version (vcgencmd version)? Oct 4 2019 15:20:55 Copyright (c) 2012 Broadcom version 8ebb91b5731d2595643426b007a9cee64217fc15 (tainted) (release) (start_cd)
  • Which kernel version (uname -a)? Linux 4.19.76-v7+ #1272 SMP Fri Oct 4 14:48:47 BST 2019 armv7l GNU/Linux
@LuciferSam86

This comment has been minimized.

Copy link
Author

@LuciferSam86 LuciferSam86 commented Oct 10, 2019

Even under raspbian running gitea under gdb gitea it works and I can connect to gitea webserver.

For someone like me that doesn't understand this stuff it's pretty crazy :)

here the screenshots : https://postimg.cc/gallery/2ba368sw2/

@pelwell

This comment has been minimized.

Copy link
Contributor

@pelwell pelwell commented Oct 10, 2019

I haven't started looking at this yet, but you might be able to speed up the investigation if you can run Gitea under strace and capture the output.

@LuciferSam86

This comment has been minimized.

Copy link
Author

@LuciferSam86 LuciferSam86 commented Oct 10, 2019

Hi pelwell, this is when I execute strace ./gitea web

execve("./gitea", ["./gitea", "web"], 0xbe908704 /* 21 vars */) = 0
brk(NULL)                               = 0x15f2000
brk(0x15f2d18)                          = 0x15f2000
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x24} ---
+++ killed by SIGSEGV (core dumped) +++

this is when I execute strace ./gitea

execve("./gitea", ["./gitea"], 0xbeaf3710 /* 21 vars */) = 0
brk(NULL)                               = 0xed7000
brk(0xed7d18)                           = 0xed7000
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x24} ---
+++ killed by SIGSEGV (core dumped) +++

I suppose you want I run gitea under gdb and run "strace" but I have no idea how to do.
Do you have some guide I can read for doing that?

Thanks!

@focmb

This comment has been minimized.

Copy link

@focmb focmb commented Oct 11, 2019

Same here after the upgrade to kernel 4.19.75-v7l+ (SMP) armv7l. Before the upgrade I am running kernel 4.19.66 and gitea was working fine. After the upgrade with no other changes gitea won't start.

@pelwell

This comment has been minimized.

Copy link
Contributor

@pelwell pelwell commented Oct 11, 2019

I can reproduce this - it's on the list to look at.

@holta

This comment has been minimized.

Copy link

@holta holta commented Oct 11, 2019

Is #3279 [raspberrypi-kernel-headers (1.20190925-2): Wrong exec format in scripts in buster] perhaps related?

@SuNNjek

This comment has been minimized.

Copy link

@SuNNjek SuNNjek commented Oct 11, 2019

Is there a way to temporarily downgrade the kernel again so Gitea can be run again? It seems the old kernel was removed from the repository

@pelwell

This comment has been minimized.

Copy link
Contributor

@pelwell pelwell commented Oct 11, 2019

We never delete firmware builds - you can install the 4.19.66 kernel using sudo rpi-update 11ef4498.

pelwell added a commit that referenced this issue Oct 11, 2019
This reverts commit c0ccb4d
("binfmt_elf: move brk out of mmap when doing direct loader exec")

See: #3271

Signed-off-by: Phil Elwell <phil@raspberrypi.org>
@pelwell

This comment has been minimized.

Copy link
Contributor

@pelwell pelwell commented Oct 11, 2019

I've found the culprit: c0ccb4d

It's now reverted on rpi-4.19.y, and Gitea runs again.

@kees Do you have an idea why this particular executable is failing on Pi?

@tmm1

This comment has been minimized.

Copy link
Contributor

@tmm1 tmm1 commented Oct 11, 2019

I'm experiencing the same issue on 4.19.75-v7l+ with a statically compiled comskip binary. Same binary works on 4.19.71-v7l+

I was able to get a backtrace by forcing core dump (ulimit -c unlimited), then loading that dump into gdb:

[New LWP 6139]
Core was generated by `/mnt/channels/channels-dvr/latest/comskip video.mpg'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x005b2090 in brk ()
(gdb) bt
#0  0x005b2090 in brk ()
#1  0x0058c07e in sbrk ()
#2  0x00564f26 in __libc_setup_tls ()
#3  0x0051d13c in __pthread_initialize_minimal_internal ()
#4  0x00564cfa in __libc_start_main ()
#5  0x0002fede in _start ()
@popcornmix

This comment has been minimized.

Copy link
Collaborator

@popcornmix popcornmix commented Oct 11, 2019

Latest rpi-update kernel may have a fix for this.

@holta

This comment has been minimized.

Copy link

@holta holta commented Oct 11, 2019

FYI the Gitea and Kiwix segmentation faults (kiwix/kiwix-tools#341) are NOT solved for me when I do apt update && apt -y dist-upgrade && reboot to upgrade raspberrypi-kernel (1.20190925+1-1) over (1.20190925-2).

CLARIF: the regression persists on (both) RPi 3 and RPi 4.

@tmm1

This comment has been minimized.

Copy link
Contributor

@tmm1 tmm1 commented Oct 11, 2019

You need to run sudo rpi-update

@holta

This comment has been minimized.

Copy link

@holta holta commented Oct 11, 2019

You need to run sudo rpi-update

Why is that specifically?
Given apt -y dist-upgrade says it already installed kernel (1.20190925+1-1) over (1.20190925-2) -- and then I rebooted.

@tmm1

This comment has been minimized.

Copy link
Contributor

@tmm1 tmm1 commented Oct 11, 2019

Because the fix was made a few hours ago and is not contained in package from 9/25?

@holta

This comment has been minimized.

Copy link

@holta holta commented Oct 11, 2019

Because the fix was made a few hours ago and is not contained in package from 9/25?

Ok then I was mistaken in assuming #3279 (comment) was the fix. Apologies.

Certainly applying kernel (1.20190925+1-1) over (1.20190925-2) is insufficient.

@holta

This comment has been minimized.

Copy link

@holta holta commented Oct 11, 2019

I can confirm that running sudo rpi-update on a Raspberry Pi 4 fixed both problems -- allowing Gitea (iiab/iiab#1999) and Kiwix (iiab/iiab#1993) to start!

FYI kernel goes from...

root@box:~# uname -a
Linux box.lan 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux

To...

root@box:~# uname -a
Linux box.lan 4.19.79-v7l+ #1273 SMP Fri Oct 11 18:23:45 BST 2019 armv7l GNU/Linux
@holta

This comment has been minimized.

Copy link

@holta holta commented Oct 13, 2019

Question: how many days is likely, until regular users get their kernel fixed here?

(via apt update etc)

@ciph

This comment has been minimized.

Copy link

@ciph ciph commented Oct 13, 2019

I just run apt-get update && apt-get dist-upgrade, then rpi-update I am now on 4.19.79-v7+ #1273 SMP Fri Oct 11 18:13:16 BST 2019 armv7l GNU/Linux.

Now gitea is starting successfully with https://dl.gitea.io/gitea/1.9.4/gitea-1.9.4-linux-arm-6
Thank you so much!

@LuciferSam86

This comment has been minimized.

Copy link
Author

@LuciferSam86 LuciferSam86 commented Oct 13, 2019

Welp, it's resolved.
I'm gonna close the ticket.

@SteVader

This comment has been minimized.

Copy link

@SteVader SteVader commented Nov 8, 2019

Thank you all so much for this. I've been wrestling with the issue on and off for almost a month before finding this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.