Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nginx Segfaulted #307

Closed
xerudro opened this issue Jul 8, 2023 · 24 comments
Closed

Nginx Segfaulted #307

xerudro opened this issue Jul 8, 2023 · 24 comments

Comments

@xerudro
Copy link

xerudro commented Jul 8, 2023

Hi, yesterday i decided to try to integrate Imunify360 solution to my VPS with HestiaCP, installed on Debian11 , with NGINX webserver. Because the server was a non-supported version i had to make the configs for integrations and preparations myself, before installing and using Imunify360. As a requirement from the Cloudlinux devs, the ModSecurity had to be installed and configured with clean settings, before the installation of the antivir and firewall solution. I asked them if they can offer me a guide, or atleast something to start from and they offered me this link : https://cloudlinux.zendesk.com/hc/en-us/articles/360018872859-How-to-install-ModSecurity-on-the-nginx-only-server-?source=search, but this is for Almalinux or CentOS, so i searched a google similar one for debian and i found one but for debian10, not 11 and i`ve tried to adapt both instructions from what i found with the one provided by the guys from Cloudlinux... and after few errors on git submodule init and submodule update, and searches on google, for fixes and workarounds "i managed" to finish the modsecurity installation. I started the installation and watched the logs for malware or attacks on the server and i noticed this errors logged by Imunify : flash kernel: [27774.579002] nginx[113980]: segfault at 55aaa76a2f50 ip 00007fb486356c35 sp 00007ffc372ed440 error 4. Searching in the syslog i found this errors : Reloaded nginx - high performance web server.
Jul 8 07:48:15 flash kernel: [27774.074330] nginx[113820]: segfault at 55aaa7555f00 ip 00007fb486356c35 sp 00007ffc372ed440 error 4 in libc-2.31.so[7fb48633e000+159000]
Jul 8 07:48:15 flash kernel: [27774.076600] Code: fe ff 49 83 c5 02 41 0f b7 6d fe 49 89 c6 4c 8d 78 fe 4d 85 e4 75 13 eb cc 0f 1f 40 00 4c 8b 63 08 48 83 c3 08 4d 85 e4 74 bb <66> 41 3b 2c 24 75 ec 49 8d 7c 24 02 4c 89 fa 4c 89 ee e8 04 76 fe
Jul 8 07:48:15 flash kernel: [27774.093507] nginx[113824]: segfault at 55aaa7555f00 ip 00007fb486356c35 sp 00007ffc372ed440 error 4
Jul 8 07:48:15 flash kernel: [27774.093509] nginx[113822]: segfault at 55aaa7555f00 ip 00007fb486356c35 sp 00007ffc372ed440 error 4 in libc-2.31.so[7fb48633e000+159000]
Jul 8 07:48:15 flash kernel: [27774.095361] in libc-2.31.so[7fb48633e000+159000]
Jul 8 07:48:15 flash kernel: [27774.097076] nginx[113823]: segfault at 55aaa7555f00 ip 00007fb486356c35 sp 00007ffc372ed440 error 4 in libc-2.31.so[7fb48633e000+159000]
Jul 8 07:48:15 flash kernel: [27774.097081] Code: fe ff 49 83 c5 02 41 0f b7 6d fe 49 89 c6 4c 8d 78 fe 4d 85 e4 75 13 eb cc 0f 1f 40 00 4c 8b 63 08 48 83 c3 08 4d 85 e4 74 bb <66> 41 3b 2c 24 75 ec 49 8d 7c 24 02 4c 89 fa 4c 89 ee e8 04 76 fe
Jul 8 07:48:15 flash kernel: [27774.097746] Code: fe ff 49 83 c5 02 41 0f b7 6d fe 49 89 c6 4c 8d 78 fe 4d 85 e4 75 13 eb cc 0f 1f 40 00 4c 8b 63 08 48 83 c3 08 4d 85 e4 74 bb <66> 41 3b 2c 24 75 ec 49 8d 7c 24 02 4c 89 fa 4c 89 ee e8 04 76 fe
Jul 8 07:48:15 flash kernel: [27774.098785]
Jul 8 07:48:15 flash kernel: [27774.106411] nginx[113818]: segfault at 55aaa7555f00 ip 00007fb486356c35 sp 00007ffc372ed440 error 4
Jul 8 07:48:15 flash kernel: [27774.107125] nginx[113821]: segfault at 55aaa7555f00 ip 00007fb486356c35 sp 00007ffc372ed440 error 4 in libc-2.31.so[7fb48633e000+159000]
Jul 8 07:48:15 flash kernel: [27774.107131] Code: fe ff 49 83 c5 02 41 0f b7 6d fe 49 89 c6 4c 8d 78 fe 4d 85 e4 75 13 eb cc 0f 1f 40 00 4c 8b 63 08 48 83 c3 08 4d 85 e4 74 bb <66> 41 3b 2c 24 75 ec 49 8d 7c 24 02 4c 89 fa 4c 89 ee e8 04 76 fe
Jul 8 07:48:15 flash kernel: [27774.107214] Code: fe ff 49 83 c5 02 41 0f b7 6d fe 49 89 c6 4c 8d 78 fe 4d 85 e4 75 13 eb cc 0f 1f 40 00 4c 8b 63 08 48 83 c3 08 4d 85 e4 74 bb <66> 41 3b 2c 24 75 ec 49 8d 7c 24 02 4c 89 fa 4c 89 ee e8 04 76 fe
Jul 8 07:48:15 flash kernel: [27774.108524] in libc-2.31.so[7fb48633e000+159000]
Jul 8 07:48:15 flash kernel: [27774.117463] Code: fe ff 49 83 c5 02 41 0f b7 6d fe 49 89 c6 4c 8d 78 fe 4d 85 e4 75 13 eb cc 0f 1f 40 00 4c 8b 63 08 48 83 c3 08 4d 85 e4 74 bb <66> 41 3b 2c 24 75 ec 49 8d 7c 24 02 4c 89 fa 4c 89 ee e8 04 76 fe
Jul 8 07:48:15 flash kernel: [27774.132899] nginx[113819]: segfault at 55aaa7555f00 ip 00007fb486356c35 sp 00007ffc372ed440 error 4 in libc-2.31.so[7fb48633e000+159000]
Jul 8 07:48:15 flash kernel: [27774.135538] Code: fe ff 49 83 c5 02 41 0f b7 6d fe 49 89 c6 4c 8d 78 fe 4d 85 e4 75 13 eb cc 0f 1f 40 00 4c 8b 63 08 48 83 c3 08 4d 85 e4 74 bb <66> 41 3b 2c 24 75 ec 49 8d 7c 24 02 4c 89 fa 4c 89 ee e8 04 76 fe
Jul 8 07:48:15 flash kernel: [27774.138279] nginx[113825]: segfault at 55aaa7555f00 ip 00007fb486356c35 sp 00007ffc372ed440 error 4 in libc-2.31.so[7fb48633e000+159000]
Jul 8 07:48:15 flash kernel: [27774.138286] Code: fe ff 49 83 c5 02 41 0f b7 6d fe 49 89 c6 4c 8d 78 fe 4d 85 e4 75 13 eb cc 0f 1f 40 00 4c 8b 63 08 48 83 c3 08 4d 85 e4 74 bb <66> 41 3b 2c 24 75 ec 49 8d 7c 24 02 4c 89 fa 4c 89 ee e8 04 76 fe
Jul 8 07:48:16 flash kernel: [27774.577842] nginx[113981]: segfault at 55aaa76a2f50 ip 00007fb486356c35 sp 00007ffc372ed440 error 4 in libc-2.31.so[7fb48633e000+159000]
Jul 8 07:48:16 flash kernel: [27774.579002] nginx[113980]: segfault at 55aaa76a2f50 ip 00007fb486356c35 sp 00007ffc372ed440 error 4
Jul 8 07:48:16 flash kernel: [27774.580153] Code: fe ff 49 83 c5 02 41 0f b7 6d fe 49 89 c6 4c 8d 78 fe 4d 85 e4 75 13 eb cc 0f 1f 40 00 4c 8b 63 08 48 83 c3 08 4d 85 e4 74 bb <66> 41 3b 2c 24 75 ec 49 8d 7c 24 02 4c 89 fa 4c 89 ee e8 04 76 fe
Jul 8 07:48:16 flash kernel: [27774.580155] in libc-2.31.so[7fb48633e000+159000]
Jul 8 07:48:16 flash kernel: [27774.585765] Code: fe ff 49 83 c5 02 41 0f b7 6d fe 49 89 c6 4c 8d 78 fe 4d 85 e4 75 13 eb cc 0f 1f 40 00 4c 8b 63 08 48 83 c3 08 4d 85 e4 74 bb <66> 41 3b 2c 24 75 ec 49 8d 7c 24 02 4c 89 fa 4c 89 ee e8 04 76 fe

I searched on google and i`ve found that they are related to ModSecurity modules... Can you guys please help me fix them ?

@martinhsv
Copy link
Contributor

Hello @xerudro ,

You haven't specified exactly what leads you to believe ModSecurity is involved with your issue. That certainly could be the case, but nothing in the output that you have provided suggests that (as far as I can see).

What versions are you using of ModSecurity, ModSecurity-nginx, and nginx are you using?

I can't really do anything with the output that you have provided. If you can provide a stack trace, that might provide some useful information.

In general, some types of issues like that can be caused by incompatibilities -- one such is if you aren't using the same pcre (pcre1 vs. pcre2) in all of those components.

@xerudro
Copy link
Author

xerudro commented Jul 10, 2023

Hi. I am using nginx 1.25.1 but i want to install it on a Debian 11, or 12 ( if possible i want to stick with 12) if not i will revert to 11. I intend to use it with HestiaCP, and imunify360... or if you can suggest a better option except CSF. I don`t like the GUI of CSF. ... I have an VPS where i host few personal sites, and i have 2 clients ( friends ) who host their sites too. Can you please help me with a guide on how to compile it without any issues ? I tried different tuts to test and i had same nginx segfaults errors ... I am so tired of tring to make it work ...

@martinhsv
Copy link
Contributor

martinhsv commented Jul 10, 2023

Hello @xerudro ,

You didn't mention anything about the other pieces of information that I highlighted.

nginx 1.25.1 uses pcre2 by default. In this case, you need to be using at least v1.0.3 of the ModSecurity-nginx connector,

And your ModSecurity (>= v3.0.7) needs to be built with pcre2 (--with-pcre2) during the configure step.

@xerudro
Copy link
Author

xerudro commented Jul 10, 2023

@martinhsv Sorry i didnt because now the server is a clean server. I just rebuilt the server, so the modsecurity and modsecurity-nginx are no more installed on the server. And why i am thinking the modsecurity is the cause is because i didnt had those problems before i tried to install and complile modsecurity. I also tried imunify on directadmin without any issus of that kind, but the support of the panel really sucks. Thats why i decided to try the self-hosted imunify...

@xerudro
Copy link
Author

xerudro commented Jul 10, 2023

Can you please help me with a well documented guide how to install and compile it ? I am not a very techy guy. I was thinking to ask if you can do it for me... but i think it`s inappropriate.

@martinhsv
Copy link
Contributor

For the build, if all you are building yourself is libModSecurity, there are compile recipes here: https://github.com/SpiderLabs/ModSecurity/wiki/Compilation-recipes-for-v3.x

For debian, you should be able to use the one for Ubuntu: https://github.com/SpiderLabs/ModSecurity/wiki/Compilation-recipes-for-v3.x#ubuntu-1804

The main exceptions in your case are likely, that:

  • at the first step (the install stage), you need to include: libpcre2-dev and pcre2-utils
  • as mentioned previously, at the ./configure step you need to specify ./configure --with-pcre2

@xerudro
Copy link
Author

xerudro commented Jul 11, 2023

Hi again, Martin. the libpcre++-dev can be removed from the apt-get line? Because of this :
E: Unable to locate package libpcre++-dev
E: Couldn't find any package by regex 'libpcre++-dev'

@airween
Copy link
Member

airween commented Jul 11, 2023

For Debian 11 you can try our repository for Debian and Ubuntu distributions:

https://modsecurity.digitalwave.hu

It contains both libmodsecurity3 and the connector for Nginx - but we support the Debian's Nginx.

@xerudro
Copy link
Author

xerudro commented Jul 11, 2023

@airween this will work for Debian 12 also ? I tend to stuck with it if possible. If it is,does requires some extra steps and configurations except the ones from the page you provided me ?

@airween
Copy link
Member

airween commented Jul 11, 2023

@airween this will work for Debian 12 also?

please read the page:

Supported Linux distribution(s)

  • Debian 12 a.k.a. Bookworm, amd64
  • Debian 11 a.k.a. Bullseye, amd64
  • Debian 10 a.k.a. Buster, amd64
  • Ubuntu 22.04 (LTS) a.k.a. Jammy, amd64
  • Ubuntu 20.04 (LTS) a.k.a. Focal, amd64
  • Ubuntu 18.04 (LTS) a.k.a. Bionic, amd64

I tend to stuck with it if possible. If it is,does requires some extra steps and configurations except the ones from the page you provided me ?

No, all necessary steps are on the site (including setting of the repository and configuring the server and the engine).

@xerudro
Copy link
Author

xerudro commented Jul 11, 2023

Thanks, but i still have an question mark, ive seen on the page that the version of the nginx is 1.18, on the server its 1.25.1 that won`t create uncompatibility issues ?

@xerudro
Copy link
Author

xerudro commented Jul 11, 2023

@airween i`ve tried to run the steps... and here is what i got :
apt-cache policy libnginx-mod-http-modsecurity libmodsecurity3 modsecurity-crs
libnginx-mod-http-modsecurity:
Installed: (none)
Candidate: 1.0.4-1pre1+020230609bpo12+d59e4ad1
Version table:
1.0.4-1
pre1+020230609bpo12+d59e4ad1 900
500 http://modsecurity.digitalwave.hu/debian bookworm-backports/main amd64 Packages
1.0.3-1+b1 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
500 http://mirror.hetzner.com/debian/packages bookworm/main amd64 Packages
libmodsecurity3:
Installed: 3.0.10-1pre1+020230609bpo12+205dac0e
Candidate: 3.0.10-1
pre1+020230609bpo12+205dac0e
Version table:
*** 3.0.10-1pre1+020230609bpo12+205dac0e 900
500 http://modsecurity.digitalwave.hu/debian bookworm-backports/main amd64 Packages
100 /var/lib/dpkg/status
3.0.9-1 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
500 http://mirror.hetzner.com/debian/packages bookworm/main amd64 Packages
modsecurity-crs:
Installed: (none)
Candidate: 3.3.4-1
bpo12+1
Version table:
3.3.4-1 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
500 http://mirror.hetzner.com/debian/packages bookworm/main amd64 Packages
3.3.4-1bpo12+1 900
500 http://modsecurity.digitalwave.hu/debian bookworm-backports/main amd64 Packages
root@flash:
# apt install libnginx-mod-http-modsecurity
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
libnginx-mod-http-modsecurity : Depends: libnginx-mod-http-ndk but it is not going to be installed
Depends: nginx-abi-1.22.1-7
E: Unable to correct problems, you have held broken packages.

@airween
Copy link
Member

airween commented Jul 11, 2023

Thanks, but i still have an question mark, ive seen on the page that the version of the nginx is 1.18, on the server its 1.25.1 that won`t create uncompatibility issues ?

Nginx version 1.18 is for Debian 11.
Debian 12 contains Nginx 1.22.

The provided packages are compatible with the Debian packages from the official upstream.

@airween
Copy link
Member

airween commented Jul 11, 2023

@xerudro - could you reformat this post? There are many strikethrough text, a bit hard to understand. And please use the code blocks to show the commands and outputs.

The following packages have unmet dependencies: libnginx-mod-http-modsecurity : Depends: libnginx-mod-http-ndk but it is not going to be installed Depends: nginx-abi-1.22.1-7 E: Unable to correct problems, you have held broken packages.

I have no idea where have you got that package, but my current packages do not depend on nginx-abi (yet). Both in official Debian repository and in Digitalwave's repo.

I assume that came from the third repository (http://mirror.hetzner.com/debian/packages), but actually I can't reach now that to check it.

Perhaps you should remove that from your sources list.

@xerudro
Copy link
Author

xerudro commented Jul 11, 2023

Thanks @airween i will rebuild the VPS and remove that repo from sources list, try again and get back to you if it worked.

@xerudro
Copy link
Author

xerudro commented Jul 11, 2023

It didnt work @airween, and i think i found the problem... the digitalwave repo uses nginx 1.22, and the panel installed on the server uses nginx 1.25.1... the repo with the culprit it`s hestia repo... because they use a newer version of nginx. I tried to install modsecurity from the digiralwave repo on a clean server, then install hestia on top of it, and the nginx was unable to restart for the panel installation... and was unable to create the panel subdomain due to a different version of nginx.

@airween
Copy link
Member

airween commented Jul 11, 2023

the digitalwave repo uses nginx 1.22, and the panel installed on the server uses nginx 1.25.1...

Yes, that won't work. We provide 3rd-party packages for stable systems with official packages.

@xerudro
Copy link
Author

xerudro commented Jul 11, 2023

So we get back to @martinhsv. Your solution will work with nginx 1.25.1 on debian 12 ? I havent tested it, i have seen the response from @airween, and i decided to test it first... because was a bit easier ... :)

@xerudro
Copy link
Author

xerudro commented Jul 13, 2023

Ive rebuild the server to ubuntu 22.04, same nginx 1.25.1 version used, i think its inside the HestiaCP repository, cloned the git repository as on the ubuntu 22 documentation you sent me @martinhsv, and i noticed some WARNING messages...

Sorry for keep bothering you, but i am trying to learn first how to handle this things... and how to solve them :

configure: MaxMind library was not found
configure: SSDEEP library was not found
configure: LUA library was not found
configure: WARNING: doxygen not found - will not generate any doxygen documentation
Are this a worry motive or will be solved with nginx-connector ?

@xerudro
Copy link
Author

xerudro commented Jul 17, 2023

@martinhsv @airween , i switched to ubuntu 22.10 and following the https://github.com/SpiderLabs/ModSecurity/wiki/Compilation-recipes-for-v3.x#ubuntu-2210, i got this warnings on compile... configure.ac:106: warning: The macro AC_TRY_COMPILE' is obsolete.
configure.ac:106: You should run autoupdate.
./lib/autoconf/general.m4:2847: AC_TRY_COMPILE is expanded from...
build/lua.m4:118: CHECK_FOR_LUA_AT is expanded from...
build/lua.m4:5: CHECK_LUA is expanded from...
configure.ac:106: the top level
configure.ac:129: warning: The macro AC_TRY_LINK' is obsolete. configure.ac:129: You should run autoupdate. ./lib/autoconf/general.m4:2920: AC_TRY_LINK is expanded from... build/pcre.m4:15: CHECK_PCRE is expanded from... configure.ac:129: the top level configure.ac:140: warning: The macro AC_HEADER_STDC' is obsolete.
configure.ac:140: You should run autoupdate.
./lib/autoconf/headers.m4:704: AC_HEADER_STDC is expanded from...
configure.ac:140: the top level
configure.ac:307: warning: AC_PROG_LEX without either yywrap or noyywrap is obsolete
./lib/autoconf/programs.m4:716: _AC_PROG_LEX is expanded from...
./lib/autoconf/programs.m4:709: AC_PROG_LEX is expanded from...
configure.ac:307: the top level
configure.ac:50: installing './ar-lib'
configure.ac:50: installing './compile'
configure.ac:147: installing './config.guess'
configure.ac:147: installing './config.sub'
configure.ac:45: installing './install-sh'
configure.ac:45: installing './missing'
parallel-tests: installing './test-driver'
examples/multiprocess_c/Makefile.am: installing './depcomp'
configure.ac: installing './ylwrap'
configure.ac:106: warning: The macro AC_TRY_COMPILE' is obsolete. configure.ac:106: You should run autoupdate. ./lib/autoconf/general.m4:2847: AC_TRY_COMPILE is expanded from... build/lua.m4:118: CHECK_FOR_LUA_AT is expanded from... build/lua.m4:5: CHECK_LUA is expanded from... configure.ac:106: the top level configure.ac:129: warning: The macro AC_TRY_LINK' is obsolete.
configure.ac:129: You should run autoupdate.
./lib/autoconf/general.m4:2920: AC_TRY_LINK is expanded from...
build/pcre.m4:15: CHECK_PCRE is expanded from...
configure.ac:129: the top level
configure.ac:140: warning: The macro AC_HEADER_STDC' is obsolete. configure.ac:140: You should run autoupdate. ./lib/autoconf/headers.m4:704: AC_HEADER_STDC is expanded from... configure.ac:140: the top level configure.ac:307: warning: AC_PROG_LEX without either yywrap or noyywrap is obsolete ./lib/autoconf/programs.m4:716: _AC_PROG_LEX is expanded from... ./lib/autoconf/programs.m4:709: AC_PROG_LEX is expanded from... configure.ac:307: the top level
Should i worry about them ?

And if you please help me with a guide on how to configure it with ngx_http_modsecurity_module.so on ubuntu 22.10, please ?

@xerudro
Copy link
Author

xerudro commented Jul 17, 2023

I am using nginx 1.25.1 as i got some problems with the panel, trying to change nginx version to 1.22.

@xerudro
Copy link
Author

xerudro commented Jul 17, 2023

*Sorry my bad, the distro is 22.04.6 LTS.

@martinhsv
Copy link
Contributor

Regarding the software not found (lua, etc.): they indicate that you did not install all of the software dependencies (like the ones on the first line of the Ubuntu 22.10 recipe to which you linked).

The messages regarding 'obsolete' autoconf constructs: autoconf has been deprecating some features (although still supported for now). I have not encountered those myself, but you may be using a more recent version. As they are marked as warnings, I wouldn't expect any difficulties due to them.

As with my previous comment ( #307 (comment) ), if if you use that version (1.22) of nginx, it uses pcre2 by default os you need to do the ModSecurity configure step with '--with-pcre2'.

@martinhsv
Copy link
Contributor

Anything further on this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants