New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rails app runs with OOM killer score set to -1000 #2105

Closed
t-tera opened this Issue Aug 1, 2018 · 5 comments

Comments

Projects
None yet
4 participants
@t-tera

t-tera commented Aug 1, 2018

Question 1: What is the problem?

My Rails app runs with OOM killer score set to -1000 (never killed).
The score should be 0.

As shown below, the score of all Passenger's processes is -1000 while that of Apache's is 0.

[root@localhost ~]# ps auxw | grep -iE "httpd" | grep -v grep
root       1557  0.1  0.3 296792  6972 ?        Ssl  22:40   0:00 /usr/sbin/httpd -DFOREGROUND
apache     1593  0.0  0.1 296792  3516 ?        S    22:40   0:00 /usr/sbin/httpd -DFOREGROUND
apache     1594  0.0  0.2 296792  4712 ?        S    22:40   0:00 /usr/sbin/httpd -DFOREGROUND
apache     1595  0.0  0.1 296792  3516 ?        S    22:40   0:00 /usr/sbin/httpd -DFOREGROUND
apache     1596  0.0  0.2 296792  3748 ?        S    22:40   0:00 /usr/sbin/httpd -DFOREGROUND
apache     1597  0.0  0.2 296792  4712 ?        S    22:40   0:00 /usr/sbin/httpd -DFOREGROUND
apache     1639  0.0  0.1 296792  3516 ?        S    22:40   0:00 /usr/sbin/httpd -DFOREGROUND
[root@localhost ~]# cat /proc/1557/oom_score_adj
0
[root@localhost ~]# cat /proc/1593/oom_score_adj
0
[root@localhost ~]# cat /proc/1594/oom_score_adj
0
[root@localhost ~]# cat /proc/1595/oom_score_adj
0
[root@localhost ~]# cat /proc/1596/oom_score_adj
0
[root@localhost ~]# cat /proc/1597/oom_score_adj
0
[root@localhost ~]# cat /proc/1639/oom_score_adj
0
[root@localhost ~]# ps auxw | grep -iE "passenger" | grep -v grep
root       1578  0.0  0.3 365376  5908 ?        Ssl  22:40   0:00 Passenger watchdog
root       1581  0.1  0.6 669136 12680 ?        SNl  22:40   0:00 Passenger core
apache     1604  1.2  3.8 292664 71332 ?        Sl   22:40   0:01 Passenger AppPreloader: /var/www/rails_projects/first_books_app
apache     1648  0.3  5.1 429288 96588 ?        Sl   22:40   0:00 Passenger AppPreloader: /var/www/rails_projects/first_books_app (forking...)
[root@localhost ~]# cat /proc/1578/oom_score_adj
-1000
[root@localhost ~]# cat /proc/1581/oom_score_adj
-1000
[root@localhost ~]# cat /proc/1604/oom_score_adj
-1000
[root@localhost ~]# cat /proc/1648/oom_score_adj
-1000

My CentOS7 VMWare image is available at:
https://drive.google.com/file/d/1SdyYDuE-LjCR0fHk8jQOxEjbldONp66a/view?usp=sharing
The root user's password is "password".
The app's top page (http://host/books/) shows its PID and OOM killer score.

It looks like this issue is quite similar to #1631.

Question 2: Passenger version and integration mode:

Open source 5.3.3 and Apache integration mode.

Question 3: OS or Linux distro, platform (including version):

CentOS Linux release 7.5.1804 (Core) - x86_64

Question 4: Passenger installation method:

[x] RubyGems, no Gemfile

Question 5: Your app's programming language (including any version managers) and framework (including versions):

Ruby 2.5.0p0, Rails 5.2.0, Apache 2.4.6

Question 6: Are you using a PaaS and/or containerization? If so which one?

No.

Question 7: Anything else about your setup that we should know?

No.

@FooBarWidget

This comment has been minimized.

Member

FooBarWidget commented Aug 1, 2018

Reproduced on Ubuntu 16.04 with a Rails app. User switching turned on. Maybe the ConfigKit refactoring introduced a regression.

@t-tera

This comment has been minimized.

t-tera commented Aug 13, 2018

It looks like Passenger enterprise 5.3.4 has the same problem.
Do I have to report it to somewhere else?

@FooBarWidget

This comment has been minimized.

Member

FooBarWidget commented Aug 13, 2018

You are reporting to the right place.

@sjmadsen

This comment has been minimized.

sjmadsen commented Oct 8, 2018

Is this expected to be addressed soon? I believe this behavior has introduced some instability on one of my production servers. PostgreSQL is putting memory pressure on the system (which is a separate problem I'm working on), but this pressure sometimes triggers the OOM Killer, which skips the large Ruby app processes and ends up killing PostgreSQL itself (which is also using a lot of memory, but only has oom_score_adj set to -900).

@FooBarWidget FooBarWidget added this to the 5.3.6 milestone Oct 10, 2018

@FooBarWidget

This comment has been minimized.

Member

FooBarWidget commented Oct 10, 2018

I have scheduled this for the next version, 5.3.6.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment