Nginx doesn't respond back after few days of run (nginx-clojure)

[rachit20]

Hi,
I am running nginx 1.11.0 and compiled the sources along with nginx-clojure-0.4.4 java module . I have written my custom java handlers . I have deployed it on Ubuntu 12 Core machine . Please find my nginx.conf

user root;
worker_processes 12;
events {
worker_connections 5000;
use epoll;
multi_accept on;
}
worker_rlimit_nofile 100000;
http {
include mime.types;
default_type application/octet-stream;
server_tokens off;
keepalive_timeout 65;
keepalive_requests 1000000;
sendfile on;
sendfile_max_chunk 256k;
tcp_nopush on;
tcp_nodelay on;
access_log off;
client_body_buffer_size 512k;
client_max_body_size 50m;
client_header_buffer_size 1k;
large_client_header_buffers 32 512k;
output_buffers 2 32k;
postpone_output 1460;
client_header_timeout 3m;
client_body_timeout 3m;
send_timeout 3m;
open_file_cache max=1000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 5;
open_file_cache_errors off;
jvm_path '/usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so';
jvm_var javaPath '/usr/lib/jvm/java-8-oracle/jre/lib/';
jvm_classpath '/opt/nginx/lib/';
jvm_options "-DMYPID=#{pno}";
jvm_options "-Xmx128m";
jvm_options "-XX:+UseParallelGC";
jvm_options "-XX:ParallelGCThreads=2";

So each worker process is starting at around memory (RES - 17172) but after few days of run it grows like anything (RES - 65090) and suddenly nginx doesn't responds . There will be no response back . As I am a java developer can't really find a way to debug . Kindly help me in fixing as i suspect when it crosses certain memory threshold it stops responding . So the only option left is to restart nginx . Kindly help and please validate my conf .

Regards,
Rachit

[xfeep]

@rachit20
Was there any error message in your error.log ?
BTW with nginx-clojure all nginx worker processes are jvm processes so we can debug them by java method, e.g. show thread stacks by jstack, get stats by jstat, dump jvm heap by jmap.

[rachit20]

No there was no error message printed .Also i cant run jstack or jmap in production as i halts the jvm for sometime which i cant afford . Can you tell me if my nginx.conf has proper configs or if you can recommend some changes if required . When is nginx-clojure 0.4.5 getting release planned?On 24-May-2017 10:41 PM, "Zhang,Yuexiang" <notifications@github.com> wrote:@rachit20 Was there any error message in your error.log ? BTW with nginx-clojure all nginx worker processes are jvm processes so we can debug them by java method, e.g. show thread stacks by jstack, get stats by jstat, dump jvm heap by jmap. I tried attaching but it does not allow on worker processes . Nginx worker process are not recognized as jvm process . How can i attach jstack or jmap with it? —You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread. Disclaimer:This e-mail may contain confidential information and is intended only for the person to whom it is addressed. If you are not the intended recipient you may not disclose, distribute or copy this document in any manner whatsoever.Prime Focus does not accept any liability for damage, loss or expense arising from this e-mail or from accessing its attachments.

[rachit20]

kernel: [7301804.909783] INFO: task nginx:79003 blocked for more than 120 seconds.
May 23 14:09:56 kernel: [7301804.909788] Not tainted 4.4.0-64-generic #85-Ubuntu
May 23 14:09:56 kernel: [7301804.909790] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 23 14:09:56 kernel: [7301804.909791] nginx D ffff8800bae03aa8 0 79003 79002 0x00000000
May 23 14:09:56 kernel: [7301804.909796] ffff8800bae03aa8 ffff88080e4e3800 ffff8808199d4c80 ffff880147590000
May 23 14:09:56 kernel: [7301804.909798] ffff8800bae04000 ffff88012b891158 ffff88012b891170 00000000012a6151
May 23 14:09:56 kernel: [7301804.909800] ffff880813f6b800 ffff8800bae03ac0 ffffffff818384d5 ffff880147590000
May 23 14:09:56 kernel: [7301804.909802] Call Trace:
May 23 14:09:56 kernel: [7301804.909811] [] schedule+0x35/0x80
May 23 14:09:56 kernel: [7301804.909814] [] rwsem_down_read_failed+0xe0/0x140
May 23 14:09:56 kernel: [7301804.909818] [] call_rwsem_down_read_failed+0x14/0x30
May 23 14:09:56 kernel: [7301804.909820] [] ? down_read+0x20/0x30
May 23 14:09:56 kernel: [7301804.909850] [] cifs_reopen_file+0x4e7/0x760 [cifs]
May 23 14:09:56 kernel: [7301804.909863] [] cifs_readpage_worker+0x1cc/0x6b0 [cifs]
May 23 14:09:56 kernel: [7301804.909865] [] ? __wait_on_bit_lock+0x91/0xb0
May 23 14:09:56 kernel: [7301804.909877] [] cifs_readpage+0x78/0x250 [cifs]
May 23 14:09:56 kernel: [7301804.909880] [] generic_file_read_iter+0x312/0x5c0
May 23 14:09:56 kernel: [7301804.909893] [] cifs_strict_readv+0xde/0x100 [cifs]
May 23 14:09:56 kernel: [7301804.909896] [] new_sync_read+0x94/0xd0
May 23 14:09:56 kernel: [7301804.909898] [] __vfs_read+0x26/0x40
May 23 14:09:56 kernel: [7301804.909900] [] vfs_read+0x86/0x130
May 23 14:09:56 kernel: [7301804.909902] [] SyS_pread64+0x95/0xb0
May 23 14:09:56 kernel: [7301804.909905] [] entry_SYSCALL_64_fastpath+0x16/0x71

I have found the above error trace in syslog . It seems there were some I/O issues.Could you please help in fixing this although there are some standard solutions available.

[xfeep]

It seems your computer has large memory but slow disk.
Although reducing vm.dirty_ratio ,e.g. vm.dirty_ratio=10 will work, it's better to reduce I/O operations. e.g.

1. move client_body_temp folder to a RAM disk if your memory is large enough
2. move client_body_temp to a RAID storage.
3. increase proxy cache size if you use Nginx as a proxy server.

[rachit20]

Thanks a lot. Yeah i am using nginx as a proxy server but i have not enabled proxy cache explicitly .Can you please put more light in this.

[xfeep]

Please read this article https://serversforhackers.com/nginx-caching .

[xfeep]

@rachit20 nginx-clojure 0.4.5 released! Its binary distribution is built with nginx 1.12.0.

[rachit20]

Oh great ! is it a stable build of nginx-clojure ?

[xfeep]

Yes. v0.4.5 is a bugfix version.