E-mail delivery

[mvl22]

Several users report a sudden flurry of mail delivery last night.

I also got this, starting: Mon, 13 Apr 2020 22:00:45 +0000.. The previous message before that was at Mon, 06 Apr 2020 19:00:06 +0000.

Could we do some brief investigation into this, looking at the logs?

Mail seems to be coming through as normal (in realish time) now.

[nikolai-b]

Thank you for the times, looking in journalctl

Apr 06 19:01:21 cyclescape7 cyclescape-mailers@5100.service[7133]: /var/www/cyclescape/shared/bundle/ruby/2.5.0/gems/resque-2.0.0/lib/resque/worker.rb:895: warning: pthread_create failed for timer: Resource temporarily unavailable, scheduling broken
Apr 06 19:01:21 cyclescape7 kernel: cgroup: fork rejected by pids controller in /system.slice/system-cyclescape\x2dmailers.slice/cyclescape-mailers@5100.service

Looking at the systemd mailers service:

sudo systemctl status cyclescape-mailers@5100.service
● cyclescape-mailers@5100.service
   Loaded: loaded (/etc/systemd/system/cyclescape-mailers.target.wants/../cyclescape-mailers@.service; indirect; vendor preset: enabled)
   Active: active (running) since Tue 2020-04-14 11:30:44 UTC; 10h ago
 Main PID: 23782 (ruby2.5)
    Tasks: 2000 (limit: 4915)
   CGroup: /system.slice/system-cyclescape\x2dmailers.slice/cyclescape-mailers@5100.service
           └─23782 resque-2.0.0: Waiting for mailers

Apr 14 11:30:44 cyclescape7 systemd[1]: Started cyclescape-mailers@5100.service.

I guess we need to manually update the systemd service to have TasksMax=infinite but I'm a bit unhappy about that....

[mvl22]

Can you help me understand what all that says?

[nikolai-b]

I'm far from an expert but searching the internet for
kernel: cgroup: fork rejected by pids controller
gives torvalds/linux@135b8b3
I think systemd manages its services using cgroups. There is an option to limit the number of forks allowed in a cgroup and the default on our system is 4915 forks, as systemd status has the line Tasks: 2000 (limit: 4915). Resque works by forking each time it gets a job [1].

We can either move away from resque to sidekiq which is more maintained, or we can look at changing the systemd services created by foreman.