Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

502 - Bad gateway on Vesta CP admin #1625

Closed
treatys opened this issue Jun 22, 2018 · 113 comments

Comments

@treatys
Copy link

commented Jun 22, 2018

Operating System (OS/VERSION):

Debian 9.4 (x86_64)

VestaCP Version:

0.9.8 (amd64), release 21

Installed Software (what you got with the installer):

Complete production server installation including Bind, Nginx, Apache, MySQL, PgSQL and dovecot+exim.

Steps to Reproduce:

Since yesterday (2018/06/22), Vesta CP panel crashes repeatedly. It reports error 502 - Bad Gateway in nginx. In /var/log/vesta/* there are no errors about this crash reported, only in /var/log/vesta/nginx-error.log, there is mentioned:

2018/06/22 14:35:29 [error] 19059#0: *49 connect() to unix:/var/run/vesta-php.sock failed (111: Connection refused) while connecting to upstream, client: 109.238.216.159, server: _, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/vesta-php.sock:", host: "taira.srv.getmedia.cz:8083"

While I restart vesta service with systemctl restart vesta, I am able to access the panel for a while, however, this is not stable and wont last long.

I don't have any idea while is this happening to me. This bug occurs on all my Vesta CP installations, so whole my cluster (3 nodes) is particullary down.

Do you have any idea what's gone wrong out there?

@treatys

This comment has been minimized.

@saurabhsharma

This comment has been minimized.

Copy link

commented Jun 22, 2018

Hello, Is there any update on this? my server also having exact same issue.

@mehargags

This comment has been minimized.

Copy link

commented Jun 22, 2018

I Can confirm this happens with me too..but only on Apache-less servers that have Nginx + PHP-FPM.

If I restart vesta service, it goes good but then the next day its crashed again. I guess may some cron job is doing this ?

I tried all default Cron Jobs for user admin... all went fine but after I ran
sudo /usr/local/vesta/bin/v-update-user-stats
Vesta Panel threw 502.
I restarted service and tried it again... this time it did not crash. So can't really say it is because of some crons. I have separate cron logs enabled but can't see any error recorded.

/var/log/vesta/nginx-error.log gives
[error] 95978#0: *25 connect() to unix:/var/run/vesta-php.sock failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: _, request: "GET /list/user/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/vesta-php.sock:", host: "IP:8908", referrer: "https://IP:8908/login/"

I am running it on port 8908 and not 8083

@korsar4eg

This comment has been minimized.

Copy link

commented Jun 24, 2018

Same issue, any updates ?
P.S. Start happening couple days ago

@saurabhsharma

This comment has been minimized.

Copy link

commented Jun 24, 2018

@gd-euss

This comment has been minimized.

Copy link

commented Jun 24, 2018

Same problem

@dpeca

This comment has been minimized.

Copy link
Collaborator

commented Jun 24, 2018

What is a sys load of your servers?

@gd-euss

This comment has been minimized.

Copy link

commented Jun 24, 2018

sys load is low. I have got 1 website running on whole virtual server.
Vesta fails even after update.
I think problem is somewhere in the cron jobs, because it fails every x minutes after vesta service restart.
Error from nginx-error.log: 2018/06/24 17:45:22 [error] 29355#0: *29 connect() to unix:/var/run/vesta-php.sock failed (111: Connection refused) while connecting to upstream, client: (my ip), server: _, request: "GET /list/updates/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/vesta-php.sock:", host: "mydomain.tld:8083", referrer: "https://mydomain.tld:8083/login/"

UPD: when appears 502 BAD Gateway, Vesta service status shows that vesta still active and running fine.

@petarkozic

This comment has been minimized.

Copy link

commented Jun 24, 2018

Same problem !
Same logs!
Same as gd-euss

@ram108

This comment has been minimized.

Copy link

commented Jun 24, 2018

My Vesta works as usual. Hetzner cloud, Ubuntu 16.04, nginx + php7.0-fpm

@treatys

This comment has been minimized.

Copy link
Author

commented Jun 24, 2018

I can confirm that the problem is caused by this CRON script:

/usr/local/vesta/bin/v-update-user-stats

Everything works until the cronjob is called. Also, as mentioned before, all my servers have really low sys load.

@maciejkrol

This comment has been minimized.

Copy link

commented Jun 25, 2018

I have the same issue here. Ubuntu 16 AWS (t2.small)
AMI ID: ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20180522 (ami-963cecf4)

@treatys @mehargags
Suspending /usr/local/vesta/bin/v-update-user-stats did not help at all.
I suspended later all CRON jobs did not help at all either.

@Skamasle

This comment has been minimized.

Copy link
Collaborator

commented Jun 25, 2018

Changing pm from dynamic to ondemand and reboot help me

@Anton98567

This comment has been minimized.

Copy link

commented Jun 25, 2018

@Skamasle sorry, but what "PM" means?

@anton-reutov

This comment has been minimized.

Copy link
Collaborator

commented Jun 25, 2018

@Anton98567, Process manager

@uskaritel

This comment has been minimized.

Copy link

commented Jun 25, 2018

Confirmed. I have the same issue on CentOS 7

@hivanski

This comment has been minimized.

Copy link

commented Jun 26, 2018

Freshly updated VestaCP have the same bug.

[2018-06-24] Security fix release 0.9.8-22
Debian 9

@mohamad4j

This comment has been minimized.

Copy link

commented Jun 26, 2018

I have same problem. Debian 9 vesta 0.9.8

@stanma9107

This comment has been minimized.

Copy link

commented Jun 26, 2018

I have the same problem
System:ubuntu 16.04

@Anton98567

This comment has been minimized.

Copy link

commented Jun 26, 2018

Same problem
Ubuntu 16.04.4 LTS
vesta 0.9.8-22

Guys, how to fix it?

tail -f /usr/local/vesta/log/nginx-error.log
 [error] 1855#0: *44 connect() to unix:/var/run/vesta-php.sock failed (111: Connection refused) while connecting to upstream, client: , server: _, request: "POST /api/index.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/vesta-php.sock:", host:
@serghey-rodin

This comment has been minimized.

Copy link
Owner

commented Jun 26, 2018

Thanks for reporting about this issue. The fix will be available soon.

@progcode

This comment has been minimized.

Copy link

commented Jun 26, 2018

Same as on Ubuntu 16.04

@quasiDigi

This comment has been minimized.

Copy link

commented Jun 26, 2018

Some error on centOs 7

tail -f /usr/local/vesta/log/nginx-error.log
     [error] 816#0: *580 connect() to unix:/var/run/vesta-php.sock failed (111: Connection refused) while connecting to upstream, client: , server: , request: "POST /softaculous/index.php?act=cp_api&api=serialize HTTP/1.1", upstream: "fastcgi://unix:/var/run/vesta-php.sock:", host: ""
@serghey-rodin

This comment has been minimized.

Copy link
Owner

commented Jun 26, 2018

It seems like this issue is caused by default pm.max_requests value.
Please try out the following fix

sudo wget http://c.vestacp.com/tmp/php-fpm.conf -O /usr/local/vesta/php/etc/php-fpm.conf
sudo service vesta restart

I'm going to test it next few hours and push as update if everything is fine.
Thanks

@quasiDigi

This comment has been minimized.

Copy link

commented Jun 26, 2018

Great. Everything seems to work fine again.

Thx @serghey-rodin

@petarkozic

This comment has been minimized.

Copy link

commented Jun 26, 2018

@serghey-rodin
Yes, this definitely works !

@Anton98567

This comment has been minimized.

Copy link

commented Jun 26, 2018

@serghey-rodin good job!

@gd-euss

This comment has been minimized.

Copy link

commented Jul 2, 2018

@mahdiyari I'm not sure about what type of OS you using, but have you tried to upgrade your server? for Debian - apt-get update, apt-get upgrade and then try to restart vesta.

@mahdiyari

This comment has been minimized.

Copy link

commented Jul 2, 2018

Already updated and upgraded! that is ubuntu 16.04
Update: Restarted all services manually and there are not 502 errors after 20 minutes
Update: again 502 error

@ninjao

This comment has been minimized.

Copy link

commented Jul 2, 2018

@mahdiyari ok lets hope that works out for you! :)

Did you restart only the "vestacp" related services? Or did you literally restart every service on your machine? Would be good to know for future people unable to do a hard reboot.

Good luck

@mahdiyari

This comment has been minimized.

Copy link

commented Jul 2, 2018

502 again!
I restarted vesta related services!

@dpeca

This comment has been minimized.

Copy link
Collaborator

commented Jul 2, 2018

and deleted unix socket before restart?

@mahdiyari

This comment has been minimized.

Copy link

commented Jul 2, 2018

Of course!

@gd-euss

This comment has been minimized.

Copy link

commented Jul 2, 2018

@mahdiyari Have you tried to have a look on what services have been upgraded and restart them too?

@dpeca

This comment has been minimized.

Copy link
Collaborator

commented Jul 2, 2018

ok, let's then use safe variant that should work for sure
edit /usr/local/vesta/php/etc/php-fpm.conf
find pm=dynamic
set to pm=ondemand

and then
service vesta restart

@mahdiyari

This comment has been minimized.

Copy link

commented Jul 2, 2018

@gd-euss how can I find upgraded services?
@dpeca Edited and restarted.

@gd-euss

This comment has been minimized.

Copy link

commented Jul 2, 2018

@mahdiyari first try dpecas variant and if doesn't work for you then try to find upgrade log in your system logs.

@dpeca

This comment has been minimized.

Copy link
Collaborator

commented Jul 2, 2018

i believe it will work because that way issue will be avoided (whatever it is)

@mahdiyari

This comment has been minimized.

Copy link

commented Jul 2, 2018

No, again 502 error!

@anton-reutov

This comment has been minimized.

Copy link
Collaborator

commented Jul 2, 2018

@mahdiyari you should try to reboot your server

@dpeca

This comment has been minimized.

Copy link
Collaborator

commented Jul 2, 2018

does it works even one second?

@mahdiyari

This comment has been minimized.

Copy link

commented Jul 2, 2018

I can't reboot my server @anton-reutov
I can access admin panel after restarting vesta service, but it shows 502 after a while

@gd-euss

This comment has been minimized.

Copy link

commented Jul 2, 2018

@mahdiyari your logs might be in /var/log/dist-upgrade (Not sure because never used Ubuntu)

@mahdiyari

This comment has been minimized.

Copy link

commented Jul 2, 2018

I restarted all services (except ssh), let's see what will happen!

@ninjao

This comment has been minimized.

Copy link

commented Jul 2, 2018

@mahdiyari you are running out of options :/

I can think of checking all your logs, making sure to resolve everything else... and restarting other services on your machine. I am only thinking of last resort options... restart your network interface? But this might also cause issues with your blockchain node. Also be careful when restarting network interface while using SSH...

I tried so many things to get my server running again, but in the end a reboot was probably what solved the issue.

Edit: Ah I see you have done something, good luck!

@mahdiyari

This comment has been minimized.

Copy link

commented Jul 2, 2018

Now, I don't have any option and still, there is 502 error
A bug comes with a silent update (release 21) and we can't solve that without a reboot!
if anyone found a working solution (without reboot) please comment here
Thanks for your time:)

Update: No matter what you have in your server! You should reboot to solve. A bad development:)

@quasiDigi

This comment has been minimized.

Copy link

commented Jul 2, 2018

everything seems to work fine after update, upgrade and reboot.
Thx

@niccolomineo

This comment has been minimized.

Copy link

commented Aug 27, 2018

@mahdiyari did you solve it? I am still fighting with it.

@Skamasle

This comment has been minimized.

Copy link
Collaborator

commented Aug 27, 2018

@niccolomineo you update and reboot ?

If you not reboot server may you cant solve it

@niccolomineo

This comment has been minimized.

Copy link

commented Aug 27, 2018

Thank you, I restarted NGINX, I hope that sorts it!

edit: Nope, problem's still there.

I did:

rm -f /var/run/vesta-php.sock
service vesta restart
service nginx restart
@Skamasle

This comment has been minimized.

Copy link
Collaborator

commented Aug 28, 2018

Try reboot your server.

@niccolomineo

This comment has been minimized.

Copy link

commented Aug 28, 2018

I did and it is working now. Let's see for how long it lasts (hopefully forever).

@gd-euss

This comment has been minimized.

Copy link

commented Aug 29, 2018

apt-get update, apt-get upgrade, reboot will help you.

@TarKHaoS

This comment has been minimized.

Copy link

commented Sep 24, 2018

apt-get update, apt-get upgrade, reboot will help you.

This definitely worked for me. Cheers!

@Olegu13

This comment has been minimized.

Copy link

commented Oct 2, 2018

This worked:

rm -f /var/run/vesta-php.sock
service vesta restart 
service nginx restart 
reboot
@koolamusic

This comment has been minimized.

Copy link

commented Jan 25, 2019

Nginx just seems to have a problem in CentOS builds
It still cannot connect

@evagabond

This comment has been minimized.

Copy link

commented Feb 15, 2019

This worked:

rm -f /var/run/vesta-php.sock
service vesta restart 
service nginx restart 
reboot

This worked for me too!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.