Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

409 error when starting caprover on captain-nginx container #761

Closed
claudiofrs opened this issue Jul 10, 2020 · 5 comments
Closed

409 error when starting caprover on captain-nginx container #761

claudiofrs opened this issue Jul 10, 2020 · 5 comments

Comments

@claudiofrs
Copy link

claudiofrs commented Jul 10, 2020

What is the problem?

Hi, after rebooting my machine, I cannot access my caprover dashboard port 3000. The ip and dns setting of my domain is fine. I can access my ip machine and give me nginx page in return. After doing troubleshooting on https://caprover.com/docs/troubleshooting.html I still got this error and my caprover still not running. I don't know is this docker issue, or caprover issue.

What should I do to make my caprover running again?
Hereby I attach captain service logs below. Please kindly check.

Thank you

root@caprover-ubuntu-s-1vcpu-1gb-sgp1-01:~# docker service logs captain-captain --since 60m
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | 
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | > caprover@0.0.0 start /usr/src/app
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | > node ./built/server.js
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | 
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | Captain Starting ...
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | Overriding skipVerifyingDomains from /captain/data/config-override.json
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:08:59.498 pm    Emptying generated and temp folders.
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:08:59.509 pm    Ensuring directories are available on host. Started.
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:08:59.516 pm    Ensuring directories are available on host. Finished.
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:08:59.524 pm    Network captain-overlay-network is already attached to service: captain-captain
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:08:59.532 pm    captain-captain (service) has already been connected to secret: captain-salt
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:08:59.534 pm    Migration not needed, skipping.
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:08:59.540 pm    Copying fake certificates...
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:08:59.546 pm    Updating Load Balancer - Setting up NGINX conf file...
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:08:59.546 pm    Locking NGINX configuration reloading...
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:08:59.560 pm    SUCCESS: UNLocking NGINX configuration reloading...
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:08:59.567 pm    Captain Nginx is already running.. 
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:08:59.572 pm    Updating NGINX service...
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:08:59.587 pm    Waiting for 5 seconds for nginx reload to take into effect
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:09:04.594 pm    NGINX is fully set up and working...
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:09:04.599 pm    Captain Certbot is already running.. 
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:09:04.601 pm    Updating Certbot service...
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:09:07.636 pm    Retrying to get containerId for captain-certbot retry count:0
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:09:10.641 pm    Retrying to get containerId for captain-certbot retry count:1
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:09:13.650 pm    Retrying to get containerId for captain-certbot retry count:2
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:09:16.654 pm    Retrying to get containerId for captain-certbot retry count:3
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:09:19.661 pm    Retrying to get containerId for captain-certbot retry count:4
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:09:22.667 pm    Retrying to get containerId for captain-certbot retry count:5
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:09:22.671 pm    executeCommand Container: captain-certbot certbot certificates --non-interactive 
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:09:24.481 pm    **** Captain is initialized and ready to serve you! ****
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:09:34.493 pm    executeCommand Container: captain-certbot certbot certificates --non-interactive 
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:09:35.754 pm    executeCommand Container: captain-certbot certbot renew --non-interactive 
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:09:37.198 pm    Updating Load Balancer - renewAllCerts
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:09:37.198 pm    Locking NGINX configuration reloading...
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:09:37.222 pm    SUCCESS: UNLocking NGINX configuration reloading...
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:09:37.223 pm    sendReloadSignal...
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:09:40.227 pm    Retrying to get containerId for captain-nginx retry count:0
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:09:43.234 pm    Retrying to get containerId for captain-nginx retry count:1
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:09:46.238 pm    Retrying to get containerId for captain-nginx retry count:2
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:09:49.245 pm    Retrying to get containerId for captain-nginx retry count:3
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | July 10th 2020, 5:09:49.255 pm    Kill HUP Container: fc2ec4ccfcf687d63a6097a7c6066a777f5ef3a007ddcb598b1ea728e828d826
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | (node:25) UnhandledPromiseRejectionWarning: Error: (HTTP code 409) unexpected - Cannot kill container: fc2ec4ccfcf687d63a6097a7c6066a777f5ef3a007ddcb598b1ea728e828d826: Container fc2ec4ccfcf687d63a6097a7c6066a777f5ef3a007ddcb598b1ea728e828d826 is not running 
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    |     at /usr/src/app/node_modules/docker-modem/lib/modem.js:301:17
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    |     at getCause (/usr/src/app/node_modules/docker-modem/lib/modem.js:331:7)
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    |     at Modem.buildPayload (/usr/src/app/node_modules/docker-modem/lib/modem.js:300:5)
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    |     at IncomingMessage.<anonymous> (/usr/src/app/node_modules/docker-modem/lib/modem.js:275:14)
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    |     at IncomingMessage.emit (events.js:326:22)
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    |     at endReadableNT (_stream_readable.js:1226:12)
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    |     at processTicksAndRejections (internal/process/task_queues.js:80:21)
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | (Use `node --trace-warnings ...` to show where the warning was created)
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | (node:25) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | (node:25) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | GET / 302 6.059 ms - 100
captain-captain.1.vu2w7xz4li4g@caprover-ubuntu-s-1vcpu-1gb-sgp1-01    | GET / 302 1.012 ms - 100

i don't understand, the container says its running but I can't access on port 3000 even with firewall sat up

root@caprover-ubuntu-s-1vcpu-1gb-sgp1-01:~# docker container ls
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS               NAMES
ef5fdba4c3c3        caprover/certbot-sleeping:v1.5.0   "/bin/sh -c 'sleep 9…"   2 minutes ago       Up About a minute   80/tcp, 443/tcp     captain-certbot.1.qtr5ciktint88kcmk0izl8bsv
01f22f2bd64e        caprover/caprover:1.7.1            "docker-entrypoint.s…"   2 minutes ago       Up 2 minutes        3000/tcp            captain-captain.1.fcvve9rsmu5v7vz8wc87328ua
e6c8ba239d3b        phpmyadmin/phpmyadmin:5.0.2        "/docker-entrypoint.…"   29 minutes ago      Up 29 minutes       80/tcp              srv-captain--mysql-phpmyadmin.1.i3joormese1rhgyqliq5vwk20
ed1467f6d237        bitnami/mariadb:10.1               "/opt/bitnami/script…"   29 minutes ago      Up 29 minutes       3306/tcp            srv-captain--maria-db.1.hfjqjhopul4ppq77ok5elu15a
cd2ab49e2aeb        img-captain-klikku:28              "docker-php-entrypoi…"   29 minutes ago      Up 29 minutes       80/tcp              srv-captain--klikku.1.3bji4fiduiomdrwk5kt3i8klu
108ff9fe46e8        bitnami/ghost:2.16.4               "/app-entrypoint.sh …"   29 minutes ago      Up 29 minutes       2368/tcp            srv-captain--blog.1.h4f36t4x1xduwsh6lnkg75omh
aa4cca17e18f        img-captain-testing-gatsby-0:0     "sh -c 'pwd; nginx -…"   29 minutes ago      Up 29 minutes       80/tcp              srv-captain--testing-gatsby-0.1.w6j75ko5cwdos8xb3th1hxolc
63698b0ac438        img-captain-ssh-container:1        "/usr/sbin/sshd -D"      29 minutes ago      Up 29 minutes       22/tcp              srv-captain--ssh-container.1.ecioei1hrkntx3dojjmjyneoe
@githubsaturn
Copy link
Collaborator

You nginx isn't running. Probably it is making your instance to restart.

1- Do you have any custom nginx configurations?
2- what's the output of docker service logs captain-nginx --since 10m ?

@claudiofrs
Copy link
Author

claudiofrs commented Jul 11, 2020

no, i don't have custom nginx config applied as long as I remember. and when I logs captain-nginx the output is empty, no logs. i was wondering why is that...

root@caprover-ubuntu-s-1vcpu-1gb-sgp1-01:~# docker service logs captain-nginx --since 10m
root@caprover-ubuntu-s-1vcpu-1gb-sgp1-01:~# docker service logs captain-nginx --since 60m
root@caprover-ubuntu-s-1vcpu-1gb-sgp1-01:~# docker service logs captain-nginx --since 60m --follow

^C
root@caprover-ubuntu-s-1vcpu-1gb-sgp1-01:~# 

here is /etc/nginx/nginx.conf content

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
	worker_connections 768;
	# multi_accept on;
}

http {

	##
	# Basic Settings
	##

	sendfile on;
	tcp_nopush on;
	tcp_nodelay on;
	keepalive_timeout 65;
	types_hash_max_size 2048;
	# server_tokens off;

	# server_names_hash_bucket_size 64;
	# server_name_in_redirect off;

	include /etc/nginx/mime.types;
	default_type application/octet-stream;

	##
	# SSL Settings
	##

	ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
	ssl_prefer_server_ciphers on;

	##
	# Logging Settings
	##

	access_log /var/log/nginx/access.log;
	error_log /var/log/nginx/error.log;

	##
	# Gzip Settings
	##

	gzip on;

	# gzip_vary on;
	# gzip_proxied any;
	# gzip_comp_level 6;
	# gzip_buffers 16 8k;
	# gzip_http_version 1.1;
	# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

	##
	# Virtual Host Configs
	##

	include /etc/nginx/conf.d/*.conf;
	include /etc/nginx/sites-enabled/*;
}


#mail {
#	# See sample authentication script at:
#	# http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
# 
#	# auth_http localhost/auth.php;
#	# pop3_capabilities "TOP" "USER";
#	# imap_capabilities "IMAP4rev1" "UIDPLUS";
# 
#	server {
#		listen     localhost:110;
#		protocol   pop3;
#		proxy      on;
#	}
# 
#	server {
#		listen     localhost:143;
#		protocol   imap;
#		proxy      on;
#	}
#}

and no captain-root.conf file inside conf.d on my etc/nginx directory but exist on captain/generated/nginx


when I do docker service scale captain-nginx=0 and scale it back to 1, it gives me endless process on preparing and starting

root@caprover-ubuntu-s-1vcpu-1gb-sgp1-01:/captain/generated/nginx/conf.d# docker service scale captain-nginx=0
captain-nginx scaled to 0
overall progress: 0 out of 0 tasks 
verify: Service converged 
root@caprover-ubuntu-s-1vcpu-1gb-sgp1-01:/captain/generated/nginx/conf.d# docker service scale captain-nginx=1
captain-nginx scaled to 1
overall progress: 0 out of 1 tasks 
1/1: preparing [=================================>                 ] 

@githubsaturn
Copy link
Collaborator

/etc/nginx/nginx.conf is not related to captain-nginx! It's for the nginx sitting on your host.

You probably have another nginx that's sitting on your host and it's in conflict with the nginx. Either you, or a script that you used, started another nginx process. You have two options:

1- Simply start over from a brand new instance.
2- Find out what's stopping captain-nginx from running and shut that down. It's more complex. But for start, you can get more info by running

docker service ps captain-nginx --no-trunc

Is it providing any more info?

@claudiofrs
Copy link
Author

I'm choosing to start over from new instance. Thanks for the help 🙏

@ramon159
Copy link

if anyone has the same problem in the future related to nginx not running, run the following command to find out who is using port 443:
sudo lsof -i:22

and if it is apache2 on ubuntu, run:
"sudo service apache2 stop"
or
"sudo systemctl disable apache2.service
"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants