New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ECONNRESET when deploying to docker swarm with mysql 5.7 #9739

Closed
Toub opened this Issue Jul 19, 2018 · 5 comments

Comments

Projects
None yet
3 participants
@Toub
Copy link

Toub commented Jul 19, 2018

Issue Summary

When deploying ghost on docker swarm with mysql 5.7.22, I got the following error after some time of inactivity.

The bug is very similar to #9694

Here is the full trace from logs:

ERROR [2018-07-19 08:13:25] "GET /" 500 39ms

NAME: InternalServerError
CODE: ECONNRESET
MESSAGE: select count(distinct posts.id) as aggregate from `posts` where (`posts`.`status` = 'published') and (`posts`.`page` = false) - read ECONNRESET

level:normal

InternalServerError: select count(distinct posts.id) as aggregate from `posts` where (`posts`.`status` = 'published') and (`posts`.`page` = false) - read ECONNRESET
    at new GhostError (/var/lib/ghost/versions/1.24.8/core/server/lib/common/errors.js:9:26)
    at prepareError (/var/lib/ghost/versions/1.24.8/core/server/web/middleware/error-handler.js:42:19)
    at Layer.handle_error (/var/lib/ghost/versions/1.24.8/node_modules/express/lib/router/layer.js:71:5)
    at trim_prefix (/var/lib/ghost/versions/1.24.8/node_modules/express/lib/router/index.js:315:13)
    at /var/lib/ghost/versions/1.24.8/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/var/lib/ghost/versions/1.24.8/node_modules/express/lib/router/index.js:335:12)
    at next (/var/lib/ghost/versions/1.24.8/node_modules/express/lib/router/index.js:275:10)
    at Layer.handle_error (/var/lib/ghost/versions/1.24.8/node_modules/express/lib/router/layer.js:67:12)
    at trim_prefix (/var/lib/ghost/versions/1.24.8/node_modules/express/lib/router/index.js:315:13)
    at /var/lib/ghost/versions/1.24.8/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/var/lib/ghost/versions/1.24.8/node_modules/express/lib/router/index.js:335:12)
    at Immediate.next (/var/lib/ghost/versions/1.24.8/node_modules/express/lib/router/index.js:275:10)
    at Immediate.<anonymous> (/var/lib/ghost/versions/1.24.8/node_modules/express/lib/router/index.js:635:15)
    at runCallback (timers.js:674:20)
    at tryOnImmediate (timers.js:645:5)
    at processImmediate [as _immediateCallback] (timers.js:617:5)

Error: read ECONNRESET
    at exports._errnoException (util.js:1020:11)
    at TCP.onread (net.js:580:26)
    --------------------
    at Protocol._enqueue (/var/lib/ghost/versions/1.24.8/node_modules/mysql/lib/protocol/Protocol.js:145:48)
    at Connection.query (/var/lib/ghost/versions/1.24.8/node_modules/mysql/lib/Connection.js:208:25)
    at /var/lib/ghost/versions/1.24.8/node_modules/knex/lib/dialects/mysql/index.js:161:18
    at Promise._execute (/var/lib/ghost/versions/1.24.8/node_modules/bluebird/js/release/debuggability.js:303:9)
    at Promise._resolveFromExecutor (/var/lib/ghost/versions/1.24.8/node_modules/bluebird/js/release/promise.js:483:18)
    at new Promise (/var/lib/ghost/versions/1.24.8/node_modules/bluebird/js/release/promise.js:79:10)
    at Client_MySQL._query (/var/lib/ghost/versions/1.24.8/node_modules/knex/lib/dialects/mysql/index.js:155:12)
    at Client_MySQL.query (/var/lib/ghost/versions/1.24.8/node_modules/knex/lib/client.js:206:17)
    at Runner.<anonymous> (/var/lib/ghost/versions/1.24.8/node_modules/knex/lib/runner.js:155:36)
    at Runner.tryCatcher (/var/lib/ghost/versions/1.24.8/node_modules/bluebird/js/release/util.js:16:23)
    at Runner.query (/var/lib/ghost/versions/1.24.8/node_modules/bluebird/js/release/method.js:15:34)
    at /var/lib/ghost/versions/1.24.8/node_modules/knex/lib/runner.js:61:21
    at tryCatcher (/var/lib/ghost/versions/1.24.8/node_modules/bluebird/js/release/util.js:16:23)
    at /var/lib/ghost/versions/1.24.8/node_modules/bluebird/js/release/using.js:185:26
    at tryCatcher (/var/lib/ghost/versions/1.24.8/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/var/lib/ghost/versions/1.24.8/node_modules/bluebird/js/release/promise.js:512:31)

To Reproduce

Here is what I did, I am not sure if the bug occurs also in non-swarm mode.

  1. Create manually the mysql network
docker network create --driver overlay --attachable system_mysql_attachable-net
  1. Déploy mysql stack

mysql-stack.yml

version: "3.1"

networks:
  system_mysql_attachable-net:
    external: true

volumes:
  mysql-data:

secrets:
   mysql_root_password:
     file: ./secrets/mysql_root_password.secret.txt

services:
  mysql57:
    image: mysql:5.7.22
    volumes:
      - mysql-data:/var/lib/mysql
    networks:
      - system_mysql_attachable-net
    secrets:
      - mysql_root_password
    environment:
      - TZ=Europe/Paris
      - MYSQL_ROOT_PASSWORD_FILE=/run/secrets/mysql_root_password
    deploy:
      placement:
        constraints: [node.role == manager]
      restart_policy:
        condition: any
docker  stack deploy -c mysql-stack.yml mysql
  1. Déploy ghost stack

ghost-stack.yml

version: "3"

networks:
  system_mysql_attachable-net:
    external: true

volumes:
  ghost-data:

services:

  ghost:
    image: ghost:1-alpine
    volumes: 
    - ghost-data:/var/lib/ghost/content
    networks:
      - system_mysql_attachable-net
    environment:
      TZ: Europe/Paris
      # see https://docs.ghost.org/docs/config#section-running-ghost-with-config-env-variables
      url: http://myghost.com
      database__client: mysql
      database__connection__host: ${EU_GHOST_DB_HOST}
      database__connection__user: ${EU_GHOST_DB_USER}
      database__connection__password: ${EU_GHOST_DB_PASSWORD}
      database__connection__database: ${EU_GHOST_DB_NAME}
    deploy:
      restart_policy:
        condition: any
docker  stack deploy -c ghost-stack.yml ghost

Technical details:

Docker image from https://github.com/docker-library/ghost/blob/f99bf268e644e076d71f21668a894305517dc2ee/1/debian/Dockerfile

  • Ghost Version: 1.24.8
  • Node Version: 6.14.3
  • Browser/OS: any
  • Database: mysql 5.7.22

This bug is probably related to knex. Current ghost uses 0.14.6. However, I don't see any fix related to this problem in last 0.15 release: https://github.com/tgriesser/knex/blob/master/CHANGELOG.md

@kevinansfield

This comment has been minimized.

Copy link
Contributor

kevinansfield commented Jul 19, 2018

Hey @Toub 👋 We ask that you please do not use GitHub for help or support, the default issue template pointed you to our forum for this type of question 😄 We use GitHub solely for bug-tracking and on-going feature development so we try to keep it noise free.

Many questions can be answered by reviewing our docs for self-hosters, our theme API, or our public API. If you can't find an answer then our forum is a great place to get community support, plus it helps create a central location for searching problems/solutions.

FYI: Many projects have their own support guidelines and GitHub will highlight them for you, or the project owners will use issue templates to point you in the right direction, please read them before opening issues

@Toub

This comment has been minimized.

Copy link
Author

Toub commented Jul 19, 2018

@kevinansfield it looks like a bug, not a support request

@kevinansfield

This comment has been minimized.

Copy link
Contributor

kevinansfield commented Jul 19, 2018

@Toub it's environment-related and you're not using the supported stack so the forum is definitely the right place to get support. If there's a real bug that's found then we can open one either here or on the appropriate repo.

@Toub

This comment has been minimized.

Copy link
Author

Toub commented Jul 19, 2018

We only support the official recommended installation.
https://docs.ghost.org/v1/docs/hosting

My apologies, I just realize that docker environment is not supported, so I guess you can close the issue.

For those interested, I solved the problem by pinging periodically the server (not ideal solution, but easy one).

@macchie

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment