New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error: read ECONNRESET (Between Ghost and MySQL) #30600

Closed
pascalandy opened this Issue Jan 31, 2017 · 7 comments

Comments

Projects
None yet
5 participants
@pascalandy
Copy link

pascalandy commented Jan 31, 2017

GENERAL SUPPORT INFORMATION

I run Docker Swarm Mode 1.13
I run percona (mysql) and Ghost.
It's running good and I can do everthing you expect from a CMS application.

BUG REPORT INFORMATION

But it seems that when the blog have no queries for over 30 minutes, the connection breaks.
To fix this I need to refresh the page 2 times and I goes back to normal.

My question: Can you help me to debug this issue ?

Description

`docker logs myghostcontainer`

ERROR: select count(distinct posts.id) as aggregate from `posts` where `posts`.`status` = 'published' and `posts`.`page` = false - read ECONNRESET
 Rendering Error Page
 Ghost caught a processing error in the middleware layer.
 Error: read ECONNRESET
    at exports._errnoException (util.js:907:11)
    at TCP.onread (net.js:558:26)
    --------------------
    at Protocol._enqueue (/usr/src/app/node_modules/mysql/lib/protocol/Protocol.js:110:48)
    at Connection.query (/usr/src/app/node_modules/mysql/lib/Connection.js:165:25)
    at /usr/src/app/node_modules/knex/lib/dialects/mysql/index.js:152:18
    at Promise._execute (/usr/src/app/node_modules/bluebird/js/release/debuggability.js:300:9)
    at Promise._resolveFromExecutor (/usr/src/app/node_modules/bluebird/js/release/promise.js:481:18)
    at new Promise (/usr/src/app/node_modules/bluebird/js/release/promise.js:77:14)
    at Client_MySQL._query (/usr/src/app/node_modules/knex/lib/dialects/mysql/index.js:146:12)
    at Client_MySQL.query (/usr/src/app/node_modules/knex/lib/client.js:189:17)
    at Runner.<anonymous> (/usr/src/app/node_modules/knex/lib/runner.js:138:36)
    at Runner.tryCatcher (/usr/src/app/node_modules/bluebird/js/release/util.js:16:23)
    at Runner.query (/usr/src/app/node_modules/bluebird/js/release/method.js:15:34)
    at /usr/src/app/node_modules/knex/lib/runner.js:61:21
    at tryCatcher (/usr/src/app/node_modules/bluebird/js/release/util.js:16:23)
    at /usr/src/app/node_modules/bluebird/js/release/using.js:185:26
    at tryCatcher (/usr/src/app/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/usr/src/app/node_modules/bluebird/js/release/promise.js:510:31)
    at Promise._settlePromise (/usr/src/app/node_modules/bluebird/js/release/promise.js:567:18)
    at Promise._settlePromise0 (/usr/src/app/node_modules/bluebird/js/release/promise.js:612:10)
    at Promise._settlePromises (/usr/src/app/node_modules/bluebird/js/release/promise.js:691:18)
    at Promise._fulfill (/usr/src/app/node_modules/bluebird/js/release/promise.js:636:18)
    at PromiseArray._resolve (/usr/src/app/node_modules/bluebird/js/release/promise_array.js:125:19)
    at PromiseArray._promiseFulfilled (/usr/src/app/node_modules/bluebird/js/release/promise_array.js:143:14)
    at Promise._settlePromise (/usr/src/app/node_modules/bluebird/js/release/promise.js:572:26)

When I refresh my browser two times, it back to normal:

`docker logs myghostcontainer`

173.176.62.36 - - [31/Jan/2017:16:27:22 +0000] "GET / HTTP/1.1" 200 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36"
149.154.167.163 - - [31/Jan/2017:16:28:01 +0000] "GET / HTTP/1.1" 200 - "-" "TelegramBot (like TwitterBot)"
173.176.62.36 - - [31/Jan/2017:16:30:14 +0000] "GET / HTTP/1.1" 200 - "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 10_2_1 like Mac OS X) AppleWebKit/602.4.6 (KHTML, like Gecko) Version/10.0 Mobile/14D27 Safari/602.1"
173.176.62.36 - - [31/Jan/2017:18:45:00 +0000] "GET /services/ HTTP/1.1" 200 - "http://edinoblog.cloudwait.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36"
173.176.62.36 - - [31/Jan/2017:18:45:59 +0000] "GET /services/ HTTP/1.1" - - "http://edinoblog.cloudwait.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36"
173.176.62.36 - - [31/Jan/2017:18:46:55 +0000] "GET / HTTP/1.1" - - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36"

Steps to reproduce the issue:**

  1. Deploy Mysql (percona)
  2. Deploy Ghost (0.11.4)
  3. Do not query Ghost for about 30 minutes.

Output of docker version:**

Client:
Version: 1.13.0
API version: 1.25
Go version: go1.7.3
Git commit: 49bf474
Built: Tue Jan 17 09:58:26 2017
OS/Arch: linux/amd64

Server:
Version: 1.13.0
API version: 1.25 (minimum version 1.12)
Go version: go1.7.3
Git commit: 49bf474
Built: Tue Jan 17 09:58:26 2017
OS/Arch: linux/amd64
Experimental: false


### Output of `docker info`:

Containers: 6
Running: 4
Paused: 0
Stopped: 2
Images: 14
Server Version: 1.13.0
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 78
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: active
NodeID: izjhpmacq0mafrf4l60qyfpvw
Is Manager: true
ClusterID: utjx7z2cjp94jbsyafyufud7s
Managers: 1
Nodes: 3
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Node Address: 10.3.128.19
Manager Addresses:
10.3.128.19:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 2f7393a47307a16f8cee44a37b262e8b81021e3e
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.8.14-docker-2
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.954 GiB
Name: N01-par1-dev-d
ID: 7I6L:WW3U:X26O:VYZI:3HZP:H4YU:T73C:SYME:PGZW:VZNR:OZCF:LFAQ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
provider=scaleway(VC1S)
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false


**Additional environment details (AWS, VirtualBox, physical, etc.):**
VPS on scaleway
@aluzzardi

This comment has been minimized.

Copy link
Member

aluzzardi commented Jan 31, 2017

@pascalandy Can you provide more detsails on how those services were deployed?

@pascalandy

This comment has been minimized.

Copy link
Author

pascalandy commented Feb 1, 2017

Sure Andrea! There is 3 nodes.

ID                           HOSTNAME        STATUS  AVAILABILITY  MANAGER STATUS
2wwk6ylidgvcsheuxhpaob42h    N11-par1-dev-d  Ready   Active
35c2v83wn4h9gmuulr8jcztvw    N12-par1-dev-d  Ready   Active
izjhpmacq0mafrf4l60qyfpvw *  N01-par1-dev-d  Ready   Active        Leader

2 networks:

ucjtg6tpsy7h        NETW_db             overlay             swarm
rg5mq9en864n        NETW_front          overlay             swarm

4 containers

docker service ls
ID            NAME        MODE        REPLICAS  IMAGE
6rpkhabild1j  mysql       replicated  1/1       devmtl/percona-fire:5.7.16-10-1
n3m7sxkz91ws  traefik     global      1/1       traefik:v1.1.2-alpine
bpcfa7hxels4  g99999001   replicated  1/1       devmtl/ghost-base:edge_2017-01-31_14-40
yiavnks8nyhv  g99999025   replicated  1/1       devmtl/ghost-base:edge_2017-01-31_14-40

The key here is the Error: read ECONNRESET. It’s like the Ghost ctn can’t find the **Mysql ctn **on the first request (when it’s was on idle for about 30 min). The second request make the blog work normally.

I’ve been messing with this since few weeks now, so I’m welling to do every possible action to resolve this :) Let me know if you need something else to work with.

Cheers!

@AkihiroSuda

This comment has been minimized.

Copy link
Member

AkihiroSuda commented Feb 1, 2017

I wonder this can be related to #29655, but not sure yet.

Do you hit this issue without Swarm-mode? (i.e. docker run on a single node?)

@pascalandy

This comment has been minimized.

Copy link
Author

pascalandy commented Feb 1, 2017

I agree it looks like to same kind of issue.

I wonder this can be related to #29655, but not sure yet.

I have a server running in prod for over 5 months using a docker-compose setup. No issue at all.

 Do you hit this issue without Swarm-mode? (i.e. docker run on a single node?)

@pascalandy

This comment has been minimized.

Copy link
Author

pascalandy commented Feb 7, 2017

I believe it might be related to #30727 (comment). Since I stop using --opt encrypted it runs normally.

Feel free to close this issue.
Cheers!

@macchie

This comment has been minimized.

@pascalandy

This comment has been minimized.

Copy link
Author

pascalandy commented Nov 16, 2018

Thanks for sharing! I decided to use SQLite3 because of the pain but it's good to know that MariaDB is solid for you :0)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment