Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[cluster setup]The first setup 3 agency is failed. #12533

Open
yangchao0512 opened this issue Aug 26, 2020 · 0 comments
Open

[cluster setup]The first setup 3 agency is failed. #12533

yangchao0512 opened this issue Aug 26, 2020 · 0 comments

Comments

@yangchao0512
Copy link

My Environment

ArangoDB Version: 3.6.3
Storage Engine: RocksDB
Deployment Mode: Cluster
Deployment Strategy: Manual Start
Configuration: cluster setup details
Infrastructure: own
Operating System: centos7
Total RAM in your machine: 64GiB for 6 machines
Disks in use: HDD
Used Package: RedHat .rpm

Component, Query & Data

Affected feature:

The cluster can‘t use anymore

AQL query (if applicable):

AQL explain (if applicable):

Dataset:

Size of your Dataset on disk:
50GB

Replication Factor & Number of Shards (Cluster only):

Replication Factor 3
Number of Shards 3

Steps to reproduce

agency node:
56.18.30.240
56.18.30.241
56.18.30.242
dbserver node and coordinator node:
56.18.30.243
56.18.30.244
56.18.30.245

  1. 56.18.30.240
    arangod --server.endpoint tcp://0.0.0.0:8531
    --agency.my-address tcp://56.18.30.240:8531
    --server.authentication false
    --agency.activate true
    --agency.size 3
    --agency.endpoint tcp://56.18.30.241:8531
    --agency.endpoint tcp://56.18.30.242:8531
    --agency.supervision true
    --database.directory /var/lib/arangodb3/agency8531/data

  2. 56.18.30.241
    arangod --server.endpoint tcp://0.0.0.0:8531
    --agency.my-address tcp://56.18.30.241:8531
    --server.authentication false
    --agency.activate true
    --agency.size 3
    --agency.endpoint tcp://56.18.30.240:8531
    --agency.endpoint tcp://56.18.30.242:8531
    --agency.supervision true
    --database.directory /var/lib/arangodb3/agency8531/data

  3. 56.18.30.242
    arangod --server.endpoint tcp://0.0.0.0:8531
    --agency.my-address tcp://56.18.30.242:8531
    --server.authentication false
    --agency.activate true
    --agency.size 3
    --agency.endpoint tcp://56.18.30.240:8531
    --agency.endpoint tcp://56.18.30.241:8531 \
    --agency.supervision true
    --database.directory /var/lib/arangodb3/agency8531/data

Problem:
The third node(242) error message:
[root@localhost arangodb3]# arangod --server.endpoint tcp://0.0.0.0:8531 \

--agency.my-address tcp://56.18.30.242:8531
--server.authentication false
--agency.activate true
--agency.size 3
--agency.endpoint tcp://56.18.30.240:8531
--agency.endpoint tcp://56.18.30.241:8531
--agency.supervision true
--database.directory /var/lib/arangodb3/agency8531/data
2020-08-26T01:22:35Z [25679] INFO [e52b0] ArangoDB 3.6.3-1 [linux] 64bit, using jemalloc, build tags/v3.6.3.1-0-g76b2e00b64b-dirty, VPack 0.1.33, RocksDB 6.2.0, ICU 58.1, V8 7.1.302.28, OpenSSL 1.1.1g 21 Apr 2020
2020-08-26T01:22:35Z [25679] INFO [75ddc] detected operating system: Linux version 3.10.0-123.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) ) #1 SMP Mon Jun 30 12:09:22 UTC 2014
2020-08-26T01:22:35Z [25679] INFO [25362] {memory} Available physical memory: 67333894144
2020-08-26T01:22:35Z [25679] WARNING [3e451] {memory} It is recommended to set NUMA to interleaved.
2020-08-26T01:22:35Z [25679] WARNING [b25a4] {memory} put 'numactl --interleave=all' in front of your command
2020-08-26T01:22:35Z [25679] INFO [43396] {authentication} Jwt secret not specified, generating...
2020-08-26T01:22:35Z [25679] INFO [144fe] using storage engine 'rocksdb'
2020-08-26T01:22:35Z [25679] INFO [3bb7d] {cluster} Starting up with role AGENT
2020-08-26T01:22:35Z [25679] INFO [6ea38] using endpoint 'http+tcp://0.0.0.0:8531' for non-encrypted requests
2020-08-26T01:22:35Z [25679] INFO [a1c60] {syscall} file-descriptors (nofiles) hard limit is 8192, soft limit is 8192
2020-08-26T01:22:35Z [25679] INFO [3844e] {authentication} Authentication is turned off, authentication for unix sockets is turned on
2020-08-26T01:22:35Z [25679] INFO [cf3f4] ArangoDB (version 3.6.3-1 [linux]) is ready for business. Have fun!
2020-08-26T01:22:35Z [25679] INFO [7b6f3] {agency} Entering gossip phase ...
2020-08-26T01:22:35Z [25679] ERROR [bed89] {agency} Got error 500 from gossip endpoint
2020-08-26T01:22:36Z [25679] ERROR [bed89] {agency} Got error 500 from gossip endpoint
2020-08-26T01:22:36Z [25679] ERROR [bed89] {agency} Got error 500 from gossip endpoint
2020-08-26T01:22:37Z [25679] ERROR [bed89] {agency} Got error 500 from gossip endpoint
2020-08-26T01:22:39Z [25679] ERROR [bed89] {agency} Got error 500 from gossip endpoint
2020-08-26T01:22:43Z [25679] ERROR [bed89] {agency} Got error 500 from gossip endpoint
2020-08-26T01:22:47Z [25679] ERROR [bed89] {agency} Got error 500 from gossip endpoint
2020-08-26T01:22:51Z [25679] ERROR [bed89] {agency} Got error 500 from gossip endpoint
2020-08-26T01:22:55Z [25679] ERROR [bed89] {agency} Got error 500 from gossip endpoint
2020-08-26T01:22:59Z [25679] ERROR [bed89] {agency} Got error 500 from gossip endpoint
2020-08-26T01:23:03Z [25679] ERROR [bed89] {agency} Got error 500 from gossip endpoint
2020-08-26T01:23:07Z [25679] ERROR [bed89] {agency} Got error 500 from gossip endpoint
2020-08-26T01:23:11Z [25679] ERROR [bed89] {agency} Got error 500 from gossip endpoint
2020-08-26T01:23:15Z [25679] ERROR [bed89] {agency} Got error 500 from gossip endpoint
2020-08-26T01:23:19Z [25679] ERROR [bed89] {agency} Got error 500 from gossip endpoint
2020-08-26T01:23:23Z [25679] ERROR [bed89] {agency} Got error 500 from gossip endpoint
2020-08-26T01:23:27Z [25679] ERROR [bed89] {agency} Got error 500 from gossip endpoint
2020-08-26T01:23:31Z [25679] ERROR [bed89] {agency} Got error 500 from gossip endpoint

Expected result:
how to fix

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants