Skip to content

Add improvements for HA docs #984

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hal-eisen-MG merged 3 commits into from
Aug 26, 2024
Merged

Add improvements for HA docs #984

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e983202
change port numbers for more obvious differences
antoniofilipovic Aug 26, 2024
d8c78e4
change all other ports
antoniofilipovic Aug 26, 2024
f8880ff
fix nuraft log
antoniofilipovic Aug 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 40 additions & 40 deletions pages/clustering/high-availability.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ These two flags are tied to the high availability feature, enable the coordinato
docker run --name instance1 -p 7687:7687 -p 7444:7444 memgraph/memgraph-mage
--data-recovery-on-startup=true \
--replication-restore-state-on-startup=true \
--management-port=10011 \
--management-port=13011 \
--bolt-port=7687 \
--experimental-enabled=high-availability
```
Expand All @@ -106,7 +106,7 @@ docker run --name coord1 -p 7690:7687 -p 7445:7444 memgraph/memgraph-mage
--coordinator-id=1
--experimental-enabled=high-availability
--coordinator-hostname=localhost
--management-port=10121
--management-port=12121
```

Coordinator IDs serve as identifiers, the coordinator port is used for synchronization and log replication between coordinators and management port is used to get health state of cluster from leader coordinator.
Expand Down Expand Up @@ -139,7 +139,7 @@ Here are the environment variables you need to use to set data instance using on

```
export MEMGRAPH_EXPERIMENTAL_ENABLED=high-availability
export MEMGRAPH_MANAGEMENT_PORT=10011
export MEMGRAPH_MANAGEMENT_PORT=13011
export MEMGRAPH_BOLT_PORT=7692
```

Expand All @@ -156,7 +156,7 @@ export MEMGRAPH_BOLT_PORT=7692
export MEMGRAPH_HA_DURABILITY=true
export MEMGRAPH_NURAFT_LOG_FILE="<path-to-log-file>"
export MEMGRAPH_COORDINATOR_HOSTNAME="localhost"
export MEMGRAPH_MANAGEMENT_PORT=10121
export MEMGRAPH_MANAGEMENT_PORT=12121
```

When using any of these environment variables, flags for `--bolt-port`, `--coordinator-port`, `--coordinator-id`,`--coordinator-hostname` and `--experimental-enabled` will be ignored.
Expand Down Expand Up @@ -207,8 +207,8 @@ If you are running `ADD COORDINATOR` Cypher queries on coordinator with --coordi
coordinators you need to use two `ADD COORDINATOR` queries:

```
ADD COORDINATOR 2 WITH CONFIG {"bolt_server": "127.0.0.1:7691", "coordinator_server": "127.0.0.1:10112"};
ADD COORDINATOR 3 WITH CONFIG {"bolt_server": "127.0.0.1:7692", "coordinator_server": "127.0.0.1:10113"};
ADD COORDINATOR 2 WITH CONFIG {"bolt_server": "127.0.0.1:7691", "coordinator_server": "127.0.0.1:10112", "management_server": "127.0.0.1:12112"};
ADD COORDINATOR 3 WITH CONFIG {"bolt_server": "127.0.0.1:7692", "coordinator_server": "127.0.0.1:10113", "management_server": "127.0.0.1:12113"};
```

</Callout>
Expand Down Expand Up @@ -560,12 +560,12 @@ You can directly use initialization file `HA_register.cypher`:

```

ADD COORDINATOR 2 WITH CONFIG {"bolt_server": "coord2:7691", "coordinator_server": "coord2:10112", "management_server": "coord2:10122"};
ADD COORDINATOR 3 WITH CONFIG {"bolt_server": "coord3:7692", "coordinator_server": "coord3:10113", "management_server": "coord3:10123"};
ADD COORDINATOR 2 WITH CONFIG {"bolt_server": "coord2:7691", "coordinator_server": "coord2:10112", "management_server": "coord2:12122"};
ADD COORDINATOR 3 WITH CONFIG {"bolt_server": "coord3:7692", "coordinator_server": "coord3:10113", "management_server": "coord3:12123"};

REGISTER INSTANCE instance_1 WITH CONFIG {"bolt_server": "instance1:7687", "management_server": "instance1:10011", "replication_server": "instance1:10001"};
REGISTER INSTANCE instance_2 WITH CONFIG {"bolt_server": "instance2:7688", "management_server": "instance2:10012", "replication_server": "instance2:10002"};
REGISTER INSTANCE instance_3 WITH CONFIG {"bolt_server": "instance3:7689", "management_server": "instance3:10013", "replication_server": "instance3:10003"};
REGISTER INSTANCE instance_1 WITH CONFIG {"bolt_server": "instance1:7687", "management_server": "instance1:13011", "replication_server": "instance1:10001"};
REGISTER INSTANCE instance_2 WITH CONFIG {"bolt_server": "instance2:7688", "management_server": "instance2:13012", "replication_server": "instance2:10002"};
REGISTER INSTANCE instance_3 WITH CONFIG {"bolt_server": "instance3:7689", "management_server": "instance3:13013", "replication_server": "instance3:10003"};
SET INSTANCE instance_3 TO MAIN;

```
Expand All @@ -591,7 +591,7 @@ services:
- ./HA_register.cypher:/tmp/init/HA_register.cypher:ro
environment:
- MEMGRAPH_HA_CLUSTER_INIT_QUERIES=/tmp/init/HA_register.cypher
command: [ "--init-file=/tmp/init/license.cypher", "--log-level=TRACE", "--data-directory=/tmp/mg_data_coord1", "--log-file=/tmp/coord1.log", "--also-log-to-stderr", "--coordinator-id=1", "--coordinator-port=10111", "--management-port=10121", "--coordinator-hostname=coord1", "--experimental-enabled=high-availability"]
command: [ "--init-file=/tmp/init/license.cypher", "--log-level=TRACE", "--data-directory=/tmp/mg_data_coord1", "--log-file=/tmp/coord1.log", "--also-log-to-stderr", "--coordinator-id=1", "--coordinator-port=10111", "--management-port=12121", "--coordinator-hostname=coord1", "--experimental-enabled=high-availability"]
networks:
memgraph_ha:
ipv4_address: 172.21.0.4
Expand All @@ -601,7 +601,7 @@ services:
container_name: coord2
volumes:
- ./license.cypher:/tmp/init/license.cypher:ro
command: [ "--init-file=/tmp/init/license.cypher", "--log-level=TRACE", "--data-directory=/tmp/mg_data_coord2", "--log-file=/tmp/coord2.log", "--also-log-to-stderr", "--coordinator-id=2", "--coordinator-port=10112", "--management-port=10122", "--coordinator-hostname=coord2", "--experimental-enabled=high-availability"]
command: [ "--init-file=/tmp/init/license.cypher", "--log-level=TRACE", "--data-directory=/tmp/mg_data_coord2", "--log-file=/tmp/coord2.log", "--also-log-to-stderr", "--coordinator-id=2", "--coordinator-port=10112", "--management-port=12122", "--coordinator-hostname=coord2", "--experimental-enabled=high-availability"]
networks:
memgraph_ha:
ipv4_address: 172.21.0.2
Expand All @@ -611,7 +611,7 @@ services:
container_name: coord3
volumes:
- ./license.cypher:/tmp/init/license.cypher:ro
command: [ "--init-file=/tmp/init/license.cypher", "--log-level=TRACE", "--data-directory=/tmp/mg_data_coord3", "--log-file=/tmp/coord3.log", "--also-log-to-stderr", "--coordinator-id=3", "--coordinator-port=10113", "--management-port=10123", "--coordinator-hostname=coord3", "--experimental-enabled=high-availability"]
command: [ "--init-file=/tmp/init/license.cypher", "--log-level=TRACE", "--data-directory=/tmp/mg_data_coord3", "--log-file=/tmp/coord3.log", "--also-log-to-stderr", "--coordinator-id=3", "--coordinator-port=10113", "--management-port=12123", "--coordinator-hostname=coord3", "--experimental-enabled=high-availability"]

networks:
memgraph_ha:
Expand All @@ -622,7 +622,7 @@ services:
container_name: instance1
volumes:
- ./license.cypher:/tmp/init/license.cypher:ro
command: ["--init-file=/tmp/init/license.cypher","--data-recovery-on-startup=true", "--log-level=TRACE", "--data-directory=/tmp/mg_data_instance1", "--log-file=/tmp/instance1.log", "--also-log-to-stderr", "--management-port=10011", "--experimental-enabled=high-availability"]
command: ["--init-file=/tmp/init/license.cypher","--data-recovery-on-startup=true", "--log-level=TRACE", "--data-directory=/tmp/mg_data_instance1", "--log-file=/tmp/instance1.log", "--also-log-to-stderr", "--management-port=13011", "--experimental-enabled=high-availability"]
networks:
memgraph_ha:
ipv4_address: 172.21.0.6
Expand All @@ -632,7 +632,7 @@ services:
container_name: instance2
volumes:
- ./license.cypher:/tmp/init/license.cypher:ro
command: ["--init-file=/tmp/init/license.cypher","--data-recovery-on-startup=true", "--log-level=TRACE", "--data-directory=/tmp/mg_data_instance2", "--log-file=/tmp/instance2.log", "--also-log-to-stderr", "--management-port=10012", "--experimental-enabled=high-availability"]
command: ["--init-file=/tmp/init/license.cypher","--data-recovery-on-startup=true", "--log-level=TRACE", "--data-directory=/tmp/mg_data_instance2", "--log-file=/tmp/instance2.log", "--also-log-to-stderr", "--management-port=13012", "--experimental-enabled=high-availability"]
networks:
memgraph_ha:
ipv4_address: 172.21.0.7
Expand All @@ -642,7 +642,7 @@ services:
container_name: instance3
volumes:
- ./license.cypher:/tmp/init/license.cypher:ro
command: ["--init-file=/tmp/init/license.cypher","--data-recovery-on-startup=true", "--log-level=TRACE", "--data-directory=/tmp/mg_data_instance3", "--log-file=/tmp/instance3.log", "--also-log-to-stderr", "--management-port=10013", "--experimental-enabled=high-availability"]
command: ["--init-file=/tmp/init/license.cypher","--data-recovery-on-startup=true", "--log-level=TRACE", "--data-directory=/tmp/mg_data_instance3", "--log-file=/tmp/instance3.log", "--also-log-to-stderr", "--management-port=13013", "--experimental-enabled=high-availability"]
networks:
memgraph_ha:
ipv4_address: 172.21.0.8
Expand All @@ -661,32 +661,32 @@ This example will show how to set up a highly available cluster in Memgraph usin

1. Start coordinator1:
```plaintext
docker run --name coord1 -p 7690:7690 -p 7444:7444 memgraph/memgraph-mage --bolt-port=7690 --log-level=TRACE --data-directory=/tmp/mg_data_coord3 --log-file=/tmp/coord1.log --also-log-to-stderr --coordinator-id=1 --coordinator-port=10111 --management-port=10121 --experimental-enabled=high-availability --coordinator-hostname=localhost
docker run --name coord1 -p 7690:7690 -p 7444:7444 memgraph/memgraph-mage --bolt-port=7690 --log-level=TRACE --data-directory=/tmp/mg_data_coord3 --log-file=/tmp/coord1.log --also-log-to-stderr --coordinator-id=1 --coordinator-port=10111 --management-port=12121 --experimental-enabled=high-availability --coordinator-hostname=localhost --nuraft-log-file=/tmp/nuraft/coord1.log
```

2. Start coordinator2:
```plaintext
docker run --name coord2 -p 7691:7691 -p 7445:7444 memgraph/memgraph-mage --bolt-port=7691 --log-level=TRACE --data-directory=/tmp/mg_data_coord2 --log-file=/tmp/coord2.log --nuraft-log-file=/tmp/nuraft/coord2.log --also-log-to-stderr --coordinator-id=2 --coordinator-port=10112 --management-port=10122 --experimental-enabled=high-availability --coordinator-hostname=localhost
docker run --name coord2 -p 7691:7691 -p 7445:7444 memgraph/memgraph-mage --bolt-port=7691 --log-level=TRACE --data-directory=/tmp/mg_data_coord2 --log-file=/tmp/coord2.log --also-log-to-stderr --coordinator-id=2 --coordinator-port=10112 --management-port=12122 --experimental-enabled=high-availability --coordinator-hostname=localhost --nuraft-log-file=/tmp/nuraft/coord2.log
```

3. Start coordinator3:
```plaintext
docker run --name coord3 -p 7692:7692 -p 7446:7444 memgraph/memgraph-mage --bolt-port=7692 --log-level=TRACE --data-directory=/tmp/mg_data_coord3 --log-file=/tmp/coord3.log --nuraft-log-file=/tmp/nuraft/coord1.log --also-log-to-stderr --coordinator-id=3 --coordinator-port=10113 --management-port=10123 --experimental-enabled=high-availability --coordinator-hostname=localhost
docker run --name coord3 -p 7692:7692 -p 7446:7444 memgraph/memgraph-mage --bolt-port=7692 --log-level=TRACE --data-directory=/tmp/mg_data_coord3 --log-file=/tmp/coord3.log --also-log-to-stderr --coordinator-id=3 --coordinator-port=10113 --management-port=12123 --experimental-enabled=high-availability --coordinator-hostname=localhost --nuraft-log-file=/tmp/nuraft/coord3.log
```

4. Start instance1:
```plaintext
docker run --name instance1 -p 7687:7687 -p 7447:7444 memgraph/memgraph-mage --bolt-port=7687 --log-level=TRACE --data-directory=/tmp/mg_data_instance1 --log-file=/tmp/instance1.log --also-log-to-stderr --management-port=10011 --experimental-enabled=high-availability --data-recovery-on-startup=true
docker run --name instance1 -p 7687:7687 -p 7447:7444 memgraph/memgraph-mage --bolt-port=7687 --log-level=TRACE --data-directory=/tmp/mg_data_instance1 --log-file=/tmp/instance1.log --also-log-to-stderr --management-port=13011 --experimental-enabled=high-availability --data-recovery-on-startup=true
```

5. Start instance2:
```plaintext
docker run --name instance2 -p 7688:7688 -p 7448:7444 memgraph/memgraph-mage --bolt-port=7688 --log-level=TRACE --data-directory=/tmp/mg_data_instance2 --log-file=/tmp/instance2.log --also-log-to-stderr --management-port=10012 --experimental-enabled=high-availability --data-recovery-on-startup=true
docker run --name instance2 -p 7688:7688 -p 7448:7444 memgraph/memgraph-mage --bolt-port=7688 --log-level=TRACE --data-directory=/tmp/mg_data_instance2 --log-file=/tmp/instance2.log --also-log-to-stderr --management-port=13012 --experimental-enabled=high-availability --data-recovery-on-startup=true
```

6. Start instance3:
```plaintext
docker run --name instance3 -p 7689:7689 -p 7449:7444 memgraph/memgraph-mage --bolt-port=7689 --log-level=TRACE --data-directory=/tmp/mg_data_instance3 --log-file=/tmp/instance3.log --also-log-to-stderr --management-port=10013 --experimental-enabled=high-availability --data-recovery-on-startup=true
docker run --name instance3 -p 7689:7689 -p 7449:7444 memgraph/memgraph-mage --bolt-port=7689 --log-level=TRACE --data-directory=/tmp/mg_data_instance3 --log-file=/tmp/instance3.log --also-log-to-stderr --management-port=13013 --experimental-enabled=high-availability --data-recovery-on-startup=true
```

### Register instances
Expand All @@ -698,8 +698,8 @@ mgconsole --port=7690
2. Connect the other two coordinator instances to the cluster.

```plaintext
ADD COORDINATOR 2 WITH CONFIG {"bolt_server": "localhost:7691", "coordinator_server": "localhost:10112", "management_server": "localhost:10122"};
ADD COORDINATOR 3 WITH CONFIG {"bolt_server": "localhost:7692", "coordinator_server": "localhost:10113", "management_server": "localhost:10123"};
ADD COORDINATOR 2 WITH CONFIG {"bolt_server": "localhost:7691", "coordinator_server": "localhost:10112", "management_server": "localhost:12122"};
ADD COORDINATOR 3 WITH CONFIG {"bolt_server": "localhost:7692", "coordinator_server": "localhost:10113", "management_server": "localhost:12123"};
```

3. Register 3 data instances as part of the cluster:
Expand All @@ -711,9 +711,9 @@ Replace `<ip_address>` with the container's IP address. This is necessary for Do
</Callout>

```plaintext
REGISTER INSTANCE instance_1 WITH CONFIG {"bolt_server": "localhost:7687", "management_server": "localhost:10011", "replication_server": "localhost:10001"};
REGISTER INSTANCE instance_2 WITH CONFIG {"bolt_server": "localhost:7688", "management_server": "localhost:10012", "replication_server": "localhost:10002"};
REGISTER INSTANCE instance_3 WITH CONFIG {"bolt_server": "localhost:7689", "management_server": "localhost:10013", "replication_server": "localhost:10003"};
REGISTER INSTANCE instance_1 WITH CONFIG {"bolt_server": "localhost:7687", "management_server": "localhost:13011", "replication_server": "localhost:10001"};
REGISTER INSTANCE instance_2 WITH CONFIG {"bolt_server": "localhost:7688", "management_server": "localhost:13012", "replication_server": "localhost:10002"};
REGISTER INSTANCE instance_3 WITH CONFIG {"bolt_server": "localhost:7689", "management_server": "localhost:13013", "replication_server": "localhost:10003"};
```

4. Set instance_3 as MAIN:
Expand All @@ -726,12 +726,12 @@ SET INSTANCE instance_3 TO MAIN;

| name | bolt_server | coordinator_server | management_server | health | role | last_succ_resp_ms |
| ------------- | -------------- | ------------------ | ----------------- | ------ | -------- | ---------------- |
| coordinator_1 | localhost:7690 | localhost:10111 | localhost:10121 | up | leader | 0 |
| coordinator_2 | localhost:7691 | localhost:10112 | localhost:10122 | up | follower | 16 |
| coordinator_3 | localhost:7692 | localhost:10113 | localhost:10123 | up | follower | 25 |
| instance_1 | localhost:7687 | "" | localhost:10011 | up | replica | 39 |
| instance_2 | localhost:7688 | "" | localhost:10012 | up | replica | 21 |
| instance_3 | localhost:7689 | "" | localhost:10013 | up | main | 91 |
| coordinator_1 | localhost:7690 | localhost:10111 | localhost:12121 | up | leader | 0 |
| coordinator_2 | localhost:7691 | localhost:10112 | localhost:12122 | up | follower | 16 |
| coordinator_3 | localhost:7692 | localhost:10113 | localhost:12123 | up | follower | 25 |
| instance_1 | localhost:7687 | "" | localhost:13011 | up | replica | 39 |
| instance_2 | localhost:7688 | "" | localhost:13012 | up | replica | 21 |
| instance_3 | localhost:7689 | "" | localhost:13013 | up | main | 91 |

### Check automatic failover

Expand All @@ -740,11 +740,11 @@ that and automatically promote the first alive REPLICA to become the new MAIN. T

| name | bolt_server | coordinator_server | management_server | health | role | last_succ_resp_ms |
| ------------- | -------------- | ------------------ | ----------------- | ------ | -------- | ------------------|
| coordinator_1 | localhost:7690 | localhost:10111 | localhost:10121 | up | leader | 0 |
| coordinator_2 | localhost:7691 | localhost:10112 | localhost:10122 | up | follower | 34 |
| coordinator_3 | localhost:7692 | localhost:10113 | localhost:10123 | up | follower | 28 |
| instance_1 | localhost:7687 | "" | localhost:10011 | up | main | 61 |
| instance_2 | localhost:7688 | "" | localhost:10012 | up | replica | 74 |
| instance_3 | localhost:7689 | "" | localhost:10013 | down | unknown | 71222 |
| coordinator_1 | localhost:7690 | localhost:10111 | localhost:12121 | up | leader | 0 |
| coordinator_2 | localhost:7691 | localhost:10112 | localhost:12122 | up | follower | 34 |
| coordinator_3 | localhost:7692 | localhost:10113 | localhost:12123 | up | follower | 28 |
| instance_1 | localhost:7687 | "" | localhost:13011 | up | main | 61 |
| instance_2 | localhost:7688 | "" | localhost:13012 | up | replica | 74 |
| instance_3 | localhost:7689 | "" | localhost:13013 | down | unknown | 71222 |

</Steps>