Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add backup&restore #1100

Merged
merged 8 commits into from
Feb 16, 2022
Merged

add backup&restore #1100

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
8d7919b
add backup&restore
foesa-yang Feb 15, 2022
efa86a7
Update 2.compile-br.md
foesa-yang Feb 15, 2022
38bfef2
Update docs-2.0/7.data-security/2.backup-restore/4.br-restore-data.md
foesa-yang Feb 16, 2022
823e3ac
Update docs-2.0/7.data-security/2.backup-restore/4.br-restore-data.md
foesa-yang Feb 16, 2022
7aa48a4
Update docs-2.0/7.data-security/2.backup-restore/4.br-restore-data.md
foesa-yang Feb 16, 2022
81be061
Update docs-2.0/7.data-security/2.backup-restore/4.br-restore-data.md
foesa-yang Feb 16, 2022
e9afef7
Update docs-2.0/7.data-security/2.backup-restore/4.br-restore-data.md
foesa-yang Feb 16, 2022
2676c9f
Update 4.br-restore-data.md
foesa-yang Feb 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 2 additions & 5 deletions docs-2.0/7.data-security/2.backup-restore/2.compile-br.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ To compile the BR, do a check of these:

To compile the BR, follow these steps:

1. Clone the `nebula-storage` repository to your machine.
1. Clone the `nebula-br` repository to your machine.

```bash
git clone https://github.com/vesoft-inc/nebula-br.git
Expand All @@ -35,8 +35,5 @@ Users can enter `bin/br version` on the command line. If the following results a

```bash
[nebula-br]$ bin/br version
Nebula Backup And Restore Ultility Tool,V-0.1.0
GitSha: 8e994dc
GitRef: master
please run "help" subcommand for more infomation.
Nebula Backup And Restore Utility Tool,V-0.6.0
```
136 changes: 87 additions & 49 deletions docs-2.0/7.data-security/2.backup-restore/4.br-restore-data.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,67 +16,105 @@ To restore data with the BR, do a check of these:

- The BR is compiled. For more information, see [Compile BR](2.compile-br.md).

- No application is connected to the target Nebula Graph cluster.

- Make sure that the target and the source Nebula Graph clusters have the same topology, which means that they have exactly the same number of hosts.

- Your account on the BR machine can log on to the meta and the storage servers via SSH without a password. For more information, see[SSH tunnels with keys](http://alexander.holbreich.org/ssh-tunnel-without-password/).

!!! caution

This account must have write privileges to the installation directory of Nebula Graph and the data directory (--data_path) of the meta/storage service.

- To perform the restoration process, a `root` user is required.
- Download [nebula-agent](https://github.com/vesoft-inc/nebula-agent) and start the agent service in each cluster(including metad, storaged, graphd) host.

- If the backup files are stored on Alibaba Cloud OSS or Amazon S3, make sure that the S3 CLI client or ossutil is installed and configured on the meta servers, the storage servers, and the BR machine. For more information, see [Alibaba Cloud ossutil Documentation](https://www.alibabacloud.com/help/zh/doc-detail/120075.htm#concept-303829) and [Amazon S3 CLI Documentation](https://docs.amazonaws.cn/cli/latest/userguide/cli-services-s3.html).

!!! caution
- No application is connected to the target Nebula Graph cluster.

Run `ln -s /<ossutil_tool_installation_path>/<ossutil64 or ossutil> /usr/local/bin/ossutil` to make the `ossutil` command effective.
- Make sure that the target and the source Nebula Graph clusters have the same topology, which means that they have exactly the same number of hosts. The number of data folders for each host is consistently distributed.

## Procedures

1. Users can use the following command to list the existing backup information:

```bash
$ ./bin/br show --storage <ip_address>
```
For example, use the following command to list the backup information in the local `/home/test/backup` path.
```bash
$ ./bin/br show --storage "local:///home/test/backup"
+----------------------------+---------------------+------------------+-------------+---------------+
| NAME | CREATE TIME | SPACES | FULL BACKUP | SPECIFY SPACE |
+----------------------------+---------------------+------------------+-------------+---------------+
| BACKUP_2021_07_29_06_51_09 | 2021-07-29 06:51:09 | basketballplayer | true | true |
| BACKUP_2021_07_29_07_55_08 | 2021-07-29 07:55:09 | basketballplayer | true | true |
+----------------------------+---------------------+------------------+-------------+---------------+
```
```bash
$ ./bin/br show --storage <ip_address>
```
For example, run the following command to list the backup information in the local `/home/nebula/backup` path.
```bash
$ ./bin/br show --storage "local:///home/nebula/backup"
+----------------------------+---------------------+------------------------+-------------+------------+
| NAME | CREATE TIME | SPACES | FULL BACKUP | ALL SPACES |
+----------------------------+---------------------+------------------------+-------------+------------+
| BACKUP_2022_02_10_07_40_41 | 2022-02-10 07:40:41 | basketballplayer | true | true |
| BACKUP_2022_02_11_08_26_43 | 2022-02-11 08:26:47 | basketballplayer,foesa | true | true |
+----------------------------+---------------------+------------------------+-------------+------------+
```

Or, you can run the following command to list the backup information stored in S3 URL `s3://127.0.0.1:9000/br-test/backup`.
```bash
$ ./bin/br show --s3.endpoint "http://127.0.0.1:9000" --storage="s3://br-test/backup/" --s3.access_key=minioadmin --s3.secret_key=minioadmin --s3.region=default
```

| Parameter | Data type | Required | Default value | Description |
foesa-yang marked this conversation as resolved.
Show resolved Hide resolved
| --- | --- | --- | --- | --- |
| `-h,-help` | - | No | None | Checks help for restoration. |
| `-debug` | - | No | None | Checks for more log information. |
| `-log` | string | No | br.log | Specifies detailed log path for restoration and backup. |
| `--storage` | string | Yes | None | The target storage URL of BR backup data. The format is: <Schema\>://<PATH\>. <br>Schema: Optional values are `local` and `s3`. <br>When selecting s3, you need to fill in `s3.access_key`, `s3.endpoint`, `s3.region`, and `s3.secret_key`.<br>PATH: The path of the storage location. |
| `--s3.access_key` | string | No | None | Sets AccessKey ID. |
| `--s3.endpoint` | string | No | None | Sets the S3 endpoint URL, please specify the HTTP or HTTPS scheme explicitly. |
| `--s3.region` | string | No | None | Sets the region or location to upload or download the backup. |
| `--s3.secret_key` | string | No | None | Sets SecretKey for AccessKey ID. |


2. Run the following command to restore data.

```
[root]# ./bin/br restore full --meta <ip_address> --storage <storage_path> --name <backup_name> --user <user_name>
```
```
$ ./bin/br restore full --meta <ip_address> --storage <storage_path> --name <backup_name>
```

For example, run the following command to upload the backup files from the local `/home/nebula/backup/` to the cluster where the meta service's address is `127.0.0.1:9559`.

For example, use the following command to restore the data in the local `/home/foesa/backup` path to the cluster where the address is `192.168.*.*:9559` and the user name of the host where the meta/storage service is located is `root`.
```
$ ./bin/br restore full --meta "127.0.0.1:9559" --storage "local:///home/nebula/backup/" --name BACKUP_2021_12_08_18_38_08
```

```
[root]# ./bin/br restore full --meta "192.168.*.*:9559" --storage "local:///home/test/backup" --name "BACKUP_2021_07_29_07_55_08" --user "root"
```
Or, you can run the following command to upload the backup files from the S3 URL `s3://127.0.0.1:9000/br-test/backup`.
```bash
$ ./bin/br restore full --meta "127.0.0.1:9559" --s3.endpoint "http://127.0.0.1:9000" --storage="s3://br-test/backup/" --s3.access_key=minioadmin --s3.secret_key=minioadmin --s3.region="default" --name BACKUP_2021_12_08_18_38_08
```

If the following information is returned, the data is restored successfully.
```bash
restore successed
```
If the following information is returned, the data is restored successfully.
```bash
Restore succeed.
```

The parameters are as follows.
!!! caution

| Parameter | Required | Default value | Description | Data type |
| --- | --- | --- | --- | --- |
| --concurrent | No | 5 | The maximum number of concurrency for Alibaba Cloud OSS. | int |
| --extra_args | No | None | The parameters of backup storage tools (OSS/HDFS/S3) for backup. | string |
| -h,-help | No | None | Checks help for restoration. | - |
| --meta | Yes | None | The IP address and port of the meta service. | string |
| --name | Yes | None | The backup name. | string |
| --storage | Yes | None | The target storage url of BR backup data. | string |
| --user | Yes | None | The username to login into the hosts where meta/storage service is located. | string |
If your new cluster hosts' IPs are not all the same as the backup cluster, after restoration, you should run `add hosts` to add the Storage host IPs in the new cluster one by one.

The parameters are as follows.

| Parameter | Data type | Required | Default value | Description |
foesa-yang marked this conversation as resolved.
Show resolved Hide resolved
| --- | --- | --- | --- | --- |
| `-h,-help` | - | No | None | Checks help for restoration. |
| `-debug` | - | No | None | Checks for more log information. |
| `-log` | string | No | br.log | Specifies detailed log path for restoration and backup. |
| `-meta` | string | Yes | None | The IP address and port of the meta service. |
| `-name` | string | Yes | None | The name of backup. |
| `--storage` | string | Yes | None | The target storage URL of BR backup data. The format is: \<Schema\>://\<PATH\>. <br>Schema: Optional values are `local` and `s3`. <br>When selecting s3, you need to fill in `s3.access_key`, `s3.endpoint`, `s3.region`, and `s3.secret_key`.<br>PATH: The path of the storage location. |
| `--s3.access_key` | string | No | None | Sets AccessKey ID. |
| `--s3.endpoint` | string | No | None | Sets the S3 endpoint URL, please specify the HTTP or HTTPS scheme explicitly. |
| `--s3.region` | string | No | None | Sets the region or location to upload or download the backup. |
| `--s3.secret_key` | string | No | None | Sets SecretKey for AccessKey ID. |

3. Run the following command to clean up temporary files if any error occurred during backup. It will clean the files in cluster and external storage. You could also use it to clean up old backups files in external storage.

```bash
$ ./bin/br cleanup --meta <ip_address> --storage <storage_path> --name <backup_name>
```

The parameters are as follows.

| Parameter | Data type | Required | Default value | Description |
| --- | --- | --- | --- | --- |
| `-h,-help` | - | No | None | Checks help for restoration. |
| `-debug` | - | No | None | Checks for more log information. |
| `-log` | string | No | br.log | Specifies detailed log path for restoration and backup. |
| `-meta` | string | Yes | None | The IP address and port of the meta service. |
| `-name` | string | Yes | None | The name of backup. |
| `--storage` | string | Yes | None | The target storage URL of BR backup data. The format is: \<Schema\>://\<PATH\>. <br>Schema: Optional values are `local` and `s3`. <br>When selecting s3, you need to fill in `s3.access_key`, `s3.endpoint`, `s3.region`, and `s3.secret_key`.<br>PATH: The path of the storage location. |
| `--s3.access_key` | string | No | None | Sets AccessKey ID. |
| `--s3.endpoint` | string | No | None | Sets the S3 endpoint URL, please specify the HTTP or HTTPS scheme explicitly. |
| `--s3.region` | string | No | None | Sets the region or location to upload or download the backup. |
| `--s3.secret_key` | string | No | None | Sets SecretKey for AccessKey ID. |
11 changes: 5 additions & 6 deletions mkdocs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -334,11 +334,11 @@ nav:
- User management: 7.data-security/1.authentication/2.management-user.md
- Roles and privileges: 7.data-security/1.authentication/3.role-list.md
- OpenLDAP authentication: 7.data-security/1.authentication/4.ldap.md
# - Backup & Restore:
# - What is Backup & Restore: 7.data-security/2.backup-restore/1.what-is-br.md
# - Compile BR: 7.data-security/2.backup-restore/2.compile-br.md
# - Use BR to back up data: 7.data-security/2.backup-restore/3.br-backup-data.md
# - Use BR to restore data: 7.data-security/2.backup-restore/4.br-restore-data.md
- Backup & Restore:
- What is Backup & Restore: 7.data-security/2.backup-restore/1.what-is-br.md
- Compile BR: 7.data-security/2.backup-restore/2.compile-br.md
- Use BR to back up data: 7.data-security/2.backup-restore/3.br-backup-data.md
- Use BR to restore data: 7.data-security/2.backup-restore/4.br-restore-data.md
- Manage snapshots: 7.data-security/3.manage-snapshot.md
- SSL: 7.data-security/4.ssl.md

Expand Down Expand Up @@ -546,7 +546,6 @@ plugins:
# Exclude files with unix-style wildcards (globs)
glob:
# Exclude all files in a directory. The path starts with the directory name in docs-2.0, such as `20.appendix/*`.
- 7.data-security/2.backup-restore/*
- nebula-flink/*
- CHANGELOG.md
- spark-connector/*
Expand Down