Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add doc: How to Setup Object Storage #135

Merged
merged 5 commits into from
Jan 28, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
10 changes: 1 addition & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,15 +69,7 @@ It will create a volume with default settings. If there Redis server is not runn

As JuiceFS relies on object storage to store data, you can specify a object storage using `--storage`, `--bucket`, `--access-key` and `--secret-key`. By default, it uses a local directory to serve as an object store, for all the options, please see `./juicefs format -h`.

To use MinIO as object store, it could be specified as:

```bash
$ ./juicefs format --storage minio \
--bucket http://1.2.3.4:9000/mybucket \
--access-key XXX \
--secret-key XXX \
localhost test
```
For the details about how to setup different object storage, please read [the guide](docs/how_to_setup_object_storage.md).

### Mount a volume

Expand Down
10 changes: 1 addition & 9 deletions README_CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,15 +70,7 @@ $ ./juicefs format localhost test

JuiceFS 还需要一个对象存储,可以通过参数 `--storage`、`--bucket`、`--access-key` 和 `--secret-key` 来指定。它默认会使用本地目录来模拟一个对象存储用于测试,详细的参数请看 `./juicefs format -h`。

如果使用 MinIO 来存数据,可以这么写:

```bash
$ ./juicefs format --storage minio \
--bucket http://1.2.3.4:9000/mybucket \
--access-key XXX \
--secret-key XXX \
localhost test
```
关于各种对象存储如何设置的详细介绍,请查看[这个文档](docs/how_to_setup_object_storage.md)。

### 挂载

Expand Down
52 changes: 52 additions & 0 deletions docs/how_to_setup_object_storage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# How to Setup Object Storage

This is a guide about how to setup object storage when format a volume. Different object storage may has different option value. Check the specific object storage for your need.

## Access key and secret key

For authentication, the access key and secret key are needed. You could specify them through `--access-key` and `--secret-key` options. Or you can set `ACCESS_KEY` and `SECRET_KEY` environment variables.
xiaogaozi marked this conversation as resolved.
Show resolved Hide resolved

Public cloud provider usually allow user create IAM (Identity and Access Management) role (e.g. [AWS IAM role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html)) or similar thing (e.g. [Alibaba Cloud RAM role](https://help.aliyun.com/document_detail/93689.html)), then assign the role to VM instance. If your VM instance already have permission to access object storage, then you could omit `--access-key` and `--secret-key` options.

## S3

S3 supports [two style URI](https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html): virtual hosted-style and path-style. The difference between them is:

- Virtual hosted-style: `https://<bucket>.s3.<region>.amazonaws.com`
- Path-style: `https://s3.<region>.amazonaws.com/<bucket>`

The `<region>` should be replaced with specific region code, e.g. the region code of US East (N. Virginia) is `us-east-1`. You could find all available regions at [here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-available-regions).

***Note: For AWS China user, you need add `.cn` to the host, i.e. `amazonaws.com.cn`. And check [this document](https://docs.amazonaws.cn/en_us/aws/latest/userguide/endpoints-arns.html) to know your region code.***

Currently, **JuiceFS only supports virtual hosted-style** and maybe support path-style in the future ([#134](https://github.com/juicedata/juicefs/issues/134)). So when you format a volume, the `--bucket` option should be virtual hosted-style URI. For example:

```bash
$ ./juicefs format \
--storage s3 \
--bucket https://<bucket>.s3.<region>.amazonaws.com \
... \
localhost test
```

You can also use S3 storage type to connect S3-compatible storage. But beware that you still need use virtual hosted-style URI. For example:

```bash
$ ./juicefs format \
--storage s3 \
--bucket https://<bucket>.<endpoint> \
... \
localhost test
```

## MinIO

[MinIO](https://min.io) is an open source high performance object storage. It is API compatible with Amazon S3. You need set `--storage` option to `minio`. Currently, JuiceFS only supports path-style URI when use MinIO storage. For example (`<endpoint>` may looks like `1.2.3.4:9000`):

```bash
$ ./juicefs format \
--storage minio \
--bucket http://<endpoint>/<bucket> \
... \
localhost test
```