ChubaoFS has been commonly used as the underlying storage infrastructure for online applications, database or data processing services and machine learning jobs orchestrated by Kubernetes. An advantage of doing so is to separate storage from compute - one can scale up or down based on the workload and independent of the other, providing total flexibility in matching resources to the actual storage and compute capacity required at any given time.
Some key features of ChubaoFS include:
Scale-out metadata management
Strong replication consistency
Specific performance optimizations for large/small files and sequential/random writes
POSIX-compatible and mountable
S3-compatible object storage interface
We are committed to making ChubaoFS better and more mature. Please stay tuned.
Small file operation performance and scalability benchmark test by mdtest.
|File Size (KB)||1||2||4||8||16||32||64||128|
Refer to chubaofs.readthedocs.io for performance and scalability of
Build Server and Client
$ git clone http://github.com/chubaofs/chubaofs.git $ cd chubaofs $ make
If the build succeeds,
cfs-client can be found in
Deploy ChubaoFS Cluster with Helm in Kubernetes
The chubaofs-helm repository can help you deploy ChubaoFS cluster quickly in containers orchestrated by kubernetes. Kubernetes 1.16+ and Helm 3 are required.
$ helm init
Add repository to download Helm chart of ChubaoFS:
$ helm repo add chubaofs https://chubaofs.github.io/chubaofs-charts $ helm repo update
Create the configuration file
chubaofs.yaml and put it in a user-defined path.
$ cat ~/chubaofs.yaml
path: data: /chubaofs/data log: /chubaofs/log datanode: disks: - disk: "/data0:21474836480" - disk: "/data1:21474836480" metanode: total_mem: "2147483648"
chubaofs-helm/chubaofs/values.yamlincludes all the parameters of ChubaoFS. Parameters
path.logdefine paths to store the data and logs of ChubaoFS server, respectively.
Label the nodes in Kubernetes according to different roles(master/metanode/datanode):
$ kubectl label node <nodename> chuabaofs-master=enabled $ kubectl label node <nodename> chuabaofs-metanode=enabled $ kubectl label node <nodename> chuabaofs-datanode=enabled
Install ChubaoFS cluster with Helm:
$ helm install chubaofs chubaofs/chubaofs --version 1.5.0 -f ~/chubaofs.yaml
Delete ChubaoFS cluster:
$ helm delete chubaofs
Refer to chubaofs-helm for deployment with Helm 2 and deployment of monitoring system.
Deploy ChubaoFS Cluster with Docker
A helper tool called
run_docker.sh (under the
docker directory) has been provided to run ChubaoFS with docker-compose.
$ docker/run_docker.sh -r -d /data/disk
Note that /data/disk can be any directory but please make sure it has at least 10G available space.
To check the mount status, use the
mount command in the client docker shell:
$ mount | grep chubaofs
To view grafana monitor metrics, open http://127.0.0.1:3000 in browser and login with
To run server and client separately, use the following commands:
$ docker/run_docker.sh -b $ docker/run_docker.sh -s -d /data/disk $ docker/run_docker.sh -c $ docker/run_docker.sh -m
For more usage:
$ docker/run_docker.sh -h
Haifeng Liu, et al., CFS: A Distributed File System for Large Scale Container Platforms. SIGMOD‘19, June 30-July 5, 2019, Amsterdam, Netherlands.
For more information, please refer to https://dl.acm.org/citation.cfm?doid=3299869.3314046 and https://arxiv.org/abs/1911.03001