Erlang Shell RPC Batchfile Makefile Roff Other
Permalink
Failed to load latest commit information.
.circleci Fix config.yml to decrease the concurrency on release Aug 10, 2018
Dockerfiles leofs: Add/Modify files related to circleci Aug 9, 2018
apps [storage] Fixed for #1093 Aug 1, 2018
docs doc: Add the explanation about rack-awareness replication Jul 13, 2018
pkg Fix LeoFS' urls Mar 9, 2017
priv [leofs] Add rack-awareness test for IT Jun 6, 2018
rel Fix launch.sh to increase the timeout for CI env Aug 10, 2018
releasenotes [leofs] Modified for v1.4.2 Jul 19, 2018
test leofs: fix test_it_del_bucket.sh to make it work for CI Jul 12, 2018
website @ 6aaf3a3 add submodule: website Aug 17, 2017
.gitignore Conditional build with sd_notify library Dec 21, 2017
.gitmodules add submodule: website Aug 17, 2017
.travis.yml leofs: Fix travis.yml and replace_otp_vsn.sh Oct 17, 2017
CHANGELOG.md [leofs] Modified for v1.4.2 Jul 19, 2018
CODE_OF_CONDUCT.md leofs: Import CoC Jun 26, 2017
LICENSE Initial import Jul 3, 2012
Makefile [leofs] Add mkdocs-related commands to Makefile Apr 6, 2018
README.md [leofs] Fix the version number of the badge Jul 20, 2018
THANKS [leofs] Add one contributor Apr 24, 2018
bootstrap.sh [leofs] Add rack-awareness test for IT Jun 6, 2018
build_mdcr_env.sh Introduce mdcr.sh -- a script to control mdcr test environment. Oct 19, 2014
git_checkout.sh Add `--pull` and `--help` option to `git_checkout.sh` Mar 20, 2015
git_switch_head.sh bug fix Oct 24, 2012
leofs-adm leofs: Fix for #618 May 11, 2018
leofs.mk [leofs] Modified for v1.4.2 Jul 19, 2018
make_libs_docs.sh Import script of making docs Oct 3, 2014
make_reltool.sh Make "make_reltool.sh" more portable Dec 26, 2017
mdcr.sh leofs: To be able to build LeoFS by a current user May 23, 2017
mkdocs.yml docs: Fix for #514 Jun 22, 2018
mv_avs_dir.sh leofs: Fix leo-project/leofs#660 Apr 5, 2017
rebar Switch the package manager from rebar3 to rebar Jan 10, 2017
rebar.config [leofs] Modified for v1.4.0 Mar 27, 2018
rebar.config.project-fifo leofs: Fix rebar.config for Erlang/OTP 17.5 Jan 17, 2017
replace_otp_vsn.sh leofs: Fix travis.yml and replace_otp_vsn.sh Oct 17, 2017

README.md

LeoFS - A Storage System for a Data Lake and the Web

Join the chat at https://gitter.im/leo-project/leofs Release Build Status

LeoFS Logo

Overview

LeoFS is an Enterprise Open Source Storage, and it is a highly available, distributed, eventually consistent object/blob store. If you are seeking a storage system that can store huge amount and various kind of files such as photo, movie, log data and so on, LeoFS is suitable for that.

LeoFS is supporting the following features:

  • Multi Protocol
    • S3-API Support
      • LeoFS is an Amazon S3 compatible storage system.
      • Switch to LeoFS to decrease your cost from more expensive public-cloud solutions.
    • REST-API Support
    • NFS Support
      • NFS support was provided from LeoFS v1.1, the current status of which is beta.
  • Large Object Support
    • LeoFS covers handling large size objects.
  • Multi Data Center Replication
    • LeoFS is a highly scalable, fault-tolerant distributed file system without SPOF.
    • LeoFS's cluster can be viewed as a huge capacity storage. It consists of a set of loosely connected nodes.
    • We can build a global scale storage system with easy operations

Architecture

leofs-architecture-1

LeoFS consists of three core components - LeoStorage, LeoGateway and LeoManager which depend on Erlang.

LeoGateway handles http-request and http-response from any clients when using REST-API OR S3-API. Also, it is already built in the object-cache mechanism (memory and disk cache).

LeoStorage handles GET, PUT and DELETE objects as well as metadata. Also, it has replicator, recoverer and queueing mechanism in order to keep running a storage node and realise eventual consistency.

LeoManager always monitors LeoGateway and LeoStorage nodes. The main monitoring status are Node status and RING’s checksum in order to realise to keep high availability and keep data consistency.

You can access a LeoFS system using Amazon S3 clients and the SDK.

Slide

The presentation - Scaling and High Performance Storage System: LeoFS was given at Erlang User Conference 2014 in Stockholm on June 2014

GOALs

  • LeoFS has been aiming to provide high reliability, high scalability, and high cost performance ratio:
    • HIGH Reliability
      • Nine nines - Operating ratios is 99.9999999%
    • High Scalability
      • Build huge-cluster at low cost
    • HIGH Cost Performance
      • Fast - Over 10Gbps
      • A lower cost than other storage
      • Provide easy management and easy operation

Further Reference

Build LeoFS with LeoFS Packages

LeoFS packages have been already provided on the Web. You're able to easily install LeoFS on your environments.

Here is the installation manual.

Build LeoFS From Source (For Developers)

Here, we explain how to build LeoFS from source code. First, you have to install the following packages to build Erlang and LeoFS.

Build Dependencies

## [CentOS]
$ sudo yum install cmake check-devel gcc gcc-c++ make
## [Ubuntu]
$ sudo apt-get install gcc g++ cmake make check libtool
### For Docker
$ apt-get install lsb-release

Install Erlang

You can install Erlang with kerl.

$ curl -O https://raw.githubusercontent.com/kerl/kerl/master/kerl
$ chmod a+x kerl
$ mkdir -p ~/bin
$ mv kerl ~/bin/
$ echo "export PATH=$PATH:~/bin" >> ~/.bashrc
$ source ~/.bashrc
  • Install Erlang (Erlang/OTP 19.3)
$ kerl build 19.3 19.3
$ kerl list builds
19.3,19.3

$ kerl install 19.3 /path/to/19.3
$  kerl list installations
19.3 /path/to/19.3

$ source /path/to/19.3/activate
$ kerl active
The current active installation is:
/path/to/19.3

Install LeoFS

Then, clone source of LeoFS and libraries from GitHub.

$ git clone https://github.com/leo-project/leofs.git
$ cd leofs
$ git checkout -b develop remotes/origin/develop
$ ./rebar get-deps
$ ./git_checkout.sh develop

Then, build LeoFS with the following commands.

$ make && make release_for_test

Now, you can find the LeoFS package as follow.

$ ls package/
leo_gateway/  leo_manager_0/  leo_manager_1/  leo_storage/  README.md

Then, we can start and access LeoFS with the following commands. Also, you're able to easily operate LeoFS with leofs-adm script.

$ package/leo_manager_0/bin/leo_manager start
$ package/leo_manager_1/bin/leo_manager start
$ package/leo_storage/bin/leo_storage start
$ package/leo_gateway/bin/leo_gateway start
$ ./leofs-adm status
 [System Confiuration]
-----------------------------------+----------
 Item                              | Value
-----------------------------------+----------
 Basic/Consistency level
-----------------------------------+----------
                    system version | 1.3.4
                        cluster Id | leofs_1
                             DC Id | dc_1
                    Total replicas | 1
          number of successes of R | 1
          number of successes of W | 1
          number of successes of D | 1
 number of rack-awareness replicas | 0
                         ring size | 2^128
-----------------------------------+----------
 Multi DC replication settings
-----------------------------------+----------
 [mdcr] max number of joinable DCs | 2
 [mdcr] total replicas per a DC    | 1
 [mdcr] number of successes of R   | 1
 [mdcr] number of successes of W   | 1
 [mdcr] number of successes of D   | 1
-----------------------------------+----------
 Manager RING hash
-----------------------------------+----------
                 current ring-hash |
                previous ring-hash |
-----------------------------------+----------

 [State of Node(s)]
-------+--------------------------+--------------+----------------+----------------+----------------------------
 type  |           node           |    state     |  current ring  |   prev ring    |          updated at
-------+--------------------------+--------------+----------------+----------------+----------------------------
  S    | storage_0@127.0.0.1      | attached     |                |                | 2017-06-02 14:59:20 +0900
-------+--------------------------+--------------+----------------+----------------+----------------------------

$ ./leofs-adm start
OK

$ ./leofs-adm status
 [System Confiuration]
-----------------------------------+----------
 Item                              | Value
-----------------------------------+----------
 Basic/Consistency level
-----------------------------------+----------
                    system version | 1.3.4
                        cluster Id | leofs_1
                             DC Id | dc_1
                    Total replicas | 1
          number of successes of R | 1
          number of successes of W | 1
          number of successes of D | 1
 number of rack-awareness replicas | 0
                         ring size | 2^128
-----------------------------------+----------
 Multi DC replication settings
-----------------------------------+----------
 [mdcr] max number of joinable DCs | 2
 [mdcr] total replicas per a DC    | 1
 [mdcr] number of successes of R   | 1
 [mdcr] number of successes of W   | 1
 [mdcr] number of successes of D   | 1
-----------------------------------+----------
 Manager RING hash
-----------------------------------+----------
                 current ring-hash | 433fe365
                previous ring-hash | 433fe365
-----------------------------------+----------

 [State of Node(s)]
-------+--------------------------+--------------+----------------+----------------+----------------------------
 type  |           node           |    state     |  current ring  |   prev ring    |          updated at
-------+--------------------------+--------------+----------------+----------------+----------------------------
  S    | storage_0@127.0.0.1      | running      | 433fe365       | 433fe365       | 2017-06-02 15:00:10 +0900
  G    | gateway_0@127.0.0.1      | running      | 433fe365       | 433fe365       | 2017-06-02 15:00:12 +0900
-------+--------------------------+--------------+----------------+----------------+----------------------------

Build a LeoFS Cluster

You can easily build a LeoFS cluster. See here.

Configure LeoFS

About the configuration of LeoFS, See here.

Benchmarking

You can benchmark LeoFS with Basho Bench, and here is a documentation to benchmark LeoFS.

Integration Test

You can test LeoFS with leofs_test whether LeoFS has issues or not before getting installed LeoFS in your dev/staging/production environment(s).

Milestones

Version 1

  • DONE - v1.0
    • Multi Data Center Replication
    • Increase compatibility S3-APIs#5
      • Other bucket operations
  • DONE - v1.1
    • NFS v3 Support (alpha)
    • Improve Web GUI Console (Option)
  • DONE - v1.2
    • NFS v3 Support (beta)
    • Watchdog
    • Automated data-compaction
  • DONE - v1.3
    • NFS v3 Support (stable)
    • Improve compatibility S3-APIs#6
  • DONE - v1.4
    • Improvement of the core features
    • Integration with distributed computing frameworks#1
      • Hadoop integration
      • Spark integration

Version 2

  • WIP - v2.0
    • Erasure Code
    • Improve Data Security for GDPR and Enterprise Storages
    • Improve compatibility S3-APIs#7
    • NFS v3 Support (stable)
      • Improve performance of the list objects, the ls command
    • Improvement of the Multi Data Center Replication
    • Searching objects by a custom-metadata
  • v2.1
    • Hinted Hand-off
    • Improve compatibility S3-APIs#8
      • Objects Expiration into a Bucket
      • Object Versioning
    • Kubernetes Persistent Volumes Support
    • Integration with distributed computing frameworks#2
      • Hadoop integration
      • Spark integration
    • Improve Web GUI console, LeoFS Center (option)
  • v2.2
    • Data Deduplication
    • Improve compatibility S3-APIs#9

Versioning Policy

LeoFS adheres to the versioning policy from v1.3.3.

Licensing

LeoFS is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

Sponsors

LeoProject/LeoFS is sponsored by Rakuten, Inc. and supported by Rakuten Institute of Technology.