LeoFS is an unstructured object/data storage for the Web and a highly available, distributed, eventually consistent storage system.
Erlang Shell Logos Batchfile Roff Makefile Java
Permalink
Failed to load latest commit information.
apps manager: Fix leo_object_storage's version Feb 10, 2017
pkg Add manager mode to the exampe config files Apr 6, 2016
priv Fix leofs-mon's templates for Grafana's schema-12 Jun 24, 2016
rel Fixed for leo-project/leofs#568 Jan 10, 2017
releasenotes leofs: Fix changelog of v1.3.2 Jan 30, 2017
test Fix test/src/basho_bench_driver_leofs.erl to add the Date header Jul 9, 2015
.gitignore leofs: Change .gitignore Jan 19, 2017
.travis.yml leofs: Add 17.5 and 18.3 of otp_release Jan 19, 2017
CHANGELOG.md leofs: Fix changelog of v1.3.2 Jan 30, 2017
LICENSE Initial import Jul 3, 2012
Makefile leofs: Add a test target, and add .travis.yml for testing it on TravisCI Jan 19, 2017
README.md leofs: Fix the version number of the badge Jan 30, 2017
THANKS Add a contributor Jan 21, 2017
bootstrap.sh Add http-cache-test config Jun 23, 2016
build_mdcr_env.sh Introduce mdcr.sh -- a script to control mdcr test environment. Oct 19, 2014
git_checkout.sh Add `--pull` and `--help` option to `git_checkout.sh` Mar 20, 2015
git_switch_head.sh bug fix Oct 24, 2012
leofs-adm Modified for v1.3.0 Aug 8, 2016
leofs.mk leofs: Modified for v1.3.2 Jan 30, 2017
make_libs_docs.sh Import script of making docs Oct 3, 2014
mdcr.sh To move test-related files to priv/test/ Oct 27, 2015
rebar Switch the package manager from rebar3 to rebar Jan 10, 2017
rebar.config leofs: Fix rebar.config for Erlang/OTP 17.5 Jan 17, 2017
rebar.config.project-fifo leofs: Fix rebar.config for Erlang/OTP 17.5 Jan 17, 2017
replace_otp_vsn.sh Bump the apps for the v1.3.1's test w/Erlang/OTP 19 Sep 29, 2016

README.md

LeoFS - A Storage System for a Data Lake and the Web

Join the chat at https://gitter.im/leo-project/leofs Release Build Status

LeoFS Logo

Overview

LeoFS is a highly available, distributed, eventually consistent object/blob store. If you are searching a storage system that is able to store huge amount and various kind of files such as photo, movie, log data and so on, LeoFS is suitable for that.

LeoFS is supporting the following features:

  • Multi Protocol
    • S3-API Support
      • LeoFS is an Amazon S3 compatible storage system.
      • Switch to LeoFS to decrease your cost from more expensive public-cloud solutions.
    • REST-API Support
    • NFS Support
      • NFS support was provided from LeoFS v1.1, the current status of which is beta.
  • Large Object Support
    • LeoFS covers handling large size objects.
  • Multi Data Center Replication
    • LeoFS is a highly scalable, fault-tolerant distributed file system without SPOF.
    • LeoFS's cluster can be viewed as a huge capacity storage. It consists of a set of loosely connected nodes.
    • We can build a global scale storage system with easy operations

Architecture

leofs-architecture-1

LeoFS consists of 3 applications - LeoFS Storage, LeoFS Gateway and LeoFS Manager which depend on Erlang.

LeoFS Gateway handles http-request and http-response from any clients when using REST-API OR S3-API. Also, it is already built in the object-cache mechanism (memory and disk cache).

LeoFS Storage handles GET, PUT and DELETE objects as well as metadata. Also, it has replicator, recoverer and queueing mechanism in order to keep running a storage node and realise eventual consistency.

LeoFS Manager always monitors LeoFS Gateway and LeoFS Storage nodes. The main monitoring status are Node status and RING’s checksum in order to realise to keep high availability and keep data consistency.

We can access LeoFS server using S3 clients and S3 client libries of each programming language.

Slide

The presentation - Scaling and High Performance Storage System: LeoFS was given at Erlang User Conference 2014 in Stockholm on June 2014

GOALs

  • LeoFS aims to provide all of 3-HIGHs as follow:
    • HIGH Reliability
      • Nine nines - Operating ratios is 99.9999999%
    • High Scalability
      • Build huge-cluster at low cost
    • HIGH Cost Performance
      • Fast - Over 10Gbps
      • A lower cost than other storage
      • Provide easy management and easy operation

Further Reference

Build LeoFS with LeoFS Packages

LeoFS packages have been already provided on the Web. You're able to easily install LeoFS on your environments.

Here is the installation manual.

Build LeoFS From Source (For Developers)

Here, we explain how to build LeoFS from source code.

First, you have to install the following packages to build Erlang and LeoFS.

## [CentOS]
$ sudo yum install libuuid-devel cmake check check-devel
## [Ubuntu]
$ sudo apt-get install build-essential libtool libncurses5-dev libssl-dev cmake check

Then, install Erlang.

##
## 1. Install libatomic
##
$ wget http://www.ivmaisoft.com/_bin/atomic_ops/libatomic_ops-7.4.4.tar.gz
$ tar xzvf libatomic_ops-7.4.4.tar.gz
$ cd libatomic_ops-7.4.4
$ ./configure --prefix=/usr/local
$ make
$ sudo make install

##
## 2. Install Erlang (18.3)
##
$ wget http://erlang.org/download/otp_src_18.3.tar.gz
$ tar xzf otp_src_18.3.tar.gz
$ cd otp_src_18.3
$ ./configure --prefix=/usr/local/erlang/18.3 \
              --enable-smp-support \
              --enable-m64-build \
              --enable-halfword-emulator \
              --enable-kernel-poll \
              --without-javac \
              --disable-native-libs \
              --disable-hipe \
              --disable-sctp \
              --enable-threads \
              --with-libatomic_ops=/usr/local
$ make
$ sudo make install

##
## 3. Set PATH
##
$ vi ~/.profile
    ## append the follows:
    export ERL_HOME=/usr/local/erlang/18.3
    export PATH=$PATH:$ERL_HOME/bin

$ source ~/.profile

Then, clone source of LeoFS and libraries from GitHub.

$ git clone https://github.com/leo-project/leofs.git
$ cd leofs
$ git checkout -b develop remotes/origin/develop
$ ./rebar get-deps
$ ./git_checkout.sh develop

Then, build LeoFS with the following commands.

$ make && make release

Now, you can find the LeoFS package as follow.

$ ls package/
leo_gateway/  leo_manager_0/  leo_manager_1/  leo_storage/  README.md

Then, we can start and access LeoFS with the following commands. Also, you're able to easily operate LeoFS with leofs-adm script.

$ package/leo_manager_0/bin/leo_manager start
$ package/leo_manager_1/bin/leo_manager start
$ package/leo_storage/bin/leo_storage start
$ package/leo_gateway/bin/leo_gateway start
$ ./leofs-adm status
 [System Confiuration]
-----------------------------------+----------
 Item                              | Value
-----------------------------------+----------
 Basic/Consistency level
-----------------------------------+----------
                    system version | 1.2.16
                        cluster Id | leofs_1
                             DC Id | dc_1
                    Total replicas | 2
          number of successes of R | 1
          number of successes of W | 1
          number of successes of D | 1
 number of rack-awareness replicas | 0
                         ring size | 2^128
-----------------------------------+----------
 Multi DC replication settings
-----------------------------------+----------
        max number of joinable DCs | 2
           number of replicas a DC | 1
-----------------------------------+----------
 Manager RING hash
-----------------------------------+----------
                 current ring-hash | 3923d007
                previous ring-hash | 3923d007
-----------------------------------+----------

 [State of Node(s)]
-------+--------------------------+--------------+----------------+----------------+----------------------------
 type  |           node           |    state     |  current ring  |   prev ring    |          updated at
-------+--------------------------+--------------+----------------+----------------+----------------------------
  S    | storage_0@127.0.0.1      | running      | 3923d007       | 3923d007       | 2015-10-30 09:44:11 +0900
-------+--------------------------+--------------+----------------+----------------+----------------------------

$ ./leofs-adm start
OK

$ ./leofs-adm status
-----------------------------------+----------
 Item                              | Value
-----------------------------------+----------
 Basic/Consistency level
-----------------------------------+----------
                    system version | 1.2.16
                        cluster Id | leofs_1
                             DC Id | dc_1
                    Total replicas | 2
          number of successes of R | 1
          number of successes of W | 1
          number of successes of D | 1
 number of rack-awareness replicas | 0
                         ring size | 2^128
-----------------------------------+----------
 Multi DC replication settings
-----------------------------------+----------
        max number of joinable DCs | 2
           number of replicas a DC | 1
-----------------------------------+----------
 Manager RING hash
-----------------------------------+----------
                 current ring-hash | 3923d007
                previous ring-hash | 3923d007
-----------------------------------+----------

 [State of Node(s)]
-------+--------------------------+--------------+----------------+----------------+----------------------------
 type  |           node           |    state     |  current ring  |   prev ring    |          updated at
-------+--------------------------+--------------+----------------+----------------+----------------------------
  S    | storage_0@127.0.0.1      | running      | 3923d007       | 3923d007       | 2015-10-30 09:44:11 +0900
  G    | gateway_0@127.0.0.1      | running      | 3923d007       | 3923d007       | 2015-10-30 09:45:27 +0900
-------+--------------------------+--------------+----------------+----------------+----------------------------

Build a LeoFS Cluster

You can easily build a LeoFS cluster.

Please refer here.

Configure LeoFS

About the configuration of LeoFS, please refer here.

Benchmarking

You can benchmark LeoFS with Basho Bench.

Here is a documentation to benchmark LeoFS.

Integration Test

We're able to easily check LeoFS with leofs_test whether LeoFS has issues or not before getting installed LeoFS in your dev/staging/production environment(s).

Milestones

Version 1

  • DONE - v1.0
    • Multi Data Center Replication
    • Increase compatibility S3-APIs#5
      • Other bucket operations
  • DONE - v1.1
    • NFS v3 Support (alpha)
    • Improve Web GUI Console (Option)
  • DONE - v1.2
    • NFS v3 Support (beta)
    • Watchdog
    • Automated data-compaction
  • WIP - v1.3
    • NFS v3 Support (stable)
    • Improve compatibility S3-APIs#6
  • v1.4
    • Hinted Hand-off
    • Integration with distributed computing frameworks
      • Hadoop integration
      • Spark integration
    • Improve compatibility S3-APIs#7
  • v1.5
    • Erasure Code
    • LeoManager's cluster replaces from mnesia dependensy to another implementation

Version 2

  • v2.0
    • NFS v3 Support (stable)
      • Improve performance of the list objects, the ls command
    • Improvement of the Multi Data Center Replication
    • Searching objects by a custom-metadata
    • Improve compatibility S3-APIs#8
      • Objects Expiration into a Bucket
      • Object Versioning
    • Improve Web GUI console, LeoFS Center (option)
  • v2.2
    • NFS v4 Support
    • Data Deduplication
    • Improve compatibility S3-APIs#8

Sponsors

LeoProject/LeoFS is sponsored by Rakuten, Inc. and supported by Rakuten Institute of Technology.