C++ M4 Shell Makefile Java Vim script
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.github
doc
lib/java
m4
src
tests
.clang-format
.gitignore
.gitmodules
.travis.yml
.vimrc
LICENSE
Makefile.am
README.md
autogen.sh
configure.ac
java_compile.sh
logo.png
set-env.sh

README.md

VeloxDFS {#mainpage}


Build Status Slack room

VeloxDFS is a decentralized distributed file system based on ChordDHT and HDFS.

This distributed file system serves as the foundation and essential component of the Velox Big Data Framework (VBDF), however, it can be used independently of the other components such as VeloxMR (MapReduce Framework).

Key features of current VeloxDFS implementation includes:

  • No central directory service such as in HDFS NameNode.
  • Zero Hop lookup file-system.
  • Instead for each file there is a file leader node which holds its metadata.
  • Highly secure and stable, total replication of the system's state.

USAGE

VeloxDFS default [launcher] eclipsed is not included in this repository since its a temporary hack and we want to let the user to choose any launcher or service managers such as:

  • systemd/init.d
  • puppet/chef/salt

Once the system is up and running, you can interact with VeloxDFS with the following commands:

$ dfs put|get|cat|ls|rm|format|show

COMPILING & INSTALLING

Further information can be found it in: Installation

Compiling requirements

  • C++14 support, this is: GCC >= 4.9, Clang >= 3.4, or ICC >= 16.0.
  • Boost library >= 1.53.
  • Sqlite3 library.
  • GNU Autotools (Autoconf, Automake, Libtool).
  • Unittest++ [optional].

For single user installation for developers

$ mkdir -p local_eclipse/{tmp,sandbox}                 # Create a sandbox directories
$ cd local_eclipse                                     # enter in the directory
$ git clone git@github.com:DICL/VeloxDFS.git           # Clone the project from github
$ cd VeloxDFS
$ sh autogen.sh                                        # Generate configure script 
$ cd ../tmp                                            # Go to building folder
$ sh ../VeloxDFS/configure --prefix=`pwd`/../sandbox # Check requirements and generate the Makefile

# If you get a boost error go the FAQ section of the README

### This last command will be needed whenever you want to recompile the source
$ make [-j#] install                                   # Compile & install add -j flag to speed up

Now edit in your ~/.bashrc or ~/.profile:

export PATH="/home/*..PATH/To/eclipse/..*/sandbox/bin":$PATH
export LIBRARY_PATH="/home/*..PATH/To/eclipse/..*/sandbox/lib"
export C_INCLUDE_PATH="/home/*..PATH/To/eclipse/..*/sandbox/include"

Default settings for VELOXDFS

"log" : {
  "type" : "LOG_LOCAL6"
  "name" : "ECLIPSE"
  "mask" : "DEBUG"
},

"cache" : {
  "numbin"      : 100,
  "size"        : 200000,
  "concurrency" : 1
},

"filesystem" : {
  "block"    : 137438953,
  "buffer"   : 512,
  "replica"  : 1
}

Further information can be found it in: Conf reference

FAQ

  • Question : configure stops with errors related to boost library.

  • Answer : It probably means that you do not have boost library installed in the default location, in such case you should specify the boost library location.

      sh ../VeloxDFS/configure --prefix ~/sandbox --with-boost=/usr/local --with-boost-libdir=/usr/local/lib
    

    In this example we assume that the boost headers are in /usr/local/include while the library files are inside /usr/local/lib.

AUTHORS