Quantcast File System
C++ C Shell Makefile Python Java Other
Latest commit f231025 Feb 17, 2017 @mikeov mikeov committed on GitHub Merge pull request #214 from quantcast/topic/access_ctime
Added getCreationTime(Path) to KfsAccess
Failed to load latest commit information.
benchmarks/mstress Update copyright messages Nov 22, 2016
cmake/Modules Merge pull request #141 from fsareshwala/master Apr 26, 2016
conf S3 IO library: allow to specify S3 storage class with object put by e… Dec 2, 2016
contrib/plugins make the monitor plugin handle write errors properly Jun 1, 2016
examples Update copyright messages Nov 22, 2016
ext Jerasure: eliminate 32 bit compiler warnings in examples directory. Jan 11, 2016
scripts Update copyright messages Nov 22, 2016
src remove KfsAccess getCreationTime and added additional fields to KfsAc… Feb 17, 2017
travis travis: fix Centos 7 java build and tarball generation Dec 5, 2016
webui Update copyright messages Nov 22, 2016
wiki QFS-323 wiki: add links to centOS7 binaries for master branch and ver… Dec 6, 2016
.appveyor.yml appveyor: remove caching the build directory -- we can only cache up … Apr 21, 2016
.gitignore gitignore: ignore cmake files Mar 29, 2016
.ratignore rat: check specific files for license information Apr 29, 2016
.travis.yml QFS-323 CentOS7 build on travis Aug 22, 2016
CHANGELOG.md changelog: add one more change and a date for the recorded changes Nov 23, 2016
CMakeLists.txt Top level cmake: add _GNU_SOURCE definition on cygwin [only for now] … Aug 13, 2016
LICENSE.txt QFS files - initial commite Aug 31, 2012
Makefile Revert "Merge pull request #210 from autocracy/fix_appveyor" Dec 21, 2016
README.md README: make 1.2.0 release and the blog post more visible Nov 30, 2016


New announcementQFS version 1.2.0 with Amazon S3 support is out. Check out the blog post here for details.

Quantcast File System

Quantcast File System (QFS) is a high-performance, fault-tolerant, distributed file system developed to support MapReduce processing, or other applications reading and writing large files sequentially.

QFS servers have been tested on 64-bit CentOS 6 extensively and run on Linux variants. The QFS client tools work on OS X and Cygwin as well.

Platform Build Status
Mac OS X Build Status
Ubuntu 14.04 Build Status
CentOS 6 Build Status
CentOS 7 Build Status

The implementation details and features of QFS are discussed in detail in the project wiki.

Getting QFS

QFS is available through various channels.

  • BigTop: QFS packages for rpm and debian based systems are available through the BigTop project.
  • Binary Distributions: There are various binary distributions packaged as tarballs available on the Binary Distributions Page.
  • Compile from source: QFS can be compiled on Linux variants, Mac OS X, and Cygwin. See the wiki for more information on how to build QFS yourself.

Trying QFS

Once you have aquired QFS through one of the methods above, you can take QFS for a quick test drive. Setting up a single node configuration to familiarize yourself with QFS is very easy.

  1. Extract the distribution tarball.

    $ tar -xzf qfs.tgz && cd qfs
  2. Set up a single node QFS instance. This will create a workspace in ~/qfsbase, start two chunk servers and one metaserver.

    $ ./examples/sampleservers/sample_setup.py -a install
    Binaries presence checking - OK.
    Setup directories - OK.
    Setup config files - OK.
    Started servers - OK.
  3. Add tools binary path toPATH

    $ PATH=${PWD}/bin/tools:${PATH}
  4. Make a temporary directory on the file system

    $ qfsshell -s localhost -p 20000 -q -- mkdir /qfs/tmp
  5. Create a file containing "Hello World", Reed-Solomon encoded, with replication 1.

    $ echo 'Hello World' | cptoqfs -s localhost -p 20000 -S -k /qfs/tmp/helloworld -d -
  6. Cat the file content.

    $ qfscat -s localhost -p 20000 /qfs/tmp/helloworld
  7. Stat the file to see encoding (RS or not), replication level, and mtime.

    $ qfsshell -s localhost -p 20000 -q -- stat /qfs/tmp/helloworld
  8. Copy the file locally to the current directory.

    $ cpfromqfs -s localhost -p 20000 -k /qfs/tmp/helloworld -d ./helloworld
  9. Remove the file from QFS.

    $ qfsshell -s localhost -p 20000 -q -- rm /qfs/tmp/helloworld
  10. Stop the servers.

    $ ./examples/sampleservers/sample_setup.py -a stop
  11. Uninstall the single node instance.

    $ ./examples/sampleservers/sample_setup.py -a uninstall

Benchmarking QFS

A performance comparison between QFS and HDFS 1.0.2 shows QFS is faster both at reading and writing 20 TB of uncompressed data on our test system, a heterogeneous cluster with 6,500 disk drives. See more information regarding this in our wiki.

Contributing to QFS

We welcome contributions to QFS in the form of enhancement requests, patches, additional tests, bug reports, new ideas, and so on. Please submit issues using our JIRA instance and refer to the QFS code contribution policy when contributing code.

Have Questions?

Join the QFS Developer mailing list or search the archives at the Google Group.

Post comments or questions to qfs-devel@googlegroups.com.


QFS is released under the Apache 2.0 license.