QFS version 2.0.
The code in the master branch is currently being tested in Quantcast production cluster.
Meta server replication is the major new feature in this release. Meta server replication provides automatic meta server fail over. With meta server replication configured QFS does not have single point of failure.
Release notes for this release are available here.
Quantcast File System
Quantcast File System (QFS) is a high-performance, fault-tolerant, distributed file system developed to support MapReduce processing, or other applications reading and writing large files sequentially.
QFS servers have been tested on 64-bit CentOS 6 extensively and run on Linux variants. The QFS client tools work on OS X and Cygwin as well.
|Mac OS X|
The implementation details and features of QFS are discussed in detail in the project wiki.
QFS is available through various channels.
- BigTop: QFS packages for rpm and debian based systems are available through the BigTop project.
- Binary Distributions: There are various binary distributions packaged as tarballs available on the Binary Distributions Page.
- Compile from source: QFS can be compiled on Linux variants, Mac OS X, and Cygwin. See the wiki for more information on how to build QFS yourself.
Once you have aquired QFS through one of the methods above, you can take QFS for a quick test drive. Setting up a single node configuration to familiarize yourself with QFS is very easy.
Extract the distribution tarball.
$ tar -xzf qfs.tgz && cd qfs
Set up a single node QFS instance. This will create a workspace in
~/qfsbase, start two chunk servers and one metaserver.
$ ./examples/sampleservers/sample_setup.py -a install Binaries presence checking - OK. Setup directories - OK. Setup config files - OK. Started servers - OK.
Add tools binary path to
Make a temporary directory on the file system
$ qfsshell -s localhost -p 20000 -q -- mkdir /qfs/tmp
Create a file containing "Hello World", Reed-Solomon encoded, with replication 1.
$ echo 'Hello World' | cptoqfs -s localhost -p 20000 -S -k /qfs/tmp/helloworld -d -
Cat the file content.
$ qfscat -s localhost -p 20000 /qfs/tmp/helloworld
Stat the file to see encoding (RS or not), replication level, and mtime.
$ qfsshell -s localhost -p 20000 -q -- stat /qfs/tmp/helloworld
Copy the file locally to the current directory.
$ cpfromqfs -s localhost -p 20000 -k /qfs/tmp/helloworld -d ./helloworld
Remove the file from QFS.
$ qfsshell -s localhost -p 20000 -q -- rm /qfs/tmp/helloworld
Stop the servers.
$ ./examples/sampleservers/sample_setup.py -a stop
Uninstall the single node instance.
$ ./examples/sampleservers/sample_setup.py -a uninstall
A performance comparison between QFS and HDFS 1.0.2 shows QFS is faster both at reading and writing 20 TB of uncompressed data on our test system, a heterogeneous cluster with 6,500 disk drives. See more information regarding this in our wiki.
Contributing to QFS
We welcome contributions to QFS in the form of enhancement requests, patches, additional tests, bug reports, new ideas, and so on. Please submit issues using our JIRA instance and refer to the QFS code contribution policy when contributing code.
Join the QFS Developer mailing list or search the archives at the Google Group.
Post comments or questions to email@example.com.
QFS is released under the Apache 2.0 license.