Skip to content
Quantcast File System
C++ C Shell Makefile Python HTML Other
Branch: master
Clone or download

Latest commit

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
benchmarks/mstress Merge branch 'master' into topic/multimaster Jan 13, 2017
cmake/Modules Cmake: use "cc" on macOS as compiler for external projects as CMAKE_C… Oct 4, 2018
conf Annotated configuration files: add chunkServer.hostname parameter des… Jan 26, 2020
contrib/plugins Monitor plugin: fix log path prepare: remove incorrect use of operato… Feb 9, 2017
examples examples: fix sample setup python script to pass the right path for c… Jul 26, 2019
ext Ext DNS: possible fix for GCC 9 build error. Feb 28, 2020
scripts Scripts: RAT update version 0.13 download URLs. Feb 28, 2020
src Common: bump version to 2.2.0 Apr 27, 2020
travis Travis: update centos 5 rpm url. Apr 27, 2020
webui Fixed some typos in configuration files Oct 1, 2017
wiki Wiki: update binary distributions page in preparation to QFS 2.2.0 re… May 1, 2020
.appveyor.yml Appveyor: update version. Jun 25, 2019
.codecov.yml Codecov: set target code coverage with default CI tests by adding cod… Oct 30, 2017
.gitignore Cleanup gitignore, leave only build and mac os specific directories. Jan 14, 2017
.ratignore Travis: create code coverage script in .tmp directory to get around r… Jan 17, 2017
.travis.yml Travis: update S3 key, and re-enable deploy to S3. Feb 9, 2019
CHANGELOG.md QFS 2.1.3 release. Oct 23, 2019
CMakeLists.txt
LICENSE.txt QFS files - initial commite Aug 31, 2012
Makefile Make file: fix hadoop java target by failing build by exiting with er… Nov 2, 2018
README.md README: update to include 2.2.0 version. May 2, 2020

README.md

QFS version 2.2.0

Release 2.2.0 constains symbolic links support, and minor bug fixes.

QFS version 2.1.3

Release 2.1.3 contains DNS resolver's number of open socket accounting bug fix. This bug might manifest itself in at least one non obvious way: chunk server might continuously fail to create or open chunk files if/when the open sockets counter becomes negative.

Release 2.1.2 contains minor bug fixes, meta and chunk servers watchdog.

Release 2.1.1 contains minor bug fixes.

Non blocking DNS resolver is the new feature in release 2.1.0. The new DNS resolver is intended to improve S3 [compatible] object store IO concurrency and reduce latency.

Meta server replication is the major new feature in release 2.0.0. Meta server replication provides automatic meta server fail over. With meta server replication configured QFS does not have single point of failure.

Release notes are available here.

Quantcast File System

Quantcast File System (QFS) is a high-performance, fault-tolerant, distributed file system developed to support MapReduce processing, or other applications reading and writing large files sequentially.

QFS used in Quantcast production cluster.

QFS servers have been tested on 64-bit CentOS 6 extensively and run on Linux variants. The QFS client tools work on OS X and Cygwin as well.

Platform Build Status
Mac OS X Build Status
Ubuntu 14.04 Build Status
Ubuntu 18.04 Build Status
CentOS 6 Build Status
CentOS 7 Build Status
Debian 9 Build Status

The implementation details and features of QFS are discussed in detail in the project wiki.

Getting QFS

QFS is available through various channels.

  • BigTop: QFS packages for rpm and debian based systems are available through the BigTop project.
  • Binary Distributions: There are various binary distributions packaged as tarballs available on the Binary Distributions Page.
  • Compile from source: QFS can be compiled on Linux variants, Mac OS X, and Cygwin. See the wiki for more information on how to build QFS yourself.

Trying QFS

Once you have aquired QFS through one of the methods above, you can take QFS for a quick test drive. Setting up a single node configuration to familiarize yourself with QFS is very easy.

  1. Extract the distribution tarball.

     $ tar -xzf qfs.tgz && cd qfs
    
  2. Set up a single node QFS instance. This will create a workspace in ~/qfsbase, start two chunk servers and one metaserver.

     $ ./examples/sampleservers/sample_setup.py -a install
     Binaries presence checking - OK.
     Setup directories - OK.
     Setup config files - OK.
     Started servers - OK.
    
  3. Add tools binary path toPATH

     $ PATH=${PWD}/bin/tools:${PATH}
    
  4. Make a temporary directory on the file system

     $ qfsshell -s localhost -p 20000 -q -- mkdir /qfs/tmp
    
  5. Create a file containing "Hello World", Reed-Solomon encoded, with replication 1.

     $ echo 'Hello World' | cptoqfs -s localhost -p 20000 -S -k /qfs/tmp/helloworld -d -
    
  6. Cat the file content.

     $ qfscat -s localhost -p 20000 /qfs/tmp/helloworld
    
  7. Stat the file to see encoding (RS or not), replication level, and mtime.

     $ qfsshell -s localhost -p 20000 -q -- stat /qfs/tmp/helloworld
    
  8. Copy the file locally to the current directory.

     $ cpfromqfs -s localhost -p 20000 -k /qfs/tmp/helloworld -d ./helloworld
    
  9. Remove the file from QFS.

     $ qfsshell -s localhost -p 20000 -q -- rm /qfs/tmp/helloworld
    
  10. Stop the servers.

     $ ./examples/sampleservers/sample_setup.py -a stop
    
  11. Uninstall the single node instance.

     $ ./examples/sampleservers/sample_setup.py -a uninstall
    

Benchmarking QFS

A performance comparison between QFS and HDFS 1.0.2 shows QFS is faster both at reading and writing 20 TB of uncompressed data on our test system, a heterogeneous cluster with 6,500 disk drives. See more information regarding this in our wiki.

Contributing to QFS

We welcome contributions to QFS in the form of enhancement requests, patches, additional tests, bug reports, new ideas, and so on. Please submit issues using our JIRA instance and refer to the QFS code contribution policy when contributing code.

Have Questions?

Join the QFS Developer mailing list or search the archives at the Google Group.

Post comments or questions to qfs-devel@googlegroups.com.

License

QFS is released under the Apache 2.0 license.

You can’t perform that action at this time.