Skip to content
FUSE-based file system backed by QingStor.
C++ Shell CMake C
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
cmake
src
test
third_party/boost_1_49_0
tools
.gitignore
.travis.yml
AUTHORS
CMakeLists.txt
COPYING
ChangeLog
INSTALL.md
README.md

README.md

qsfs

Build Status License

qsfs is a FUSE based filesystem that allows you to mount a qingstor bucket in Linux.

Features

  • Large subset of POSIX including reading/writing files, directories, symlinks, .etc, (chmod/chown/utimens will be supported later).
  • File renames via server-side move.
  • File Transfer:
    • Large files uploads via multipart parallel uploads.
    • Large files downloads via parallel byte-range downloads.
    • Large files transfer in chunks (10 MB chunks by default). If you are uploading large files (e.g. larger than 1 GB), you can increase the transfer buffer size and the max parallel transfers.
  • Data integrity via MD5 hashes.
  • Cache:
    • In-memory metadata caching.
    • In-memory file data caching. For a big file, partial file data may been stored in a local disk file when the im-memory file cache is not available.
  • Logging/Debugging:
    • Support to log messages to console or to a directory.
    • Support turn on debug message to log by specifying debug option -d, this option will also enable FUSE debug mode.
    • You can turn on debug message from curl by specifying option -U.

Installation

See the INSTALL for installation instructions.

Usage

Enter your QingCloud API access keys in a file /path/to/cred, go to QingCloud Console to create them if you do not have one:

 $ echo YourAccessKeyId:YourSecretKey > /path/to/cred

Make sure the file /path/to/cred has proper permissions (if you get 'permissions' error when mounting):

 $ chmod 600 /path/to/cred

Run qsfs with an existing bucket mybucket and directory /path/to/mountpoint:

 $ [sudo] qsfs mybucket /path/to/mountpoint -c=/path/to/cred

If you encounter any errors, enable debug output:

 $ [sudo] qsfs mybucket /path/to/mountpoint -c=/path/to/cred -d

You can also mount on boot by entering the following line to /etc/fstab:

qsfs#mybucket /path/to/mountpoint fuse _netdev,-z=pek3a,-c=/path/to/cred,allow_other 0 0

Notice: in /etc/fstab, you need to use short options (for example, use short option -c instead of long option --credentials) in order to pass them to qsfs.

You can log messages to console:

 $ [sudo] qsfs mybucket /path/to/mountpoint -c=/path/to/cred -f

Or you can log messages to log file by specifying a directory /path/to/logdir/:

 $ [sudo] qsfs mybucket /path/to/mountpoint -c=/path/to/cred -l=/path/to/logdir/

Specify log level (INFO,WARN,ERROR and FATAL):

 $ [sudo] qsfs mybucket /path/to/mountpoint -c=/path/to/cred -L=INFO -d

To umount:

 $ [sudo] fusermount -uqz /path/to/mountpoint

or

 $ [sudo] umount -l /path/to/mountpoint

For help:

 $ qsfs -h

Command line options

Supported general options are listed as following,

short full type required usage
-z --zone string Y Specify zone or region
-c --credentials string N Specify credentials file, default path is /etc/qsfs.cred
-l --logdir string N Specify log directory, default path is /var/log/qsfs/
-L --loglevel string N Specify min log level (INFO,WARN,ERROR or FATAL), message lower than this level don't logged; default value is INFO.
-F --filemode octal N Specify the permission bits in st_mode for file objects without x-qs-meta-mode header. The value is given in octal representation; default value is 0644
-D --dirmode octal N Specify the permission bits in st_mode for directory objects without x-qs-meta-mode header. The value is given in octal representation; default value is 0755
-u --umaskmp octal N Specify the permission bits in st_mode for the mount point directory. This option only works when you set with the fuse allow_other option. The resulting permission bits are the ones missing from the given umask value. The value is given in octal representation; default value is 0000
-r --retries integer N Specify number of times to retry a failed transaction, default value is 3 times
-R --reqtimeout integer N Specify time (seconds) to wait before giving up connection, default value is 300 seconds
-Z --maxcache integer N Specify max in-memory cache size (MB) for files, default value is 200 MB
-k --diskdir string N Specify the directory to store file data when in-memory cache is not availabe, default path is /tmp/qsfs_cache/
-t --maxstat integer N Specify max count (K) of cached stat entrys, default value is 2 K
-e --statexpire integer N Specify expire time (minutes) for stat entries, negative value will disable stat expire, default is no expire
-i --maxlist integer N Specify max count of files of ls operation. A value of zero will list all files, default is to list all files
-y --fscap integer N Specify filesystem capacity (GB), default value is 1PB
-n --numtransfer integer N Specify max number file tranfers to run in parallel, you can increase the value when transfer large files, default value is 5
-b --bufsize integer N Specify file transfer buffer size (MB), this should be larger than 8MB, default value is 10 MB
-j --prefetchsize integer N Specify file read max prefetch size (MB), default value is 20 MB
-H --host string N Specify host name, default value is qingstor.com
-p --protocol string N Specify protocol (https or http) default value is https
-P --port integer N Specify port, default is 443 for https and 80 for http

Supported miscellaneous options are list as following,

short full type required usage
-J --prefetch bool N Enable read prefetch
-m --contentMD5 bool N Enable writes with MD5 hashs to ensure data integrity
-K --keeplogdir bool N Do not clear log directory at beginning
-C --nodatacache bool N Clear the file data cache
-f --forground bool N Turn on log to STDERR and enable FUSE foreground mode
-s --single bool N Turn on FUSE single threaded option - disable multi-threaded
-d --debug bool N Turn on debug messages to log
-g --fusedbg bool N Enable FUSE debug option
-U --curldbg bool N Turn on debug message from curl
-h --help bool N Print qsfs help
-V --version bool N Print qsfs version

You can also specify FUSE specific mount options with -o opt [,opt...] e.g. nonempty, allow_other, etc. See the FUSE's README for the full set.

Limitations

Generally qingstor cannot offer the same performance or semantics as a local file system. More specifically:

  • Random writes or appends to files require rewriting the entire file
  • Metadata operations such as listing directories have poor performance due to network latency
  • Eventual consistency can temporarily yield stale data
  • No atomic renames of files or directories
  • No coordination between multiple clients mounting the same bucket
  • No hard links

Frequently Asked Questions

Support

If you notice any issue, please open an issue on GitHub. Please search the existing issues and see if others are also experiencing the issue before opening a new issue. Please include the version of qsfs, Compiler version, CMake version, and OS you’re using.

License

See the LICENSE for details. In summary, qsfs is licensed under the Apache License (Version 2.0, January 2004).

You can’t perform that action at this time.