Skip to content
a full featured file system for online data storage
Python Other
  1. Python 98.5%
  2. Other 1.5%
Branch: master
Clone or download

Latest commit

Nikratio Properly fix crash on fsync()
The first attempt (commit 3ac82f3) mis-identified the problem (el.flush() is
CacheEntry.flush - a sync function, not BlockCache.flush()).

Fixes: #159
Latest commit 595b774 Apr 6, 2020


Type Name Latest commit message Commit time
Failed to load latest commit information.
.github/ISSUE_TEMPLATE Fix typo in Issue report template (#141) Jan 12, 2020
bin Refine warning configuration Oct 7, 2018
contrib contrib/ test with more block sizes. Apr 8, 2019
developer-notes Update (#139) Dec 26, 2019
rst Fix doc formatting (#158) Apr 3, 2020
src/s3ql Properly fix crash on fsync() Apr 5, 2020
tests Properly fix crash on fsync() Apr 5, 2020
util Removed references to old homepage. Aug 1, 2015
.ackrc Added .ackrc Sep 21, 2018
.dir-locals.el Whitespace cleanup Feb 3, 2015
.gitignore Update gitignore Oct 11, 2018
.hgignore Updated .hgignore Sep 20, 2018
.hgtags Added tag release-2.32 for changeset 50656759b71b Nov 6, 2018
.travis.yml Update travis setup to install libfuse3. Nov 24, 2019
AUTHORS Released 3.4.0 Mar 19, 2020
Changes.txt Turn flush_local into async function. Apr 3, 2020
LICENSE Added LICENSE (GPLv3). Jun 18, 2011 Don't include Mar 10, 2016
README.HG Document extra steps required for installing development version. Jan 22, 2013
README.rst README changes (#143) Jan 13, 2020 Don't check unmodified files for whitespace and pyflakes issues. Aug 16, 2014
setup.cfg Add test target to and bump minimum pytest version Mar 24, 2019 Migrate from trio < 0.12 to trio 0.12. Mar 19, 2020



S3QL is a file system that stores all its data online using storage services like Google Storage, Amazon S3, or OpenStack. S3QL effectively provides a virtual drive of dynamic, infinite capacity that can be accessed from any computer with internet access.

S3QL is a standard conforming, full featured UNIX file system that is conceptually indistinguishable from any local file system. Furthermore, S3QL has additional features like compression, encryption, data de-duplication, immutable trees and snapshotting which make it especially suitable for online backup and archival.

S3QL is designed to favor simplicity and elegance over performance and feature-creep. Care has been taken to make the source code as readable and serviceable as possible. Solid error detection and error handling have been included from the very first line, and S3QL comes with extensive automated test cases for all its components.


  • Transparency. Conceptually, S3QL is indistinguishable from a local file system. For example, it supports hardlinks, symlinks, standard unix permissions, extended attributes and file sizes up to 2 TB.

  • Dynamic Size. The size of an S3QL file system grows and shrinks dynamically as required.

  • Compression. Before storage, all data may be compressed with the LZMA, bzip2 or deflate (gzip) algorithm.

  • Encryption. After compression (but before upload), all data can be AES encrypted with a 256 bit key. An additional SHA256 HMAC checksum is used to protect the data against manipulation.

  • Data De-duplication. If several files have identical contents, the redundant data will be stored only once. This works across all files stored in the file system, and also if only some parts of the files are identical while other parts differ.

  • Immutable Trees. Directory trees can be made immutable, so that their contents can no longer be changed in any way whatsoever. This can be used to ensure that backups can not be modified after they have been made.

  • Copy-on-Write/Snapshotting. S3QL can replicate entire directory trees without using any additional storage space. Only if one of the copies is modified, the part of the data that has been modified will take up additional storage space. This can be used to create intelligent snapshots that preserve the state of a directory at different points in time using a minimum amount of space.

  • High Performance independent of network latency. All operations that do not write or read file contents (like creating directories or moving, renaming, and changing permissions of files and directories) are very fast because they are carried out without any network transactions.

    S3QL achieves this by saving the entire file and directory structure in a database. This database is locally cached and the remote copy updated asynchronously.

  • Support for low bandwidth connections. S3QL splits file contents into smaller blocks and caches blocks locally. This minimizes both the number of network transactions required for reading and writing data, and the amount of data that has to be transferred when only parts of a file are read or written.

Development Status

S3QL is considered stable and suitable for production use. Starting with version 2.17.1, S3QL uses semantic versioning. This means that backwards-incompatible versions (e.g., versions that require an upgrade of the file system revision) will be reflected in an increase of the major version number.

Supported Platforms

S3QL is developed and tested under Linux. Users have also reported running S3QL successfully on OS-X, FreeBSD and NetBSD. We try to maintain compatibility with these systems, but (due to lack of pre-release testers) we cannot guarantee that every release will run on all non-Linux systems. Please report any bugs you find, and we will try to fix them.

Typical Usage

Before a file system can be mounted, the backend which will hold the data has to be initialized. This is done with the mkfs.s3ql command. Here we are using the Amazon S3 backend, and nikratio-s3ql-bucket is the S3 bucket in which the file system will be stored.

mkfs.s3ql s3://ap-south-1/nikratio-s3ql-bucket

To mount the S3QL file system stored in the S3 bucket nikratio_s3ql_bucket in the directory /mnt/s3ql, enter:

mount.s3ql s3://ap-south-1/nikratio-s3ql-bucket /mnt/s3ql

Now you can instruct your favorite backup program to run a backup into the directory /mnt/s3ql and the data will be stored on Amazon S3. When you are done, the file system has to be unmounted with

umount.s3ql /mnt/s3ql

Need Help?

The following resources are available:

Please report any bugs you may encounter in the GitHub Issue Tracker.


The S3QL source code is available on GitHub.

Professional Support

Professional support is offered via Rath Consulting.

You can’t perform that action at this time.