Skip to content
fs123 network filesystem
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
client
core123
docs
exportd
include
lib
misc
test
.gitignore
.travis.yml
GNUmakefile
LICENSE.txt
README.md

README.md

fs123

fs123 is a scalable, high-performance, network filesystem.

A simple web protocol with just 6 request types leverages the entire HTTP ecosystem (load balancers, proxies, redirects, etc.) and makes it easy to implement and debug custom file servers.

fs123 provides consistency guarantees with path-specific, configurable timeouts, which enables aggressive in-kernel, local-disk, and shared http-proxy caching of positive and negative accesses. It accelerates PATH or PYTHONPATH searches, and is WAN/cloud-friendly.

The client and server are written in C++. Each is a single user-space binary that does not require root privileges. The client uses the FUSE low-level API, and can work through network outages (or even offline) once the on-disk cache is primed. The libevent-based server easily delivers data at 40Gbps.

Build and installation

Prerequisites

We run fs123 in production on CentOS7 using a mix of CentOS-provided libraries (libfuse, libcurl, libsodium) and tools andlibraries that we have built and installed from sources (gcc6, libgflags, libevent)

We have successfully run fs123's self-tests in docker images running Ubuntu 18.04(LTS) bionic, Ubuntu 16.04(LTS) xenial, Fedora 29 and CentOS7 using the distro's provided compilers and libraries. See the scripts in misc/prereqs.* for hints about how to install fs123's prerequisites in different distros.

The prerequisites to compile and link the client and server are:

  • gcc 8.3.1 (Fedora29), 8.2.0 (bionic), 5.5.0 (xenial), devtoolset-8-gcc-8.2.1 (CentOS7) have been tested. Any compiler with full C++14 support should work.
  • gmake 3.82 has been tested.
  • libfuse 2.9.9 (on Fedora29), 2.9.7 (on bionic), 2.9.4 (on xenial), 2.9.2 (on CentOS7) have been tested
  • libsodium 1.0.17 (on Fedora29, CentOS7), 1.0.16 (on bionic), 1.0.8 (on xenial)
  • libcurl 7.61 (on Fedora29), 7.58.0 (on bionic), 7.43.0 (on xenial), 7.19.0 (on CentOS7) have been tested
  • libgflags 2.2.1 (on bionic), 2.1.2 (on xenial, Fedora29), 2.1.1 (on CentOS7) have been tested
  • libevent 2.1.8 (on Fedora29, bionic), 2.0.21 (on xenial, CentOS7) have been tested

To run the tests (invoked by make check) the following are needed. The standard distro versions should be fine:

  • valgrind
  • time
  • attr
  • curl
  • python2, python3
  • e2fsprogs

Compiling

A GNUmakefile is provided. There are (currently) no autoconf or other meta-build tools.
To build, test and install the client, server, and libraries common to both:

make
make check
make install

The install rule defaults to installing binaries in /usr/local, but the destination can be changed by setting PREFIX on the command line, e.g.,

make PREFIX=/absolute/path install

The GNUmakefile leaves targets and intermediate files in the directory from which it is run, but it also supports out-of-tree builds. To avoid clutter, consider cd-ing to a build directory before calling make, e.g.,

$ mkdir build
$ cd build
$ make -f ../GNUmakefile check

The Makefile follows standard conventions for CXX, CC, TARGET_ARCH, CPPFLAGS, CFLAGS, CXXFLAGS, LDFLAGS, and LDLIBS. If prerequisites are installed in places that are not automatically found by your toolchain, set one or more environment variables before calling make. E.g., on xenial 16.04, one can use gcc6 from the ppa:ubuntu-toolchain-r/test repository, with:

$ env CC=gcc-6 CXX=g++-6 make -f .../GNUmakefile check

Running

Server

To start the server, run

$ fs123p7exportd --log_destination %stderr --port 4321 --chroot=  --export_root=/some/where
N[0.0]+0.000 complaint_delta_timestamp start time: 1554218181.000765 2019-04-02 15:16:21+0000
N[1.0]+0.001 main thread started on 127.0.0.1:4321 at 0.000358426

By default, the export daemon runs as a normal foreground process. Job-control works in the usual ways with nohup, &, bg, wait, etc. It can be fully disconnected from the calling terminal (i.e., daemonized) with the --daemonize option. If --log-destination is not specified, logs go to the LOG_USER syslog facility.

It's best if the --export_root is on a filesystem that supports the FS_IOC_GETVERSION ioctl, e.g., ext3, ext4, xfs or most other local disk-backed filesystems, but NOT tmpfs or nfs or docker overlay mounts. If you see warnings like this:

W[4.0]+128.333 do_estale_cookie_common_(fullpath=/exported)
W[4.1] ioctl(17, -2146929151, 0x7f19f846208c): Inappropriate ioctl for device

it means that FS_IOC_GETVERSION is not supported, in which case, add --estale_cookie_src=st_ino to the command line.

By default, the export daemon will bind the 127.0.0.1 address, and therefore will only be accessible to clients on localhost. To make it accessible remotely, add --bindaddr=0.0.0.0 to the command line.

Check that the port is working:

$ curl -v http://localhost:4321/fs123/7/2/a
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 4321 (#0)
> GET /fs123/7/2/a HTTP/1.1
> Host: localhost:4321
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< fs123-estalecookie: 0
< Cache-Control: public,max-age=60,stale-while-revalidate=30
< Date: Tue, 02 Apr 2019 15:20:50 GMT
< fs123-errno: 0
< Content-Type: application/octet-stream
< fs123-trsum: b2235900f57ebbd229db1741bd0e2232
< Content-Length: 120
< 
16877 2 0 0 4096 1554218149 1554218149 1554218140 85513372 681248835 681248835 746338405 60 8 4096 0
* Connection #0 to host localhost left intact
1554218149681248835$

The reply should be a 200, with a Cache-control header, an fs123-errno: 0 header, and a body consisting of numerical values corresponding to the values in the export_root's stat.

Client

Now create a mount-point and start up a client.

$ mkdir /tmp/mtpt
$ fs123p7 mount http://localhost:4321 /tmp/mnt
$ ls /tmp/mnt
... <contents of export root> ...
$ cd /tmp/mnt
$ ...

If you started fs123p7exportd with --bindaddr=0.0.0.0, then you should be able to mount the exported filesystem on another machine, e.g.,

$ mkdir /tmp/mnt
$ fs123p7 mount http://HOSTNAME:4321 /tmp/mnt
$ ls /tmp/mnt
... <contents of export root> ...
$ cd /tmp/mnt
$ ...

Documentation

Documentation is a work-in-progress. The fs123p7 and fs123p7exportd accept --help options, which give a complete list of options with extremely cursory descriptions. Files in the docs/ directory describe certain aspects of fs123 in great detail, but there are large gaps, and occasional anachronisms.

You can’t perform that action at this time.