Skip to content

Commit

Permalink
rework policy code
Browse files Browse the repository at this point in the history
  • Loading branch information
trapexit committed May 28, 2014
1 parent 345d0bb commit aab90b0
Show file tree
Hide file tree
Showing 35 changed files with 982 additions and 990 deletions.
72 changes: 33 additions & 39 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,36 +10,30 @@ Why create mergerfs when those exist? mhddfs isn't really maintained or flexible
Policies
========

Filesystem calls are broken up into 4 categories of policies: search, action, create, and none.

Below shows each policy class, the FUSE calls they impact, and the policy names.

#### Policy classifications ####
| Class | FUSE calls | Policies |
|-------|------------|----------|
| search | access, getattr, getxattr, listxattr, open, readlink | First Found (ff), First Found w/ Permission (ffwp), Newest (newest) |
| action | chmod, link, removexattr, rmdir, setxattr, truncate, unlink, utimens | First Found (ff), First Found w/ Permission (ffwp), Newest (newest), All Found (all) |
| create | create, mkdir, mknod | Existing Path (ep), Most Free Space (mfs), Existing Path Most Free Space (epmfs), Random (rand) |
| none | fallocate, fsync, ftruncate, ioctl, read, readdir, rename, statfs, symlink, write, release | |

#### Descriptions ####
| Class/Policy | Description |
Filesystem calls are broken up into 4 functional categories: search, action, create, and none. These categories can be assigned a policy which dictates how [mergerfs](http://github.com/trapexit/mergerfs) behaves while when action on the filesystem. Any policy can be assigned to a category though some aren't terribly practical. For instance: rand (Random) may be useful for **create** but could lead to very odd behavior if used for **search** or **action**. Since the input for any policy is the source mounts and fusepath and the output a vector of targets the choice was made to simplify the implementation and allow a policies usage in any category. **NOTE:** In any policy which can return more than one location (currently only **all**) the first value will be used in **search** and **create** policies since they can only ever act on 1 filepath.

#### Functional classifications ####
| Class | FUSE calls |
|-------|------------|
| search | access, getattr, getxattr, listxattr, open, readlink |
| action | chmod, link, removexattr, rmdir, setxattr, truncate, unlink, utimens |
| create | create, mkdir, mknod |
| none | fallocate, fsync, ftruncate, ioctl, read, readdir, rename, statfs, symlink, write, release |

#### Policy descriptions ####
| Policy | Description |
|--------------|-------------|
| search/ff | Given the order the paths were provided at mount time act on the first one found (regardless if stat would return EACCES). |
| search/ffwp | Given the order the paths were provided at mount time act on the first one found which you have access (stat does not error with EACCES). |
| search/newest | If multiple files exist return the one with the most recent mtime. |
| action/ff | (same as search/ff) |
| action/ffwp | (same as search/ffwp) |
| action/newest | (same as search/newest) |
| action/all | Attempt to apply the call to each file found. If any sub call succeeds the entire operation succeeds and other errors ignored. If all fail then the last error is reported. |
| create/ep | Choose the first path which is found. |
| create/mfs | Assuming the path is found to exist (ENOENT would not be returned) use the drive with the most free space available. |
| create/epmfs | If the path exists in multiple locations use the one with the most free space. Otherwise fall back to mfs. |
| create/rand | Pick a destination at random. Again the dirname of the full path must exist somewhere. |
| ff (first found) | Given the order the paths were provided at mount time act on the first one found (regardless if stat would return EACCES). |
| ffwp (first found w/ permissions) | Given the order the paths were provided at mount time act on the first one found which you have access (stat does not error with EACCES). |
| newest (newest file) | If multiple files exist return the one with the most recent mtime. |
| all (all files found) | Attempt to apply the call to each file found. If any sub call succeeds the entire operation succeeds and other errors ignored. If all fail then the last error is reported. |
| mfs (most free space) | Assuming the path is found to exist (ENOENT would not be returned) use the drive with the most free space available. |
| epmfs (existing path, most free space) | If the path exists in multiple locations use the one with the most free space. Otherwise fall back to mfs. |
| rand (random) | Pick a destination at random. Again the dirname of the full path must exist somewhere. |

#### statvfs ####

Since we aren't trying to stripe data across drives the free space of the mountpoint is just that of the source mount with the most free space at the moment.
Since we aren't trying to stripe data across drives or move them should ENOSPC be return on a write the free space of the mountpoint is just that of the source mount with the most free space at the moment.

**NOTE:** create is really a search for existence and then create. The 'search' policy applies to the first part. If the [dirname](http://linux.die.net/man/3/dirname) of the full path is not found to exist [ENOENT](http://linux.die.net/man/3/errno) is returned.

Expand All @@ -50,17 +44,17 @@ Usage
$ mergerfs -o create=epmfs,search=ff,action=ff <mountpoint> <dir0>:<dir1>:<dir2>
```

| Option | Values | Default |
|--------|--------|---------|
| search | ff, ffwp, newest | ff |
| action | ff, ffwp, newest, all | ff |
| create | ep, mfs, epmfs, rand | epmfs |
| Option | Default |
|--------|--------|
| search | ff |
| action | ff |
| create | epmfs |

Building
========

Need to install FUSE development libraries (libfuse-dev).
Optionally need libattr1 (libattr1-dev).
* Need to install FUSE development libraries (libfuse-dev).
* Optionally need libattr1 (libattr1-dev).


```
Expand All @@ -77,7 +71,7 @@ Runtime Settings
<mountpoint>/.mergerfs
```

There is a pseudo file available at the mountpoint which currently allows for the runtime modification of policies. The file will not show up in readdirs but can be stat'ed, read, and writen. Most other calls will fail with EPERM, EINVAL, or whatever may be appropriate for that call. Anything not understood while writing will result in EINVAL otherwise the number of bytes written will be returned.
There is a pseudo file available at the mountpoint which allows for the runtime modification of policies. The file will not show up in readdirs but can be stat'ed, read, and writen. Most other calls will fail with EPERM, EINVAL, or whatever may be appropriate for that call. Anything not understood while writing will result in EINVAL otherwise the number of bytes written will be returned.

Reading the file will result in a newline delimited list of current settings as followed:

Expand Down Expand Up @@ -106,15 +100,15 @@ If xattrs has been enabled you can also use [{list,get,set}xattrs](http://linux.

```
[trapexit:/tmp/mount] $ attr -l .mergerfs
Attribute "mergerfs.action" has a 3 byte value for .mergerfs
Attribute "mergerfs.create" has a 6 byte value for .mergerfs
Attribute "mergerfs.search" has a 3 byte value for .mergerfs
Attribute "mergerfs.action" has a 2 byte value for .mergerfs
Attribute "mergerfs.create" has a 5 byte value for .mergerfs
Attribute "mergerfs.search" has a 2 byte value for .mergerfs
[trapexit:/tmp/mount] $ attr -g mergerfs.action .mergerfs
Attribute "mergerfs.action" had a 3 byte value for .mergerfs:
Attribute "mergerfs.action" had a 2 byte value for .mergerfs:
ff
[trapexit:/tmp/mount] 1 $ attr -s mergerfs.action -V ffwp .mergerfs
Attribute "mergerfs.action" set to a 3 byte value for .mergerfs:
Attribute "mergerfs.action" set to a 4 byte value for .mergerfs:
ffwp
```
21 changes: 12 additions & 9 deletions src/access.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -45,19 +45,19 @@ using mergerfs::Policy;

static
int
_access(const Policy::Search::Func searchFunc,
const vector<string> &srcmounts,
const string fusepath,
const int mask)
_access(const fs::SearchFunc searchFunc,
const vector<string> &srcmounts,
const string fusepath,
const int mask)
{
int rv;
string path;
fs::PathVector path;

path = searchFunc(srcmounts,fusepath).full;
searchFunc(srcmounts,fusepath,path);
if(path.empty())
return -ENOENT;

rv = ::eaccess(path.c_str(),mask);
rv = ::eaccess(path[0].full.c_str(),mask);

return ((rv == -1) ? -errno : 0);
}
Expand All @@ -74,9 +74,12 @@ namespace mergerfs
const config::Config &config = config::get();

if(fusepath == config.controlfile)
return 0;
return _access(*config.search,
config.srcmounts,
"/",
mask);

return _access(config.policy.search,
return _access(*config.search,
config.srcmounts,
fusepath,
mask);
Expand Down
51 changes: 51 additions & 0 deletions src/buildmap.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
/*
The MIT License (MIT)
Copyright (c) 2014 Antonio SJ Musumeci <trapexit@spawn.link>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
*/

#include <algorithm>

template<typename K,typename V>
class buildmap
{
public:
buildmap(const K &key,
const V &val)
{
_map.insert(std::make_pair(key,val));
}

buildmap<K,V> &operator()(const K &key,
const V &val)
{
_map.insert(std::make_pair(key,val));
return *this;
}

operator std::map<K,V>()
{
return _map;
}

private:
std::map<K,V> _map;
};
51 changes: 51 additions & 0 deletions src/buildvector.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
/*
The MIT License (MIT)
Copyright (c) 2014 Antonio SJ Musumeci <trapexit@spawn.link>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
*/

#include <algorithm>

template<typename V, bool SORT = false>
class buildvector
{
public:
buildvector(const V &val)
{
_vector.push_back(val);
}

buildvector<V,SORT> &operator()(const V &val)
{
_vector.push_back(val);
return *this;
}

operator std::vector<V>()
{
if(SORT == true)
std::sort(_vector.begin(),_vector.end());
return _vector;
}

private:
std::vector<V> _vector;
};
70 changes: 70 additions & 0 deletions src/category.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
/*
The MIT License (MIT)
Copyright (c) 2014 Antonio SJ Musumeci <trapexit@spawn.link>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
*/

#include <string>
#include <vector>

#include "category.hpp"
#include "buildvector.hpp"

#define CATEGORY(X) Category(Category::Enum::X,#X)

namespace mergerfs
{
const std::vector<Category> Category::_categories_ =
buildvector<Category,true>
(CATEGORY(invalid))
(CATEGORY(action))
(CATEGORY(create))
(CATEGORY(search));

const Category * const Category::categories = &_categories_[1];

const Category &Category::invalid = Category::categories[Category::Enum::invalid];
const Category &Category::action = Category::categories[Category::Enum::action];
const Category &Category::create = Category::categories[Category::Enum::create];
const Category &Category::search = Category::categories[Category::Enum::search];

const Category&
Category::find(const std::string str)
{
for(int i = Enum::BEGIN; i != Enum::END; ++i)
{
if(categories[i] == str)
return categories[i];
}

return invalid;
}

const Category&
Category::find(const Category::Enum::Type i)
{
if(i >= Category::Enum::BEGIN &&
i < Category::Enum::END)
return categories[i];

return invalid;
}
}

0 comments on commit aab90b0

Please sign in to comment.