Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fuse-overlayfs: add C plugins system #119

Merged
merged 11 commits into from Oct 31, 2019

Conversation

@giuseppe
Copy link
Member

giuseppe commented Sep 13, 2019

Alternative for #79

Add a simple plugin mechanism that will help to expand fuse-overlayfs
functionalities, in particular it allows to load data from a layer on
demand.

A plugin is loaded into fuse-overlayfs using the option:

-o plugins=path/to/plugin.so:path/to/another/plugin.so

A layer can use a plugin with the syntax:

-o lowerdir=//plugin-name/DATA-FOR-THE-PLUGIN/path

Each time a file/directory is looked up, if a plugin is registered for
a layer, the plugin is first notified about the request.

After the callback is invoked, fuse-overlayfs still expects the data
to be accessible at the specified directory.

Signed-off-by: Giuseppe Scrivano gscrivan@redhat.com

@giuseppe giuseppe changed the title fuse-overlayfs: add C plugins system [WIP] fuse-overlayfs: add C plugins system Sep 13, 2019
@lgtm-com

This comment has been minimized.

Copy link

lgtm-com bot commented Sep 13, 2019

This pull request introduces 3 alerts when merging dd49509 into 74fb3dd - view on LGTM.com

new alerts:

  • 3 for Comparison result is always the same
@giuseppe giuseppe force-pushed the giuseppe:c-plugins branch 3 times, most recently from b512206 to 5b64df4 Sep 13, 2019
@giuseppe

This comment has been minimized.

Copy link
Member Author

giuseppe commented Sep 13, 2019

a test plugin: https://gist.github.com/giuseppe/a669ed7248de557a9b5fd272ffe2a4f4

It allows to use a shared storage owned by another user as a lower layer.

as root:

# podman create --name foo fedora ls
# cp -r $(podman mount -l) rootfs
# podman umount -l
# podman rm -l
# ./convert rootfs

The last command convert each file to mode 0755, the original permission is stored in an extended attribute.

As rootless then we can do:

$ podman unshare
# mkdir upper workdir merged
# ./fuse-overlayfs -o plugins=$(pwd)/test-plugin.so -o lowerdir=//test//rootfs/,upperdir=upper,workdir=workdir merged
# podman run --rm -ti --rootfs $(pwd)/merged bash

The new syntax for lowerdir starting with // says to fuse-overlayfs to use a plugin to handle that layer.

The original files from the image are owned by root, the fuse-overlayfs plugin makes sure the original permissions are used inside of the container.

A second rootless user could use the same command and re-use the lower layer from the root storage.

@giuseppe

This comment has been minimized.

Copy link
Member Author

giuseppe commented Sep 13, 2019

@vrothberg @rhatdan ⬆️ this is the idea I had in mind for sharing the storage for rootless users.

@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Sep 13, 2019

This would only allow us to copy one image. How would I set this up to allow the use of 100 images?
Does convert switch all file protections to 755? Even ones that are looser?

@giuseppe

This comment has been minimized.

Copy link
Member Author

giuseppe commented Sep 13, 2019

@rhatdan yes, it must switch all protections to 0755, also ones that are looser. No file must be world writeable as it can affect other users of the image.

root, or more in general the owner of a storage, must explicitly convert or copy+clone (if it is still needed with native overlay) each layer that is going to be shared. If the underlying file system support reflinks though, there is only the additional cost of creating inodes.

@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Sep 13, 2019

We might be able to take advantage of Overlay MetaCopy=on as well.

@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Sep 13, 2019

Could container storage do this for us, and just take a group of images or all images and mount them up to be used by fuse-overlay?

@giuseppe

This comment has been minimized.

Copy link
Member Author

giuseppe commented Sep 14, 2019

Could container storage do this for us, and just take a group of images or all images and mount them up to be used by fuse-overlay?

yes once the low level bits are working, I think we'll need to move the logic of creating such images into containers/storage so that we could have something: podman image make-shared $IMAGE

@giuseppe giuseppe force-pushed the giuseppe:c-plugins branch from 961f108 to 486b8f2 Sep 14, 2019
@vrothberg

This comment has been minimized.

Copy link
Member

vrothberg commented Sep 16, 2019

Currently under water with tackling bugs. Will have a look as soon as I find time. Thanks for the ping, @giuseppe !

@giuseppe giuseppe force-pushed the giuseppe:c-plugins branch 2 times, most recently from c8f8c47 to 571b72f Sep 16, 2019
@giuseppe giuseppe force-pushed the giuseppe:c-plugins branch from 571b72f to fa9204e Sep 26, 2019
@giuseppe giuseppe force-pushed the giuseppe:c-plugins branch from fa9204e to 0a8c375 Oct 3, 2019
@giuseppe giuseppe changed the title [WIP] fuse-overlayfs: add C plugins system fuse-overlayfs: add C plugins system Oct 3, 2019
@giuseppe giuseppe force-pushed the giuseppe:c-plugins branch from 0a8c375 to 795fd78 Oct 3, 2019
@giuseppe

This comment has been minimized.

Copy link
Member Author

giuseppe commented Oct 10, 2019

ready for review

giuseppe added 3 commits Sep 11, 2019
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
this is the first step towards supporting data from lower layers
coming from other sources.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
@giuseppe giuseppe force-pushed the giuseppe:c-plugins branch from dfde1a8 to 577f0c5 Oct 20, 2019
Add a simple plugin mechanism that will help to expand fuse-overlayfs
functionalities, in particular it allows to load data from a layer on
demand.

A plugin is loaded into fuse-overlayfs using the option:

-o plugins=path/to/plugin.so:path/to/another/plugin.so

A layer can use a plugin with the syntax:

-o lowerdir=//plugin-name/DATA-FOR-THE-PLUGIN/path

Each time a file/directory is looked up, if a plugin is registered for
a layer, the plugin is first notified about the request.

After the callback is invoked, fuse-overlayfs still expects the data
to be accessible at the specified directory.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
giuseppe added 2 commits Sep 14, 2019
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
differently than dev/ino number, use the uppermost mode found.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
@giuseppe giuseppe force-pushed the giuseppe:c-plugins branch from 577f0c5 to 62abdcf Oct 20, 2019
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
@giuseppe giuseppe force-pushed the giuseppe:c-plugins branch 4 times, most recently from 17f8ffa to 4792fd0 Oct 23, 2019
giuseppe added 2 commits Oct 23, 2019
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
use EXIT_FAILURE and EXIT_SUCCESS instead.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
@giuseppe giuseppe force-pushed the giuseppe:c-plugins branch from dba5172 to 3fa1034 Oct 24, 2019
@giuseppe

This comment has been minimized.

Copy link
Member Author

giuseppe commented Oct 24, 2019

a plugin for CRFS: https://github.com/giuseppe/crfs-plugin

if no -o plugins is specified, load them from $PKGLIBEXECDIR (usually
has the value /usr/libexec/fuse-overlayfs).

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
@giuseppe giuseppe force-pushed the giuseppe:c-plugins branch 2 times, most recently from 978ecd2 to 161914c Oct 31, 2019
Closes: #136

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
@giuseppe giuseppe force-pushed the giuseppe:c-plugins branch from 161914c to 042846d Oct 31, 2019
@giuseppe

This comment has been minimized.

Copy link
Member Author

giuseppe commented Oct 31, 2019

@rhatdan let's merge if there are no problems and I will cut a new release

@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Oct 31, 2019

LGTM

@rhatdan rhatdan merged commit fa0cd99 into containers:master Oct 31, 2019
2 checks passed
2 checks passed
LGTM analysis: C/C++ No new or fixed alerts
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.