Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fuse-overlayfs: add C plugins system #119

Merged
merged 11 commits into from Oct 31, 2019
Merged

Conversation

giuseppe
Copy link
Member

Alternative for #79

Add a simple plugin mechanism that will help to expand fuse-overlayfs
functionalities, in particular it allows to load data from a layer on
demand.

A plugin is loaded into fuse-overlayfs using the option:

-o plugins=path/to/plugin.so:path/to/another/plugin.so

A layer can use a plugin with the syntax:

-o lowerdir=//plugin-name/DATA-FOR-THE-PLUGIN/path

Each time a file/directory is looked up, if a plugin is registered for
a layer, the plugin is first notified about the request.

After the callback is invoked, fuse-overlayfs still expects the data
to be accessible at the specified directory.

Signed-off-by: Giuseppe Scrivano gscrivan@redhat.com

@giuseppe giuseppe changed the title fuse-overlayfs: add C plugins system [WIP] fuse-overlayfs: add C plugins system Sep 13, 2019
@lgtm-com
Copy link

lgtm-com bot commented Sep 13, 2019

This pull request introduces 3 alerts when merging dd49509 into 74fb3dd - view on LGTM.com

new alerts:

  • 3 for Comparison result is always the same

@giuseppe giuseppe force-pushed the c-plugins branch 3 times, most recently from b512206 to 5b64df4 Compare September 13, 2019 09:09
@giuseppe
Copy link
Member Author

a test plugin: https://gist.github.com/giuseppe/a669ed7248de557a9b5fd272ffe2a4f4

It allows to use a shared storage owned by another user as a lower layer.

as root:

# podman create --name foo fedora ls
# cp -r $(podman mount -l) rootfs
# podman umount -l
# podman rm -l
# ./convert rootfs

The last command convert each file to mode 0755, the original permission is stored in an extended attribute.

As rootless then we can do:

$ podman unshare
# mkdir upper workdir merged
# ./fuse-overlayfs -o plugins=$(pwd)/test-plugin.so -o lowerdir=//test//rootfs/,upperdir=upper,workdir=workdir merged
# podman run --rm -ti --rootfs $(pwd)/merged bash

The new syntax for lowerdir starting with // says to fuse-overlayfs to use a plugin to handle that layer.

The original files from the image are owned by root, the fuse-overlayfs plugin makes sure the original permissions are used inside of the container.

A second rootless user could use the same command and re-use the lower layer from the root storage.

@giuseppe
Copy link
Member Author

@vrothberg @rhatdan ⬆️ this is the idea I had in mind for sharing the storage for rootless users.

@rhatdan
Copy link
Member

rhatdan commented Sep 13, 2019

This would only allow us to copy one image. How would I set this up to allow the use of 100 images?
Does convert switch all file protections to 755? Even ones that are looser?

@giuseppe
Copy link
Member Author

@rhatdan yes, it must switch all protections to 0755, also ones that are looser. No file must be world writeable as it can affect other users of the image.

root, or more in general the owner of a storage, must explicitly convert or copy+clone (if it is still needed with native overlay) each layer that is going to be shared. If the underlying file system support reflinks though, there is only the additional cost of creating inodes.

@rhatdan
Copy link
Member

rhatdan commented Sep 13, 2019

We might be able to take advantage of Overlay MetaCopy=on as well.

@rhatdan
Copy link
Member

rhatdan commented Sep 13, 2019

Could container storage do this for us, and just take a group of images or all images and mount them up to be used by fuse-overlay?

@giuseppe
Copy link
Member Author

Could container storage do this for us, and just take a group of images or all images and mount them up to be used by fuse-overlay?

yes once the low level bits are working, I think we'll need to move the logic of creating such images into containers/storage so that we could have something: podman image make-shared $IMAGE

@vrothberg
Copy link
Member

Currently under water with tackling bugs. Will have a look as soon as I find time. Thanks for the ping, @giuseppe !

@giuseppe giuseppe force-pushed the c-plugins branch 2 times, most recently from c8f8c47 to 571b72f Compare September 16, 2019 13:16
@giuseppe giuseppe changed the title [WIP] fuse-overlayfs: add C plugins system fuse-overlayfs: add C plugins system Oct 3, 2019
@giuseppe
Copy link
Member Author

ready for review

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
this is the first step towards supporting data from lower layers
coming from other sources.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Add a simple plugin mechanism that will help to expand fuse-overlayfs
functionalities, in particular it allows to load data from a layer on
demand.

A plugin is loaded into fuse-overlayfs using the option:

-o plugins=path/to/plugin.so:path/to/another/plugin.so

A layer can use a plugin with the syntax:

-o lowerdir=//plugin-name/DATA-FOR-THE-PLUGIN/path

Each time a file/directory is looked up, if a plugin is registered for
a layer, the plugin is first notified about the request.

After the callback is invoked, fuse-overlayfs still expects the data
to be accessible at the specified directory.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
differently than dev/ino number, use the uppermost mode found.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
@giuseppe giuseppe force-pushed the c-plugins branch 4 times, most recently from 17f8ffa to 4792fd0 Compare October 24, 2019 08:55
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
use EXIT_FAILURE and EXIT_SUCCESS instead.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
@giuseppe
Copy link
Member Author

a plugin for CRFS: https://github.com/giuseppe/crfs-plugin

if no -o plugins is specified, load them from $PKGLIBEXECDIR (usually
has the value /usr/libexec/fuse-overlayfs).

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
@giuseppe giuseppe force-pushed the c-plugins branch 2 times, most recently from 978ecd2 to 161914c Compare October 31, 2019 11:44
Closes: containers#136

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
@giuseppe
Copy link
Member Author

@rhatdan let's merge if there are no problems and I will cut a new release

@rhatdan
Copy link
Member

rhatdan commented Oct 31, 2019

LGTM

@rhatdan rhatdan merged commit fa0cd99 into containers:master Oct 31, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants