Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fuse-overlayfs: add C plugins system #119

Merged
merged 11 commits into from Oct 31, 2019
Merged

Conversation

@giuseppe
Copy link
Member

@giuseppe giuseppe commented Sep 13, 2019

Alternative for #79

Add a simple plugin mechanism that will help to expand fuse-overlayfs
functionalities, in particular it allows to load data from a layer on
demand.

A plugin is loaded into fuse-overlayfs using the option:

-o plugins=path/to/plugin.so:path/to/another/plugin.so

A layer can use a plugin with the syntax:

-o lowerdir=//plugin-name/DATA-FOR-THE-PLUGIN/path

Each time a file/directory is looked up, if a plugin is registered for
a layer, the plugin is first notified about the request.

After the callback is invoked, fuse-overlayfs still expects the data
to be accessible at the specified directory.

Signed-off-by: Giuseppe Scrivano gscrivan@redhat.com

@giuseppe giuseppe changed the title fuse-overlayfs: add C plugins system [WIP] fuse-overlayfs: add C plugins system Sep 13, 2019
@lgtm-com
Copy link

@lgtm-com lgtm-com bot commented Sep 13, 2019

This pull request introduces 3 alerts when merging dd49509 into 74fb3dd - view on LGTM.com

new alerts:

  • 3 for Comparison result is always the same

@giuseppe giuseppe force-pushed the c-plugins branch 3 times, most recently from b512206 to 5b64df4 Sep 13, 2019
@giuseppe
Copy link
Member Author

@giuseppe giuseppe commented Sep 13, 2019

a test plugin: https://gist.github.com/giuseppe/a669ed7248de557a9b5fd272ffe2a4f4

It allows to use a shared storage owned by another user as a lower layer.

as root:

# podman create --name foo fedora ls
# cp -r $(podman mount -l) rootfs
# podman umount -l
# podman rm -l
# ./convert rootfs

The last command convert each file to mode 0755, the original permission is stored in an extended attribute.

As rootless then we can do:

$ podman unshare
# mkdir upper workdir merged
# ./fuse-overlayfs -o plugins=$(pwd)/test-plugin.so -o lowerdir=//test//rootfs/,upperdir=upper,workdir=workdir merged
# podman run --rm -ti --rootfs $(pwd)/merged bash

The new syntax for lowerdir starting with // says to fuse-overlayfs to use a plugin to handle that layer.

The original files from the image are owned by root, the fuse-overlayfs plugin makes sure the original permissions are used inside of the container.

A second rootless user could use the same command and re-use the lower layer from the root storage.

@giuseppe
Copy link
Member Author

@giuseppe giuseppe commented Sep 13, 2019

@vrothberg @rhatdan ⬆️ this is the idea I had in mind for sharing the storage for rootless users.

@rhatdan
Copy link
Member

@rhatdan rhatdan commented Sep 13, 2019

This would only allow us to copy one image. How would I set this up to allow the use of 100 images?
Does convert switch all file protections to 755? Even ones that are looser?

@giuseppe
Copy link
Member Author

@giuseppe giuseppe commented Sep 13, 2019

@rhatdan yes, it must switch all protections to 0755, also ones that are looser. No file must be world writeable as it can affect other users of the image.

root, or more in general the owner of a storage, must explicitly convert or copy+clone (if it is still needed with native overlay) each layer that is going to be shared. If the underlying file system support reflinks though, there is only the additional cost of creating inodes.

@rhatdan
Copy link
Member

@rhatdan rhatdan commented Sep 13, 2019

We might be able to take advantage of Overlay MetaCopy=on as well.

@rhatdan
Copy link
Member

@rhatdan rhatdan commented Sep 13, 2019

Could container storage do this for us, and just take a group of images or all images and mount them up to be used by fuse-overlay?

@giuseppe
Copy link
Member Author

@giuseppe giuseppe commented Sep 14, 2019

Could container storage do this for us, and just take a group of images or all images and mount them up to be used by fuse-overlay?

yes once the low level bits are working, I think we'll need to move the logic of creating such images into containers/storage so that we could have something: podman image make-shared $IMAGE

@vrothberg
Copy link
Member

@vrothberg vrothberg commented Sep 16, 2019

Currently under water with tackling bugs. Will have a look as soon as I find time. Thanks for the ping, @giuseppe !

@giuseppe giuseppe force-pushed the c-plugins branch 2 times, most recently from c8f8c47 to 571b72f Sep 16, 2019
@giuseppe giuseppe changed the title [WIP] fuse-overlayfs: add C plugins system fuse-overlayfs: add C plugins system Oct 3, 2019
@giuseppe
Copy link
Member Author

@giuseppe giuseppe commented Oct 10, 2019

ready for review

giuseppe added 3 commits Oct 20, 2019
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
this is the first step towards supporting data from lower layers
coming from other sources.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Add a simple plugin mechanism that will help to expand fuse-overlayfs
functionalities, in particular it allows to load data from a layer on
demand.

A plugin is loaded into fuse-overlayfs using the option:

-o plugins=path/to/plugin.so:path/to/another/plugin.so

A layer can use a plugin with the syntax:

-o lowerdir=//plugin-name/DATA-FOR-THE-PLUGIN/path

Each time a file/directory is looked up, if a plugin is registered for
a layer, the plugin is first notified about the request.

After the callback is invoked, fuse-overlayfs still expects the data
to be accessible at the specified directory.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
giuseppe added 2 commits Oct 20, 2019
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
differently than dev/ino number, use the uppermost mode found.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
@giuseppe giuseppe force-pushed the c-plugins branch 4 times, most recently from 17f8ffa to 4792fd0 Oct 24, 2019
giuseppe added 2 commits Oct 24, 2019
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
use EXIT_FAILURE and EXIT_SUCCESS instead.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
@giuseppe
Copy link
Member Author

@giuseppe giuseppe commented Oct 24, 2019

a plugin for CRFS: https://github.com/giuseppe/crfs-plugin

if no -o plugins is specified, load them from $PKGLIBEXECDIR (usually
has the value /usr/libexec/fuse-overlayfs).

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
@giuseppe giuseppe force-pushed the c-plugins branch 2 times, most recently from 978ecd2 to 161914c Oct 31, 2019
Closes: containers#136

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
@giuseppe
Copy link
Member Author

@giuseppe giuseppe commented Oct 31, 2019

@rhatdan let's merge if there are no problems and I will cut a new release

@rhatdan
Copy link
Member

@rhatdan rhatdan commented Oct 31, 2019

LGTM

@rhatdan rhatdan merged commit fa0cd99 into containers:master Oct 31, 2019
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants