File storage interface improvements #256

alexdutton · 2020-08-19T12:21:04Z

Currently for preliminary review.

See inveniosoftware/rfc#37 for background.

Not all tests pass yet, as the changes to PyFSFileStorage haven't been reflected in the tests.

alexdutton · 2020-08-25T12:17:32Z

Things that we think still need to be done to this:

tests
using Locations to bootstrap where files are stored
letting backends provide checksum functionality (but probably still provide the default implementation)

alexdutton · 2020-10-06T21:47:43Z

I think this is getting there. There's some documentation on what file storage backends now look like at https://github.com/inveniosoftware/invenio-files-rest/blob/9bc8a83163fc50d9b407e29a82d7ef6701770f48/docs/new-storage-backend.rst

I've moved the old implementation into invenio_files_rest.storage.legacy and duplicated the tests as was to ensure no regressions.

The general idea was:

Create a new storage factory for marshalling multiple backends
To reduce the amount of code that needs to be written for a backend, and the understanding needed to do it. Thus, for simple cases, the implementer gets to ignore progress_callback, the file size limit checking and the checksumming
Tidy up the set_uri interface to not use positional arguments, so backends can be more nuanced about what FileInstance attributes they're updating.
Maintain backwards compatibility. Current settings and external storage implementations should still work.

Things still to do:

The tests pass, but I've not checked coverage to see what new functionality needs additional tests
More documentation, including linking in the new page
Adding deprecation warnings, and marking which code supporting backwards compatibility can be removed in the future. I was thinking about adding a dependency on the deprecation package to do this.
Reworking all those commits to be an a lot tidier, because they're a mess (due to lots of exploratory coding and changing of approach)
Updating invenio-s3 to use the new interface

This adds overridable config variables for mapping storage backend names to classes (by default drawn from declared entry points) and setting a default backend for new files. It also adds a pyfs storage backend entry point, and makes it the default. Storage backends should be `django_files_rest.storage:FileStorage` subclasses, and declared with a `django_files_rest.storage` entrypoint.

…gress At the moment, the storage backend is responsible for computing checksums and reporting the size of files based on the number of bytes read. However, these are common concerns across all storage backends, and so this functionality can be pulled the other side of the storage backend interface. This class is therefore intended to wrap any file-like object, and to checksum the file as it is read. It also counts bytes and calls the progress callback if one is provided. In a subsequent commit we'll integrate it with FileInstance, and remove the functionality from the PyFS storage backend.

This new storage factory uses a `FileInstance.storage_backend` attribute to determine which backend to use. It uses the `FILES_REST_DEFAULT_STORAGE_BACKEND` config value when the file instance doesn't already have a storage backend. It can be extended to customize how the backend is chosen, and how it is initialised. The file path is chosen based on the `id` of the file instance, but this can be customised too. It's not set as the default storage factory to preserve backwards compatibility.

…nit classes This property can now instantiate class-based storage factories, i.e. the new factory in `invenio_files_rest.storage.factory`. It's possible the deprecation should be in the pyfs storage factory itself.

These aren't Py3.6-compatible, so will probably need to be removed later, or replaced with comment-based annotations.

This will mean we can later remove this functionality from the storage backend.

This is exposed as a class property on FileStorage and can be accessed as `FileStorage.backend_name`. It returns the associated backend name in the application config, which can then be used to find the class.

It no longer needs to do checksumming, and the filepath can be moved to the superclass.

All together because this has been very exploratory.

There's always a better option than metaclasses. In this case, a classmethod, which also simplifies getting the backend name on instances (`type(instance).backend_name` → `instance.get_backend_name()`)

lnielsen

Impressive job! Thanks a lot for working on this and keeping backward compatibility. I've made a couple of comments

I think the main comment is probably the removal of copy(). It might be easiest to discuss over a telecon. Honestly, I think we can go ahead without adding it back in, but perhaps we make a comment in the code, that this could optimized in the code at a later stage.

lnielsen · 2020-11-20T10:07:03Z

invenio_files_rest/config.py

+FILES_REST_DEFAULT_STORAGE_BACKEND = 'pyfs'
+"""The default storage backend name."""
+
+FILES_REST_STORAGE_BACKENDS = {


Minor: I see some potential troubles here.

How are users expected to provide storage backend?

Only via specifying entry points? In this case, we should probably not make it config, but just a method on the Flask extension.

Allow developers/instances to specify a dict of storage backends? in this case, it makes sense to have a config. However when FILES_REST_STORAGE_BACKENDS is overwritten, then below code will still execute causing all storage backends to be loaded, even if we don't want them.

Thus in both cases, I think it makes sense to move the entry point iteration to the Flask extension to avoid unnecessarily loading storage backends that we don't need.

I was supporting both so that there was an obvious default behaviour (entry points), but that they could override this if desired. The "overriding" use case was that they wanted to change the behaviour of a backend they'd already used, but which they don't maintain (e.g. one in invenio-files-rest, or invenio-s3). In this case defining a new entrypoint with the same name wouldn't work, so they'd have to define the name→backend mapping explicitly as config.

However, yes, it's not great to do the entrypoint dereferencing here regardless of whether it's overridden. If you're happy with the rationale above I can move this to the extension loading code.

I should also write documentation for this.

lnielsen · 2020-11-20T10:12:05Z

invenio_files_rest/models.py

@@ -791,13 +807,45 @@ def verify_checksum(self, progress_callback=None, chunk_size=None,
                               else (self.checksum == real_checksum))
            self.last_check_at = datetime.utcnow()
        return self.last_check
+        return current_files_rest.storage_factory(fileinstance=self, **kwargs)


Minor: Two returns - seems like a merge problem I think?

Yep, it does look that way 😬. Will fix!

lnielsen · 2020-11-20T10:21:34Z

invenio_files_rest/models.py

+        preferred_location = (  # type: typing.Optional[Location]
+            Location(uri=default_location) if default_location else None
+        )
+        return self.initialize(


Question: Previously the init_contents() didn't have a return value. I assume init_contents() is intended to be replaced completely with initialize(), hence would it make sense to keep it like that, or is the return value need somewhere?

You're right; there's no return from initialize(), so the return in init_contents() is unnecessary. I'll get it removed :-)

And yep, I was intending that init_contents() be deprecated.

lnielsen · 2020-11-20T10:27:05Z

invenio_files_rest/models.py

-        self.set_uri(
-            *self.storage(**kwargs).initialize(size=size),
-            readable=False, writable=True)
+        if hasattr(current_files_rest.storage_factory, 'initialize'):


Comment: Perhaps better with isinstance(current_files_rest.storage_factory, StorageBackend)?

Yep. I think this was a hangover from when I'd considered keeping the same base class between old and new approaches, but then decided against it.

lnielsen · 2020-11-20T10:31:34Z

invenio_files_rest/models.py

+        'uri', 'size', 'checksum', 'writable', 'readable', 'storage_class'
+    }
+
+    def update_file_metadata(


Question: Should set_uri() be deprecated?

Yes please! Its use of positional arguments didn't seem particularly clear for people using it, and seems inflexible when trying to omit attribute names or add new attribute names later.

lnielsen · 2020-11-20T10:36:02Z

invenio_files_rest/storage/base.py

+            )
+        self._size = result['size']
+        if not result['checksum']:
+            result['checksum'] = self.checksum(chunk_size=chunk_size)


Question: is this for the case where the checksum is not calculated as part of write stream and we want to ask the backend to do it instead?

Yep. The idea was to make the backend implementation as light-touch as possible and for it to fill in the gaps around what you've provided, unless you want to take control of those things yourself.

In this case, setting the hash name to None will result in checksum not being set from the default write implementation, at which point it'll ask the backend to calculate the checkum explicitly.

lnielsen · 2020-11-20T10:37:39Z

invenio_files_rest/storage/base.py

+            'uri': self.uri,
+            'readable': True,
+            'writable': False,
+            'storage_class': 'S',


Question: Should this be taken from FILES_REST_DEFAULT_STORAGE_CLASS?

Yep! WIll fix.

lnielsen · 2020-11-20T10:44:14Z

invenio_files_rest/storage/base.py


    def copy(self, src, chunk_size=None, progress_callback=None):
        """Copy data from another file instance.

        :param src: Source stream.
        :param chunk_size: Chunk size to read from source stream.
        """
-        fp = src.open(mode='rb')
-        try:
+        warnings.warn(


Minor: The reason for having copy on the storage backend is that it could potentially perform the copy using the backend. Say if both files reside in the same storage system, it would be a lot more efficient to do an internal copy or that system, than having to pass all data through Invenio.

That's true. It shouldn't be too much effort for me to reinstate it.

lnielsen · 2020-11-20T10:51:57Z

invenio_files_rest/storage/pyfs.py

        """Storage initialization."""
-        self.fileurl = fileurl
+        # if isinstance(args[0], str):


Question: Leftover?

Yep; will remove.

lnielsen · 2020-11-20T13:53:48Z

tests/test_cli.py

@@ -103,6 +103,7 @@ def test_simple_workflow(app, db, tmpdir):
    ], obj=script_info)
    assert 0 == result.exit_code

+    print(tmpdir.listdir())


Comment: Left-over?

Yep; will remove.

alexdutton · 2020-11-23T18:33:40Z

@lnielsen All replies done. Will work on the improvements soon, if you're happy with the suggested approach.

I'll do everything as fixups for easier reviewing, and then squash them once we're happy.

lnielsen added this to Backlog in Invenio v3.4 + SemanticUI via automation Aug 19, 2020

lnielsen moved this from Backlog to Blocked in Invenio v3.4 + SemanticUI Aug 28, 2020

alexdutton force-pushed the rfc-37 branch 3 times, most recently from b9280c6 to 067b71c Compare October 6, 2020 13:14

alexdutton added 23 commits October 13, 2020 23:15

Add some type annotations

7774b7d

These aren't Py3.6-compatible, so will probably need to be removed later, or replaced with comment-based annotations.

Deprecate passing **kwargs to the storage factory

bca7ee2

Use the passthrough checksummer

f5244b3

This will mean we can later remove this functionality from the storage backend.

Add a metaclass for FileStorage to return the backend name

c39871a

This is exposed as a class property on FileStorage and can be accessed as `FileStorage.backend_name`. It returns the associated backend name in the application config, which can then be used to find the class.

Simplify the pyfs storage backend implementation

37bd798

It no longer needs to do checksumming, and the filepath can be moved to the superclass.

fixup! Add a new extensible storage factory

568cf75

fixup! Add a new extensible storage factory

9bcd900

intermediate

6e94128

Lots of things. Needs pulling apart before merging.

5899f79

All together because this has been very exploratory.

Remove debugging print statements

2e1a82a

Fix mypy errors

282e618

Remove StorageBackend metaclass

c62cfa0

There's always a better option than metaclasses. In this case, a classmethod, which also simplifies getting the backend name on instances (`type(instance).backend_name` → `instance.get_backend_name()`)

Fix typo

88e5fa8

Reinstate _write_stream, and use context managers

705b95e

Improve checksumming

93bd542

Add the beginning of documentation for writing storage backends

a8fbd15

Opening via the open() method should only be for reading

8a25500

Add delete() to list of methods in docs that need defining

7820f84

Simplify by removing now-unused PassthroughChecksum

d4e660e

alexdutton added 3 commits October 13, 2020 23:16

pydocstyle fixes

0aeefc4

Remove unused method on base StorageBackend

8a55a0b

list to tuple for __all__, to placate pydocstyle

f03154a

alexdutton force-pushed the rfc-37 branch from a84efaa to f03154a Compare October 13, 2020 22:16

alexdutton added 3 commits October 14, 2020 09:23

docstring fixes for pydocstyle

501f35c

Remove unused imports and unused Py3.7+ annotations future

5ac96fe

PEP-8 style fixes. Mostly line-length-related.

5546899

alexdutton self-assigned this Oct 14, 2020

alexdutton added 8 commits October 14, 2020 11:20

More PEP-8 fixes

6b33fd8

The last PEP-8 warnings

da397f8

Run isort, for CI happiness

f56a7c3

Remove more extraneous code

3b0868f

Add migration for backend_name

985d64d

Coalesce on storage_backend, not backend_name for model attr

55d059f

fixup! Add migration for backend_name

0fede69

More style fixes in the alembic migration

dd1a557

alexdutton added this to In Review in InvenioRDM October Board Oct 20, 2020

lnielsen removed this from Blocked in Invenio v3.4 + SemanticUI Oct 28, 2020

lnielsen removed this from In Review in InvenioRDM October Board Oct 28, 2020

lnielsen added this to In Review in InvenioRDM November (really December now) Board Oct 28, 2020

lnielsen added 2 commits November 20, 2020 06:59

global: copyright headers update

142660d

models: remove python 2 support

4879f59

lnielsen self-requested a review November 20, 2020 08:09

lnielsen self-assigned this Nov 20, 2020

lnielsen reviewed Nov 20, 2020

View reviewed changes

lnielsen moved this from In Review to Pending merge in InvenioRDM November (really December now) Board Nov 20, 2020

lnielsen force-pushed the master branch 3 times, most recently from abd43b6 to d1fb4d2 Compare October 20, 2021 13:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

File storage interface improvements #256

File storage interface improvements #256

alexdutton commented Aug 19, 2020

alexdutton commented Aug 25, 2020

alexdutton commented Oct 6, 2020 •

edited

lnielsen left a comment

lnielsen Nov 20, 2020

alexdutton Nov 23, 2020

lnielsen Nov 20, 2020

alexdutton Nov 23, 2020

lnielsen Nov 20, 2020

alexdutton Nov 23, 2020

alexdutton Nov 23, 2020

lnielsen Nov 20, 2020

alexdutton Nov 23, 2020

lnielsen Nov 20, 2020

alexdutton Nov 23, 2020

lnielsen Nov 20, 2020

alexdutton Nov 23, 2020

lnielsen Nov 20, 2020

alexdutton Nov 23, 2020

lnielsen Nov 20, 2020

alexdutton Nov 23, 2020

lnielsen Nov 20, 2020

alexdutton Nov 23, 2020

lnielsen Nov 20, 2020

alexdutton Nov 23, 2020

alexdutton commented Nov 23, 2020

File storage interface improvements #256

Are you sure you want to change the base?

File storage interface improvements #256

Conversation

alexdutton commented Aug 19, 2020

alexdutton commented Aug 25, 2020

alexdutton commented Oct 6, 2020 • edited

lnielsen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexdutton commented Nov 23, 2020

alexdutton commented Oct 6, 2020 •

edited