Skip to content

Commit

Permalink
Additions to strage and migration
Browse files Browse the repository at this point in the history
  • Loading branch information
topless committed May 21, 2019
1 parent ac17d0e commit b398d73
Show file tree
Hide file tree
Showing 2 changed files with 20 additions and 9 deletions.
11 changes: 6 additions & 5 deletions docs/usage/migration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,15 @@ When we create a bucket we need to provide a Location, otherwise the default
one is used.

.. note::
Before migrating our models, we need to transfer the actual files in the
new storage with the new location. Then we can just bulk update our
FileInstance objects pointing to the new bucket.
Before updating our records to point to the new :code:`Location`, we need
to copy the actual files in the new storage with the new location.
Then we can just bulk update our FileInstance objects pointing to the new
bucket.

There is a task :code:`invenio_files_rest.tasks.migrate_file`
which is used to migrate a file instance to a new location. A new location
which is used to migrate a :code:`FileInstance` to a new location. A new location
might be in remote system on a different bucket, even on a different storage
backend. If the process does not complete successfully then the destination
FileInstance is removed completely. The task also accepts an argument
:code:`FileInstance` is removed completely. The task also accepts an argument
:code:`post_fixity_check` which enables the file checksum verification after
the operation is complete.
18 changes: 14 additions & 4 deletions docs/usage/storage_backends.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,25 @@ Storage Backends
In order to get started we need to setup and configure a storage backend.
Storage will serve as an interface for the actual file access.

In the configuration of the application, we can set the path of the factory that we want to use in order to create a storage instance.
Each file instance uses the storage factory to create a storage interface for itself.
In the configuration of the application, we can set the path of the factory
that we want to use in order to create a storage instance. Each file instance
uses the storage factory to create a storage interface for itself.

Under the hood the module uses `PyFilesystem <https://www.pyfilesystem.org/>`_
to save files. This file system abstraction gives us the flexibility to be able
to swap storages very easy. The storage backend can also be a cloud service, as
an example you can have a look at
`invenio-s3 <https://invenio-s3.readthedocs.io/>`_ which offers integration with
any S3 REST API compatible object storage.


Build your own Storage Backend
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Advanced topic on how to implement and connect your own storage Backend for
invenio-files-rest.

In order to use a different storage backend, we just need to subclass the FileStorage class,
and provide implementations for some of the methods, as follows:
In order to use a different storage backend, we just need to subclass the
FileStorage class, and provide implementations for some of the methods, as follows:

Mandatory methods to implement:

Expand All @@ -37,3 +46,4 @@ Optional methods to implement:
* :code:`_init_hash`
* :code:`_compute_checksum`
* :code:`_write_stream`

0 comments on commit b398d73

Please sign in to comment.