Added support for setting file IDs explicitly. #70
Conversation
…lly with MongoDB.
|
Fixes #69. |
|
Most of depot design is centered around the concept that users should never care about the IDs. They are an opaque autogenerated thing. You should usually access the files through their SQLAlchemy or Ming interface, so the ID is only helpful for DEPOT itself to retrieve the right file when the column of models is accessed. If possible I'm for keeping ids out of the hands of users. If the goal is to restore a backup, then the backup should restore the same exact IDs and thus I don't get the need for passing the Ids to the file saving function. Are you trying to make&restore the backup through depot itself? Because that wasn't what depot was built for and it takes for granted that your backup tools depend on the storage backend |
|
I'm using a depot as the backup itself. I guess it's intended to be as much
import/export as backup and restore, so I wanted to be able to pull the
data out from whatever backend is being used without having to deal with
the specifics, nor to have to rewrite the database as it goes in or out.
Thanks,
Jamie
…On Tue, Sep 21, 2021 at 11:22 AM Alessandro Molina ***@***.***> wrote:
Most of depot design is centered around the concept that users should
never care about the IDs. They are an opaque autogenerated thing. You
should usually access the files through their SQLAlchemy or Ming interface,
so the ID is only helpful for DEPOT itself to retrieve the right file when
the column of models is accessed. If possible I'm for keeping ids out of
the hands of users.
If the goal is to restore a backup, then the backup should restore the
same exact IDs and thus I don't get the need for passing the Ids to the
file saving function. Are you trying to make&restore the backup through
depot itself? Because that wasn't what depot was built for and it takes for
granted that your backup tools depend on the storage backend
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<https://github.com/amol-/depot/pull/70#issuecomment-924095098>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAETCEOS67I2LXMUMRCVTQLUDCPMHANCNFSM5BA7SDCA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
--
Jamie McCusker (she/they)
Director, Data Operations
Tetherless World Constellation
Rensselaer Polytechnic Institute
***@***.*** ***@***.***>
http://tw.rpi.edu
|
| self.fs.delete(file_id) | ||
| assert not self.fs.exists(file_id) | ||
|
|
||
| self.fs.create(FILE_CONTENT, fileid=file_id) |
There was a problem hiding this comment.
I think you want a assert self.fs.exists(file_id) affert this line. Otherwise we wouldn't be testing much.
|
Will setting a custom ID not conflict with the logic in I came onto this MR as I want to set a path prefix for uploaded files. |
No, since everything uses UUIDs and validates them as such. If you're inserting a UUID, you'll be fine. |
|
I'm considering a different approach to this one. Instead of allowing the users to pass ID explicitly, which is something I want to avoid as the ID should be totally opaque and an implementation detail. The idea is that |
|
Works for me, looking forward to the PR. |
@jpmccu See https://github.com/amol-/depot/blob/master/docs/userguide.rst#performing-backups-between-storages for docs, the PR is merged. |
Use with care, especially with MongoDB. This is probably going to be a controversial PR, but I wanted to put it out there. It should be 100% backwards compatible.