S3 Versioning

[fdefilippo]

How about supporting s3 buckets with versioning enabled? Minio with the new API 7.x supports them.
thnx

$ s3mc ls --versions local/test
[2020-07-11 10:39:00 CEST]  1.7KiB group (v3, 14d06c4b-a2a0-4ab7-afe1-e1edb8cd8676)
[2020-07-11 10:38:59 CEST]  1.7KiB group (v2, 6f02c9fd-da14-497c-8355-e0350a275f4e)
[2020-07-11 10:38:58 CEST]  1.7KiB group (v1, 3abd225c-ad0e-4f32-934b-806823053959)
[2020-07-23 23:06:38 CEST]    615B send.sh (v2, e9b064e3-a5ad-4756-b94f-b57ba48b9853)
[2020-07-23 23:06:35 CEST]    615B send.sh (v1, 2d1acf6e-ff5b-43e9-95f3-0dd5f1dcf02e)
[2020-06-02 16:54:02 CEST]  1.1KiB test.sh (v1, null)
[2020-07-22 12:13:12 CEST]      0B uno (v6, 4827a02d-4860-4ae9-bd79-b33299330317, delete-marker)
[2020-07-12 11:40:18 CEST]  1.7KiB uno (v5, 2da7024f-e7f9-4ba0-9810-66142b881016)
[2020-07-12 11:39:48 CEST]  1.7KiB uno (v4, c63802ef-28a7-4216-bb61-05976bbb52cc)
[2020-07-12 11:30:59 CEST]  1.7KiB uno (v3, 7bad9ec9-7bce-4dce-bdbf-caddfef44896)
[2020-07-12 11:30:27 CEST]  1.7KiB uno (v2, 77f02b51-f09d-4273-9d9c-5c842008a123)
[2020-07-12 11:30:12 CEST]  1.7KiB uno (v1, 28c2e63a-9d6c-432f-98b8-6748deba9d52)

[drakkan]

Hi,

the next week I should have access to a versioned S3 bucket (thank to one of the SFTPGo sponsors) and I'll evaluate this feature in more detail.

My first impression is that you can enable versioning and recover files using the AWS console, while you should see the current version via SFTP/FTP. Basically I agree with s3fs maintainers.

Do you know if the AWS transfer services expose versioning over SFTP/FTP? We could take inspiration from them to understand how they list files etc.. I never used these services and I have no way to test them. Thank you

[fdefilippo]

Hi, for the versioned bucket I can give you access right away, In my mind the versioned files should simply be shown with a tag:

mc ls --versions clusters3/sftpgo

[2021-04-25 11:31:21 CEST] 676KiB 116f6699-27c8-41a6-9d6e-dc774b39d364 v3 PUT test.txt
[2021-04-25 11:31:19 CEST] 676KiB 945cf8f5-a788-4c74-90e9-b244e87d4cec v2 PUT test.txt
[2021-04-25 11:31:17 CEST] 676KiB 87878be4-d27d-47b7-be5e-7ab0a2f27f76 v1 PUT test.txt

sftp> ls (should show something like)

test.txt
test.txt:v2
test.txt:v3

ps: if you think I can help you with accessing the versioned bucket email me to: francesco at defilippo dot org

[drakkan]

Hi, for the versioned bucket I can give you access right away, In my mind the versioned files should simply be shown with a tag:

mc ls --versions clusters3/sftpgo

[2021-04-25 11:31:21 CEST] 676KiB 116f6699-27c8-41a6-9d6e-dc774b39d364 v3 PUT test.txt
[2021-04-25 11:31:19 CEST] 676KiB 945cf8f5-a788-4c74-90e9-b244e87d4cec v2 PUT test.txt
[2021-04-25 11:31:17 CEST] 676KiB 87878be4-d27d-47b7-be5e-7ab0a2f27f76 v1 PUT test.txt

sftp> ls (should show something like)

test.txt
test.txt:v2
test.txt:v3

this is exactly my concern, we are adding metadata to protocols that does not support them, and so we add something to file names. To work with versions we need to parse these metadata before doing S3 operations (for example see CopyObject docs, the version id is required). So you can have issues with some filenames.

To proper support this feature, probably the best strategy is to store all the files within the data provider and keep the metadata there. This will also allow to support dead properties for WebDAV, but it has several drawbacks, I'm not sure I want to add something like this now. Maybe when we'll have a plugin system

ps: if you think I can help you with accessing the versioned bucket email me to: francesco at defilippo dot org

This is not required anymore. I would appreciate if you can test AWS transfer for SFTP/FTP and report how it works with versioning enabled, thank you

[drakkan]

I don't plan to work on this sorry. However it would be interesting to understand how AWS transfer for SFTP/FTP handles versioning to better evaluate if add support for this feature