Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support templating name of files stored via Rclone backend #4606

Merged
merged 1 commit into from
Jun 24, 2023

Conversation

dalbani
Copy link
Contributor

@dalbani dalbani commented Jun 24, 2023

What problem are we solving?

Although the Rclone backend implemented in #4402 works fine, the current implementation has the limitation of storing all files at the root of a Rclone remote, next to each other in the same "folder".
In a SeaweedFS deployment like mine with, hmm, tens of thousands of volume files, I'd like to have a way to split all those files into subfolders.

How are we solving the problem?

By introducing a new configuration property that lets the user provide a template generating the filename from the volume UUID.

How is the PR tested?

Manually, by configuring such a backend:

[storage.backend]
[storage.backend.rclone.default]
enabled = true
remote_name = "onedrive"
key_template = "SeaweedFS/{{ slice . 0 2 }}/{{ slice . 2 4 }}/{{ slice . 4 }}"

And checking that volume files were uploaded in OneDrvive in such a structure:

SeaweedFS/
  5f/
    e5/
      03d8-0aba-4e42-b1c8-6116bf862f71

Which matches the output of volume.list:

...
volume id:8  
  size:106774344
  collection:"bucket-1"
  ...
  read_only:true
  version:3
  modified_at_second:1687612319
  remote_storage_name:"rclone.default"
  remote_storage_key:"SeaweedFS/5f/e5/03d8-0aba-4e42-b1c8-6116bf862f71" 
...

Checks

  • I have added unit tests if possible.
  • I will add related wiki document changes and link to this PR after merging.

Other considerations

If this template functionality is deemed useful for the Rclone backend, I propose to implement it in a generic fashion, so that the S3 backend can benefit from it as well.

@chrislusf chrislusf merged commit 0d9ba8c into seaweedfs:master Jun 24, 2023
5 checks passed
@dalbani
Copy link
Contributor Author

dalbani commented Jun 24, 2023

Thanks for this quick reaction.

FYI, I have just documented this new functionality in https://github.com/seaweedfs/seaweedfs/wiki/Cloud-Tier#rclone.

@dalbani dalbani deleted the rclone-backend-key-template branch June 24, 2023 15:04
kmlebedev pushed a commit to kmlebedev/seaweedfs that referenced this pull request Dec 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants