add aliyunoss

[trueai-org]

hello,

I have already refactored the code and will now provide separate, individual merge PRs for each service. Currently, I have added support for Alibaba Cloud Object Storage Service (OSS) and have provided a test account for evaluation. Of course, you are also welcome to register for an Alibaba Cloud account through their official website for testing purposes.

• test account

# endpoint
oss-cn-hongkong.aliyuncs.com

# AccessKey ID
LTAI5tAtEtqbBduh3enTY3xh

# AccessKey Secret
o5LYRgIGKJOT5W8obYURQW3wPtQwU4

# region
cn-hongkong

# bucket name
hk-duplicati-test

eg.

[kenkendk]

Overall, I think it looks good 👍

[trueai-org]

@kenkendk Thanks, there's no need to worry about the preview issue. Under the Chinese translation preview, the display of open-collective is fine

Prieview Link:

https://github.com/trueai-org/duplicati/blob/master-github-alioss/README.zh-CN.md

[ts678]

I'm happy to see that the project owner likes this, but I have to ask about the status of the S3 option, which maybe now works.

Alibaba oss error 阿里云对象存储报错 #4994

2.0.7.100_canary_2023-12-27

--s3-disable-chunk-encoding added to the AWS backend (only useful for some providers)

Possibly the capabilities or performance differs? We get a lot of requests for new backends, but sometimes existing code will run. Storj is another provider that eventually added an S3 gateway, but I think (not sure) that they charge extra if one goes that way.

Amazon S3 Storage Providers says that rclone supports (but how nicely?) Alibaba Cloud (Aliyun) Object Storage System (OSS).

Although I worry about long term support, short term support, even at initial launch, is a concern, if this turns into a big launch.
User Manual is in English, as is most of the forum work. Add tencent cos, Tencent Cloud Storage supports global services #4236 hasn't had much impact, but load may depend on what marketing was done. As was predicted there, here we are doing Alibaba.

[trueai-org]

@ts678

Thank you for your suggestion. However, I'd like to point out that Alibaba, Tencent, and Baidu, being global industry giants, should operate independently and not rely on AWS. Each storage support should be implemented as a separate module.

Similar to cloud storage providers like QNAP and Synology, who have added individual modules for Alibaba, Tencent, and Baidu, rather than relying on a single implementation. This modular approach is more rational. Of course, it's also beneficial that AWS S3 provides compatibility support.

QNAP eg.

[gpatel-fr]

I have given a quick try to your PR (there is a test build here).
Since (from the code) you don't support creating a folder inside the bucket, I have done a test backup with the --prefix option (if anyone wants to use the test account, it would be wise to do the same).
It seems to work, but it begs the question to why the backend dialog includes a field for a folder if you don't support to create one ?

[gpatel-fr]

@ts678

status of the S3 option

I think that more and more S3 support is broken with recent providers - IIRC that's because there is added multichunk upload capability that is not supported by AWS Duplicati code (and it's not trivial to add it).
That's why I'm hoping to bring the most recent Minio driver in Duplicati where this multichunk support is baked in, so no need to change anything in Duplicati.

[trueai-org]

I have given a quick try to your PR (there is a test build here). Since (from the code) you don't support creating a folder inside the bucket, I have done a test backup with the --prefix option (if anyone wants to use the test account, it would be wise to do the same). It seems to work, but it begs the question to why the backend dialog includes a field for a folder if you don't support to create one ?

@gpatel-fr

Thanks.

The concept of folders, even though virtual in an object storage system, primarily serves to offer a more flexible approach to data management and organization. For instance, if there are multiple backup tasks, Task 1 might be designated to back up to the "/My Videos" directory, while Task 2 might target the "/My Photos" directory. Utilizing this virtual folder structure enables easy specification of distinct backup destinations for different tasks, thereby enhancing the efficiency and intuitiveness of data organization and retrieval.

Alibaba Cloud OSS (Object Storage Service) is fundamentally a distributed object storage system designed to store and access any type of data, such as website content, mobile application data, backup and recovery, big data analytics, and more. In OSS, data is stored as objects within buckets. An object consists of the object data and metadata, and each object is assigned a unique identifier (Key) for retrieval.

In an object storage system like OSS, the concept of a "folder" is virtual rather than physical, as found in traditional file systems. OSS does not require the creation of folders in the way file systems do because it is not based on a hierarchical file system structure but manages data through the object's Key. Object Keys can include the slash (/) character, making the paths appear folder-like, but this is merely part of the Key, and OSS does not create actual folders based on the slashes in Keys.

For example, if you upload an object and name its Key "images/photo.jpg," OSS stores it as a single object with the Key "images/photo.jpg." Although this appears as if there is a file named "photo.jpg" inside an "images" folder, in OSS, "images" is not an actual folder but part of the object's Key.

Alibaba Cloud OSS offers a way to simulate folder functionality by using slashes (/) in object Keys, creating a logical structure similar to folders for organizing and managing objects. Users can upload, download, and manage objects using the OSS management console, OSS APIs, or SDKs, as if they were operating files and folders in a file system. However, it's important to understand that these "folders" do not physically exist; they are simply part of the object Keys.

Therefore, OSS does not support the creation of actual folders due to its design philosophy and storage mechanism, which differ from traditional file systems. By using slashes in object Keys to simulate folder structures, OSS provides a flexible way to organize data while maintaining the simplicity and scalability of object storage.

[trueai-org]

Simplified, when uploading files, it automatically creates folders (virtual ones), so there's no need to create folders separately.

[ts678]

Sometimes for efficiency, the pseudo-folder prefix is given the object store so that the filtering can be remote rather than local.
Add support for B2 'prefix' API parameter when listing files #3803 was a case that was added. I haven't looked at this new trio.
We also get interesting edge cases where people vary from the usual path plan which I think doesn't want an initial slash used.

[trueai-org]

Sometimes for efficiency, the pseudo-folder prefix is given the object store so that the filtering can be remote rather than local. Add support for B2 'prefix' API parameter when listing files #3803 was a case that was added. I haven't looked at this new trio. We also get interesting edge cases where people vary from the usual path plan which I think doesn't want an initial slash used.

@ts678 yeah, aliyun oss the code already implements similar optimizations, specifically loading files based on prefixes/folders.

 var listObjectsRequest = new ListObjectsRequest(bucketName)
 {
     MaxKeys = 1000,
     Marker = nextMarker,
     Prefix = prefix,
 };

[kenkendk]

Looks solid, lets get feedback from users.

[kenkendk]

@trueai-org Do you have reports from users?

[trueai-org]

@trueai-org Do you have reports from users?

@kenkendk

Currently, we have not received any bug reports or feedback on issues from users regarding Alibaba Cloud's OSS. Additionally, I have not yet engaged in widespread discussions or promotions on domestic community platforms such as Douyin, Xiaohongshu, Baidu, Zhihu, Tieba, etc., or with external company teams and partner clients. As for the OAuth-related issues (duplicati/oauth-server#1
), my current plan is to address these matters first and then produce a dedicated introductory video and guide, which will be released across the web to provide more comprehensive information and support.