add Baidu NetDisk, add Aliyun Drive, add Aliyun OSS

[trueai-org]

Baidu Netdisk, as of September 2022, has surpassed 800 million users, encompassing both domestic and international users. It offers users free storage space ranging from 1TB to 8TB, providing ample capacity for diverse storage needs. Supporting multiple platforms including Windows, macOS, iOS, and Android, Baidu Netdisk facilitates easy file management across various devices. Its robust file sharing and collaboration features enable secure link sharing and multi-person document collaboration, enhancing work efficiency. In terms of data security, Baidu Netdisk employs multiple encryption technologies to ensure user data safety and features a recycle bin to prevent accidental file deletion. Intelligent categorization, search capabilities, and batch operations make file management more convenient. For premium users, it also offers accelerated services through its Super Membership. We have integrated Duplicati and have been using Baidu Netdisk as a backup storage solution for over two years, proving its reliability and stability as an efficient and secure data management solution. Baidu Netdisk is more than just a storage tool; it's an ideal data management partner for individual users and enterprise clients.

[trueai-org]

Tencent Cloud Storage was what I used to provide, but now I am offering Baidu Cloud Storage. It has been tested for over two years. If you need keys and an account, please send me an email. Thank you

[trueai-org]

add AliYun Drive, with its large capacity, multi-platform support, convenient sharing, high-level security protection, intelligent management, and unlimited speed transmission, has become an efficient and reliable cloud storage solution.

AliYun Drive, known for being the fastest-growing cloud storage in the market and offering unlimited speed transmission, has emerged as an efficient and leading cloud storage solution.

[gpatel-fr]

Hello

thanks for your contribution. I'm afraid that I can't review it seriously at the moment, not enough time available.
I have browsed it very quickly, and there is a problem with language. While I don't mind that source files for a system used in China contains comments in Chinese, an English translation should be provided if someone not speaking Chinese needs to read your code.

Also, for texts displayed by the software, the convention is that the 'neutral' language is English, and national languages translations should be provided in separate language files (.po files, the Gnu translation system). I'm afraid that currently the nice UI that existed at some time to manage translations is not working anymore, but it's always possible to provide the .po files separately. See this pull request for an example:
#5021

About the test account, it's nice but is there something that scales to more than one developer/tester ? Most hosters are providing test accounts that allow to do a no-strings attached evaluation of the backend, to take an example I have a 5 GB account with OneDrive (Microsoft) and it's more than enough to validate the Duplicati driver. I had tried to do the same with Alibaba, unfortunately they insist to get a phone number, and it's very annoying to have to change phone number if it's spammed by an unscrupulous seller so I have passed on providing support for Alibaba, and Google too for the same reason. If there is something like these test accounts for your provider, it would be ideal.

Finally, I have not looked in detail, but what is specific about OAuth with your provider that the 'normal' OAuth Duplicati support can't do ?

[trueai-org]

Thank you. I have received your questions, and I will optimize and correct them one by one.
Now, my add aliyun oss,
Aliyun Object Storage Service (OSS) is a massive, secure, low-cost, and highly reliable cloud storage service, offering up to 99.995% service availability. It offers a variety of storage types to choose from, comprehensively optimizing storage costs.

[trueai-org]

@gpatel-fr Please give me the contact information, I will send you the test account, this is my email: circle@trueai.org

[trueai-org]

@gpatel-fr Please give me the contact information, I will send you the test account, this is my email: circle@trueai.org

[trueai-org]

@kenkendk @gpatel-fr Hello, Can you apply for the merger as soon as possible? I have sent you an email and explanation before. The Alibaba Cloud team hopes that our products and that piece can be launched. If you have any questions, please contact me as soon as possible. Thanks.

[trueai-org]

Hi, Do you have any questions? Please contact me as soon as possible if you have any questions

[gpatel-fr]

Hi, Do you have any questions?

yes

• could you please avoid doing any unnecessary code reformatting in your changes ? For example, EditUriBuiltins.js. This lead to big diffs to read, and it's not showing the substantial diffs first, diluting them in a mass of reformatting changes. Reformatting the code is fine, but it should be done in separate PR that could be read with different concerns.

• In the case of this file, it's especially annoying because I have problems with these substantials changes: you are hardcoding your Oauth server in generic Duplicati code. I am currently in the process in allowing users to set their own server (instead of the default Duplicati one) in allow for alternate oauth server to work #5099, so this is definitely not going in a direction I love. An open source project should not bind users to specific internet resources. If you want to distribute your own binaries with your own hard coded server, it is fine. But the generic Duplicati project should be Internet resources neutral (even if we are still far from this ideal). Also, these changes are leading in yet more duplicated code (the scope.oauth_start_token_creation function). That's very annoying in the case where there are things to make better or fix.

• about Chinese, I may have made my suggestions not clear enough: what is fine: Chinese comments in your driver specific code, provided that you add English translation. You have added these translations, great. Chinese README version is fine. What is NOT fine are Chinese messages displayed to the user in Duplicati code - even IN YOUR SPECIFIC DRIVER.
  All translations are to be done with the standard Duplicati mechanism (.po files). I know that currently the Duplicati translation server is not used and it is sad - but it is something that I hope to be fixed at some point and when it happens, your direct untranslatable messages will be a pain to fix. So all translations from English in user displayed messages have to be provided in a separate language file. Also, unless I am mistaken, for the messages that you have made translatable (I think), you did not provide a language file change.

• because of all that I have not even begun to properly read your changes (except in this small part of EditUriBuiltins.js). I have limited time to dedicate to your changes.

Finally about your proposal to provide me with a special access, I have already said that I was reticent about that. I am providing support for many providers with the help of standard free accounts - they are limited (a few GB) and not really usable for anything but for testing, but it's enough. What will happen if someone else wants to fix a problem in your driver ? they would have no account I guess ? Will you provide such an access for anyone asking for it to fix Duplicati code ?
Anyway, you asked for my personal mail: there is no need for it. I am active on the Duplicati forum (forum.duplicati.com), my alias is the same as on Github, and the forum allows for private messages. If you want to ask questions privately please copy ts678 and jojo-1000, they are the most active people helping on the forum.

[ts678]

The Alibaba Cloud team hopes that our products and that piece can be launched.

I'm sort of confused. Are you a third party developer, or affiliated with Alibaba and Baidu (which got a big promotional ad here).
From what I can tell, these are separate companies, and I'm trying to figure out what's officially backed and what's independent.

There is already enough trouble with reverse-engineered third party efforts, and providers that dislike software that's not theirs.
Ideally Duplicati has reliable upstream developers to work with if for example a library has bugs. Are the library devs responsive?

Behind those devs should be a provider that approves of such use, for example by providing official APIs and developer support.
Support directly from the Duplicati team (or maybe even via library providers) is somewhat impeded the by language difference.

https://github.com/aliyun/aliyun-oss-csharp-sdk is a version newer than the 2020 version 2.12 that you use, if I read that right.
The alibaba-cloud-csharp-sdk license data looks a little inconsistent (one says Apache, one MIT), but there are a lot of libraries...

Any idea if the libraries have been proven on .NET 8 and beyond? Duplicati is trying to get off .NET Framework, but isn't out yet.

you are hardcoding your Oauth server in generic Duplicati code.

I haven't looked to see how use compares to Duplicati's. It may be different, but using an independent server raises other issues.
I'm not going to try to follow the code further, as I'm not a Duplicati dev. On the other hand, you're deep into our OAuth server.

[gpatel-fr]

@ts678

There is already enough trouble with reverse-engineered third party efforts, and providers that dislike software that's not theirs.
Ideally Duplicati has reliable upstream developers to work with if for example a library has bugs

There is already a driver (Mega) written without official support. All things considered, I'd say that this driver's author would probably be more helpful than Google or Microsoft (for Google Drive or OneDrive) in case of problems for Duplicati. If a driver has real trouble and there is no support from the party causing it, there is always the possibility of dropping it.

[ts678]

If a driver has real trouble and there is no support from the party causing it, there is always the possibility of dropping it.

Alternatively, try to find where things are before putting them in. Dropping things after they're in use is an awkward thing...
Sometimes we can help migration using something that rclone does, but they don't seem to support these providers yet.

I see you're trying to fix Storj (no published API) right now. I'm not sure if we ever got Jottacloud all back after their change.
Fortunately, we had a continuing relationship with the original developer-for-Duplicati, otherwise it would have gone worse.

Regardless, it sounds like you're aware of the issue, but less bothered by it. I'll agree that giant companies can be a pain too.

[gpatel-fr]

@ts678

I see you're trying to fix Storj (no published API)

I'm not sure that it is quite correct, there is a Go library officially supported by the provider and the driver developer interfaces with it. The provider does not support any other language than Go, but this is a supported base.

[ts678]

"no published API" refers to RAW API documentation, which does mention the C and Go versions. Some people reverse engineer whatever a storage provider provides into a language they want. The uplink.NET plan is to ship the binaries, which is not quite as portable as C#, but it does help with receiving fixes. We also have a helpful developer to call on, if that's required, so not too bad.

The provider does not support any other language than Go

Their C library is uplink-c, and appears supported. I'm not sure which one is "better" supported. I looked at both release notes for API issues, because we have an issue author wondering if API changed. I looked for the API and saw it's apparently not published, however that's not so bad because the C and Go libraries are. I don't know what we have coming here for Aliyun and Baidu work.

[trueai-org]

@gpatel-fr @ts678

Thanks!

First of all, I appreciate everyone's suggestions and explanations. I will address the issues mentioned and propose appropriate solutions.

1. Due to our team's long-term use of an older branch of Duplicati, the current submissions are based on this previous branch, which might have led to a significant number of modifications. This could make the merging process appear cumbersome. I will break down the submissions to mitigate this issue.

2. Given the extensive support included in a single merge, such as for Alibaba Cloud Drive, Baidu Netdisk, and Alibaba Cloud OSS, I plan to open a separate PR for each support. This approach should provide more clarity.

3. For services like Alibaba Cloud Drive, Baidu Netdisk, and Alibaba Cloud OSS, which are predominantly used by Chinese users, we have provided separate servers, including https://duplicati.cn/ and https://duplicati.com.cn/, https://duplicati.net/ , to cater to the Chinese audience. These domains are owned by us, and we have communicated via email our intention to become the Chinese site's collaborative developers to offer better services to users in China. Of course, if this approach is not recommended, we will not use separate authorization servers.

4. Regarding our role as a third-party service provider, we applied to Alibaba Cloud for collaboration using Duplicati as our application name. Therefore, the official team at Alibaba Cloud wishes for Duplicati to integrate Alibaba Cloud into the Duplicati application, which is why we are advancing this effort.

5. I will address issues related to Alibaba Cloud Drive, Baidu Netdisk, Alibaba Cloud OSS, Chinese translations, test accounts, agreements, etc., in a new PR and will optimize the code further. If there are still issues, I am open to addressing them.

[gpatel-fr]

@trueai-org

we have provided separate servers

this is not a concern for me. It may be for the original project author, who owns the .com domain, though.

if this approach is not recommended, we will not use separate authorization servers.

you running your own separate authorization server is not a problem at all from my point of view.

Regarding our role as a third-party service provider

I understand that you are a business working with Alibaba. Well, as far as I know, Duplicati license has no problem with getting money for software support so you are welcome to do that.
I think that @ts678 concern is about long term support.
My concern is also to not include any local specific code in Duplicati core (for maintenance reasons), although my point of view differs slightly of @ts678 about driver code - if a provider wants to stop supporting a third party solution, it's their right and if it effectively disables third party support (and NOT only 'not support' as with Mega) the only solution is to drop it as one can't connect to other people servers against their wishes. I think that it is not a very likely thing to happen because ultimately people connecting to provider servers are providers customers, so it is not a huge concern as fighting own customers is not a common occurrence.

For the record, I don't think that you have addressed all my points, but I'm quite willing to wait for you mulling things over as I have lots of other concerns.

[kenkendk]

@trueai-org I think that the support for Baidu and Aliyun storage is very much in line with the core of Duplicati. There is a bit of a divide where services in China are not exposed or at least known in the English-speaking part of the world. And the same goes the other way, so Duplicati does not cater well to a Chinese audience, for example, the main website is in English only. I will have to think a bit about how I think this should be handled, but I see no reason that we should not support multiple languages on the main website.

For the auth part, I think it is problematic if there is a third-party involved in the auth process. This intermediary has the resposibility to stop functionality, leak access and much more. If we really need to use a third party intermediary I think this should be somehow communicated to the users. The screenshot popups appears to be a the OAuth consent dialogs. If this is the case I would prefer if this can be integrated into the C# based OAuth service: https://github.com/duplicati/oauth-server

It would need to have multi-language support of course.

[trueai-org]

hello,

@trueai-org I think that the support for Baidu and Aliyun storage is very much in line with the core of Duplicati. There is a bit of a divide where ....

@kenkendk @gpatel-fr Thank you for your suggestions. Given the network and firewall issues in China, Chinese users often face difficulties accessing smooth internet services. This is why we applied for Chinese domain names to deploy third-party authorization servers independently. Of course, if network issues are not a concern, I will next attempt to prioritize implementation on https://github.com/duplicati/oauth-server.

Regarding Baidu Netdisk and Alibaba Cloud Drive, I will continue to submit new PRs to complete the service provision.

I have already refactored the code and will now provide separate, individual merge PRs for each service. Currently, I have added support for Alibaba Cloud Object Storage Service (OSS) and have provided a test account for evaluation. Of course, you are also welcome to register for an Alibaba Cloud account through their official website for testing purposes.

aliyun oss PR, add test account
#5106