feat(core): service add HuggingFace file system

[morristai]

Description

close: #3605

Addition Note

• HuggingFace doesn't host official HTTP API docs. Detailed HTTP request API information can be found on the HuggingFace Hub's source code.
• For message.rs, I'm unsure whether we should drop unused fields or add #[allow(dead_code)] to disable warnings.

[Wauplin]

Hey there! I'm @Wauplin, maintainer of huggingface_hub -the Python client library for the Hugging Face Hub-. This is a very cool integration! I quickly had a look and added a few high-level comments about the existing huggingface ecosystem (to better align the naming conventions).

I also wanted to point you to https://github.com/huggingface/hf-hub which is the rust client to interact with the Hub. Though it's not as complete as the Python equivalent, it already has support for most file read/write use cases -including compatibility with the HF cache system- . It would be good to check what's there to avoid duplicating logic across libraries. cc @Narsil who worked on this

[Xuanwo]

I also wanted to point you to https://github.com/huggingface/hf-hub which is the rust client to interact with the Hub. Though it's not as complete as the Python equivalent, it already has support for most file read/write use cases -including compatibility with the HF cache system- .

Thank you for the advice, hf-hub is an excellent project! However, OpenDAL prefers to implement all APIs using its own HttpClient to ensure full control, eliminating the need to interact with various SDKs and keep it's deps tree as low as possible.

[Wauplin]

Thank you for the advice, hf-hub is an excellent project! However, OpenDAL prefers to implement all APIs using its own HttpClient to ensure full control, eliminating the need to interact with various SDKs and keep it's deps tree as low as possible.

Fair enough @Xuanwo! I just wanted to make sure you know about it :)

[Xuanwo]

I just wanted to make sure you know about it :)

You are so nice! And thanks for the review. ❤️

[morristai]

Seems not fixed. With ConfigDeserializer, we don't need to use write map.get("abc") by hand.

(It appears that I am unable to reply in that review block, so I'll respond here.)
@Xuanwo, Sorry, it's my negligence. But I want to point out that I'm referring to service Etcd. It seems like it's also miss refactored?
https://github.com/apache/incubator-opendal/blob/0711e6ce97660380628b72a010d092fd29ec9553/core/src/services/etcd/backend.rs#L200

[Xuanwo]

Sorry, it's my negligence. But I want to point out that I'm referring to service Etcd. It seems like it's also miss refactored?

Yes, there is an on-going effort to refactor them one by one: #3240

[morristai]

Yes, there is an on-going effort to refactor them one by one: #3240

I believe there might be a misunderstanding. After checking the refactoring progress list and the pull request code, it appears that Etcd has already been refactored, but the from_map part seems to have been overlooked. Could you please verify this from your end? I'm not certain if I've missed something here. (By the way, I am currently on a business trip, so my response may be delayed.)

[Xuanwo]

I believe there might be a misunderstanding. After checking the refactoring progress list and the pull request code, it appears that Etcd has already been refactored, but the from_map part seems to have been overlooked. Could you please verify this from your end?

Seems etcd missed this part, we need to fix it, tracked at #3702

By the way, I am currently on a business trip, so my response may be delayed

Enjoy your trip!

[Xuanwo]

Thanks!

[Xuanwo]

I have setup a repo at https://huggingface.co/datasets/opendal/huggingface-testdata

Would you like to add a behaivor test for this?

[morristai]

I have setup a repo at https://huggingface.co/datasets/opendal/huggingface-testdata

Would you like to add a behaivor test for this?

Yes, I'd love to work on that.