Add support for HuggingFace to httpfs

[samansmink]

This PR adds native support for HuggingFace urls to query hugging face datasets directly from DuckDB.

Features

• hf:// url format support (including specifying the branch)
• hf:// url globbing
• HUGGINGFACE secret type
  • manually provided token
  • fetched automatically from ~/.cache/huggingface/token
• autoload httpfs extension on:
  • huggingface secret creation
  • query any hf:// url
• supports pagination
• supports duckdb's retry mechanism

How to use

(optionally) load credentials using a token

CREATE SECRET hf1 (TYPE HUGGINGFACE, TOKEN 'hf_my_very_secret_token');

(optionally) load credentials using ~/.cache/huggingface/token

CREATE SECRET hf2 (TYPE HUGGINGFACE, PROVIDER cache);

Now query some data (if secret was created, this can be private data)

SELECT * FROM parquet_scan('hf://datasets/me/my_data/some_file/test.parquet')
SELECT * FROM parquet_scan('hf://datasets/me/my_data/some_file/with/a/glob/**/t[eab]st.parquet')

To query from custom branches:

SELECT * FROM parquet_scan('hf://datasets/me@my-branch/my_data/some_file/test.parquet')
SELECT * FROM parquet_scan('hf://datasets/me@~parquet/my_data/some_file/test.parquet')

Testing

Everything is tested that was added, but it is a bit rudimentary still and could use improvement: I added a test that queries some dataset in my huggingface account with my token. This should be improved by moving a token for a test account into DuckDB CI then creating a dedicated job for this.

Also I tested the pagination by exposing a limit param manually. This forces HF's API to return the list query results one by one

Future work

• fully implement generic bearer token support and add tests for this

[samansmink]

hey @lhoestq @severo @Wauplin, I have a first version working in this PR, happy to take any feedback here!

[lhoestq]

This looks amazing :) Nothing to add to what @severo and @Wauplin said except minor comments

[Mytherin]

Looks good from my side

[samansmink]

I've processed all comments, If CI passes this is good to go from my side.

Thanks for all the reviews!

[Mytherin]

Can you solve the merge conflict?

[Mytherin]

Thanks!

[severo]

Thanks for supporting hf://! What are the next steps regarding the docs and the release?

[AndreaFrancis]

Hi @samansmink, I loved the native HF implementation and am preparing some documents to share on the Dataset Viewer page: https://huggingface.co/docs/datasets-server/duckdb (still in progress).
I was trying the credential_chain provider, but it seems to have been an issue the first time, or maybe I am using it incorrectly.

These are the steps to reproduce:

1. Access to duckdb
2. Ensure that there is a configured token in ~/.cache/huggingface/token or HF_TOKEN is set
3. Try to create the secret using credential_chain
   Invalid Input Error: Secret provider 'credential_chain' not found for type 'huggingface'
4. Now, try to execute another HF operation like CREATE SECRET hf_token (TYPE HUGGINGFACE, TOKEN 'your_hf_token'); or try to get some files using the hf httpfs
5. Try to create the secret using credential_chain again -> OK

See the following example:

It works after another httpfs usage for HF. I'm not sure if this is expected or not, but it might lead to confusion for users trying to set the token the very first time.

[samansmink]

@AndreaFrancis ah good catch, PR is up here #12112