Feat: Implement hf:// / "hugging face" integration in datafusion-cli

[xinlifoobar]

Which issue does this PR close?

Closes #10720

Rationale for this change

What changes are included in this PR?

Are these changes tested?

# xinli @ arch-dev in ~/source/repos/datafusion/datafusion-cli on git:dev/xinli/hfstore o [12:16:09] 
$ ./target/debug/datafusion-cli
DataFusion CLI v38.0.0
> SELECT count(*) AS count
FROM 'hf://datasets/cais/mmlu/astronomy/dev-00000-of-00001.parquet';
+-------+
| count |
+-------+
| 5     |
+-------+
1 row(s) fetched. 
Elapsed 2.469 seconds.

> create external table test stored as parquet location "hf://datasets/cais/mmlu/astronomy/";
0 row(s) fetched. 
Elapsed 1.398 seconds.

> select count(*) from test;
+----------+
| COUNT(*) |
+----------+
| 173      |
+----------+
1 row(s) fetched. 
Elapsed 1.199 seconds.

> select * from test limit 2
;
+--------------------------------------------------------------------------------------+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+
| question                                                                             | subject   | choices                                                                                                                                                                                                                                                                                                                          | answer |
+--------------------------------------------------------------------------------------+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+
| You cool a blackbody to half its original temperature. How does its spectrum change? | astronomy | [Power emitted is 1/16 times as high; peak emission wavelength is 1/2 as long., Power emitted is 1/4 times as high; peak emission wavelength is 2 times longer., Power emitted is 1/4 times as high; peak emission wavelength is 1/2 as long., Power emitted is 1/16 times as high; peak emission wavelength is 2 times longer.] | 3      |
| What drives differentiation?                                                         | astronomy | [Spontaneous emission from radioactive atoms., The minimization of gravitational potential energy., Thermally induced collisions., Plate tectonics.]                                                                                                                                                                             | 1      |
+--------------------------------------------------------------------------------------+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+
2 row(s) fetched. 
Elapsed 1.098 seconds.

Are there any user-facing changes?

[xinlifoobar]

Hi @alamb. I am still working on this for completing UTs, and E2Es and fixing bad code styles... Could you please help do a pre-quick review of the ideas behind... I did not find a simple way to implement this other than creating a wrapper ObjectStore impl on top of HttpStore.

[edmondop]

This is awesome, can you add a section here ? https://github.com/apache/datafusion/blob/main/docs/source/user-guide/cli/datasources.md

[xinlifoobar]

Hi @alamb. I am still working on this for completing UTs, and E2Es and fixing bad code styles... Could you please help do a pre-quick review of the ideas behind... I did not find a simple way to implement this other than creating a wrapper ObjectStore impl on top of HttpStore.

This is now able to be reviewed now. I completed most of the refining of the initial code.

[xinlifoobar]

This is currently not working due to:

1. The hugging face user access token is case-sensitive.
2. a previous change enforces every option value in lower case. https://github.com/apache/datafusion/pull/9723/files.

I will figure out the history to see whether this will be feasible.

[alamb]

I am sorry @xinlifoobar for the delayed review -- I am traveling this week (actually presenting about DataFusion at SIGMOD: https://2024.sigmod.org/industrial-list.shtml)

[xinlifoobar]

I am sorry @xinlifoobar for the delayed review -- I am traveling this week (actually presenting about DataFusion at SIGMOD: https://2024.sigmod.org/industrial-list.shtml)

Ya, the first time I found this on my timeline on LinkedIn, and am glad to be part of this awesome project.

I would pause updating on this PR since it is extremely large IMO and difficult for reviewers. Let me know your thoughts on it and I could do an update in the following iterations.

[alamb]

I would pause updating on this PR since it is extremely large IMO and difficult for reviewers. Let me know your thoughts on it and I could do an update in the following iterations.

If it is too complicated, maybe we should just stop working on it (or maybe we should put the code into a datafusion-contrib repo 🤔 )

[alamb]

😮 -- tests too!

[xinlifoobar]

I saw duckdb build their organizations and datasets, instead of depending on the random existing ones, to make their CI safe. It would be too early for this PR to do so... It is still in an earlier stage.

[xinlifoobar]

I would pause updating on this PR since it is extremely large IMO and difficult for reviewers. Let me know your thoughts on it and I could do an update in the following iterations.

If it is too complicated, maybe we should just stop working on it (or maybe we should put the code into a datafusion-contrib repo 🤔 )

Ya. re-implementing the datastore and associated facilities is code-consuming. Do you think the objectstore solution is the right way to go? If so, I could split part of the code into datafusion-contrib repo.

[alamb]

Sorry for the delay -- I paln to review this tomorrow

[xinlifoobar]

Sorry for the delay -- I paln to review this tomorrow

Sorry, I lost Github connections for a couple of days and just returned. Also Thanks. please take your time.

[alamb]

This is still on my list, but I am behind in my reviews