[Data] Fix Ray databricks UC reader with dynamic Databricks notebook scope token

[WeichenXu123]

Why are these changes needed?

Set Ray databricks UC reader token by header Authorization: Bearer instead of requests.post auth param.

This fixes bugs on databricks dynamic token.

Testing code:

os.environ["DATABRICKS_TOKEN"] = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiToken().get()

import ray

ds = ray.data.read_databricks_tables(
  warehouse_id='...',
  catalog='hive_metastore',
  schema='test_sb_dr',
  query='select * from students limit 10',
)

ds.to_pandas()

Result:

Before:

After:

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[raulchen]

@WeichenXu123 @c21 do we have a test for this reader?

[WeichenXu123]

@WeichenXu123 @c21 do we have a test for this reader?

Yes, see test_databricks_uc_datasource test function.

[WeichenXu123]

The PR is ready for reviewing. @raulchen

[WeichenXu123]

CC @c21

[c21]

LGTM