Stop requestslib from overriding auth headers with data from .netrc file

[susodapop]

Resolves #337

Description

This pull request refactors the credentials handling in python_submissions.py using the workaround suggested in #337. The desired behaviour from this change is not obvious when looking only at the code diff. However the idea here is to explicitly order Python requests to use Bearer authentication with an explicitly provided token. If this is not done explicitly, there is an edge case where a local .netrc file can override an auth token present in profiles.yml.

There's no automated test for this because I'm not sure how to deposit a .netrc file in a test environment.

I reproduced the issue manually by adding this to ~/.netrc on my workstation:

machine <my-workspace>.cloud.databricks.com
login token
password <expired_token>

I removed the skip marker on test_python_uc_databricks_sql_endpoint and ran this integration test on the main branch. The test failed with this exception:

ERROR    stdout_log:eventmgr.py:66 �[0m18:58:04     b'{"error_code":"403","message":"Invalid access token."}'
ERROR    file_log:eventmgr.py:66 �[0m13:58:04.325701 [error] [MainThread]:    b'{"error_code":"403","message":"Invalid access token."}'

Then I applied the change in this pull request and re-ran the test. It passed.

Checklist

• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

[andrefurlan-db]

it needs to work with oauth as well. Token is not passed in config

[susodapop]

I've updated the code to use the HeaderFactory provided by databricks.sdk and verified it still works with PAT auth. Need to do a quick test with OAuth and maybe basic auth to make sure those cases are also handled.

[susodapop]

I found that the same issue we're fixing here (.netrc overrides auth headers in requests made by dbt-databricks) also affects our databricks-sdk-py repo (.netrc overrides auth headers in requests made for oauth handshake).

The fix in this PR will fix dbt-databricks .netrc woes if the auth method is PAT. But it will only fix the issue for oauth use-cases after databricks-sdk-py is updated with this patch databricks/databricks-sdk-py#107 and we update the dbt-databricks dependency to incorporate that fix.

Note: I found the same bug in databricks-sql-connector's custom implementation of OAuth handshakes but this doesn't affect dbt-databricks because the dbt adapter uses the OAuth implementation in the SDK.

[susodapop]

Now that databricks-sdk has pushed the parallel fix for oauth I've rebased on main. This included a refactor of SessionCredentials base on a breaking change in databricks-sdk.

[susodapop]

Looks like there is now a parallel PR that bumps the databricks-sdk version and includes the RefreshableCredentials refactor: #349

[susodapop]

Alright, I've pushed the final version of this code. I moved the BearerAuth class into connections.py because we're about to use it there to fix an issue with streaming tables / materialized views.

I also simplified the code by setting the Session.auth property rather than setting the .auth property for every single network request.