dbt can't connect to Kyuubi/Spark on EKS using Thrift via LDAP (username/password)

[bdkcode]

Describe the bug

dbt-spark currently doesn't allow connecting to a Kyuubi/Spark on EKS using Thrift via LDAP. It fails with the following error on dbt debug

17:39 $ dbt debug
01:39:34  Running with dbt=1.0.0
dbt version: 1.0.0
python version: 3.8.0
python path: /Users/deepakk/.pyenv/versions/3.8.0/bin/python3.8
os info: macOS-10.15.5-x86_64-i386-64bit
Using profiles.yml file at /Users/deepakk/.dbt/profiles.yml
Using dbt_project.yml file at /Users/deepakk/PycharmProjects/dbt_spark_poc/dbt_project.yml

Configuration:
  profiles.yml file [OK found and valid]
  dbt_project.yml file [OK found and valid]

Required dependencies:
 - git [OK found]

Connection:
  host: host_redacted
  port: 443
  cluster: None
  endpoint: None
  schema: default
  organization: 0
  Connection test: [ERROR]

1 check failed:
dbt was unable to connect to the specified database.
The database returned the following error:

  >Runtime Error
  Database Error
    failed to connect

Steps To Reproduce

The following profile in ~/.dbt/profiles.yml fails to connect to the Spark server

dbt_spark_poc_stage:
  outputs:
    dev:
      type: spark
      host: redacted
      method: thrift
      user: redacted
      password: redacted
      port: 443
      schema: default
      threads: 1
  target: dev

whereas the same information passed in as jdbc:hive://host_redacted:443/default (with username/password) via DataGrip works.

This behavior can be traced down to this code here: https://github.com/dbt-labs/dbt-spark/blob/main/dbt/adapters/spark/connections.py#L376 which basically ignores the password passed in through profiles.yml.

I was able to connect to the Spark server directly from PyHive using:

    cursor = hive.connect(host='host', database='test_db', username='redacted', auth='LDAP', password='redacted', port=443).cursor()
    cursor.execute('show databases')
    print(cursor.fetchone())
    print(cursor.fetchall())

Expected behavior

When the password is present with auth: LDAP, use those credentials to connect to the Spark server, and return that connection instead.

System information

The output of dbt --version:

07:18 $ dbt --version
installed version: 1.0.0
   latest version: 1.0.0

Up to date!

Plugins:
  - spark: 1.0.0
  - spark: 1.0.0

The operating system you're using:
macOS Catalina 10.15.5

The output of python --version:
Python 3.8.0

Additional context

I have a fix for this locally, please suggest how to proceed with the next steps.

[pixie79]

Did you have the code / fix for this - it looks like Kyuubi could be a good fit for our needs using on EKS

[VersusFacit]

@bdkcode Heya bdk! Appreciate the detailed writeup. You said you had a locally working piece of code? Want help getting that merged? Let me know. We could get a PR with a test or two and a changelog entry :)

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.