Feature/python model beta #126
Conversation
|
This is now based on #148. Please take a look at it first. |
dbt/adapters/databricks/impl.py
Outdated
| if status != "Finished": | ||
| raise dbt.exceptions.RuntimeException( | ||
| "python model run ended in state" | ||
| f"{status} with state_message\n{json_resp['results']['data']}" |
There was a problem hiding this comment.
I noticed that if the python code actually failed, status will still be finish, but json_resp['results']['resultType'] will be error, and json_resp['results']['cause'] will contain traceback. I reused your code for jobs API and pushed the changes here. Feel free to review and let me know if any adjustment needed!
There was a problem hiding this comment.
Thanks! I'll address it.
| password=token, | ||
| host=f"https://{host}", | ||
| api_version="1.2", | ||
| command_name=command_name, |
There was a problem hiding this comment.
Is this command_name being recorded somewhere after the run?
There was a problem hiding this comment.
It will be part of user agent in the API request.
There was a problem hiding this comment.
@ChenyuLInx @jtcohen6 btw, is it possible to set a simple string in the user agent of API requests of dbt-spark to identify the API calls are from dbt similar to ODBC connections?
There was a problem hiding this comment.
@ChenyuLInx I opened an issue in dbt-spark dbt-labs/dbt-spark#460. Thanks.
|
https://github.com/databricks/jaffle_shop/pull/1/files#r969108078 |
Yes, so far it's a limitation of python model. |
|
@ueshin Since it's a known issue, then I think this PR works well. I don't find other issue right now |
|
Thanks! merging. |
Description
Ports the changes for python model v1 from
dbt-sparkbut use APIs below instead.