Skip to content
This repository has been archived by the owner on Jan 4, 2022. It is now read-only.

Problem running dbx execute with example #28

Closed
dexianta opened this issue Oct 23, 2020 · 3 comments
Closed

Problem running dbx execute with example #28

dexianta opened this issue Oct 23, 2020 · 3 comments

Comments

@dexianta
Copy link
Contributor

dexianta commented Oct 23, 2020

I'm not able to run dbx execute --cluster-name my-cluster --job job-sample. As far as I understand, in dbx execute, wheel was built, copied to dbfs, and jobs/sample/entrypoint.py was serialized to directly run on the interactive cluster, without providing --conf-file. However the Job class defined in project_name/common.py requires --conf-file, thus exception were thrown
image

========================================
So far, it seems to me that:

  • dbx execute is suitable to launch script without external config files, since it's using 1.2 API command execute.
  • if I want to run the local code on interactive cluster (for fast dev iteration), I'll need a separate job json with "exisiting_cluster_id". then run launch (which is /job/run-now underneath)
@renardeinside
Copy link
Contributor

renardeinside commented Oct 25, 2020

dbx execute is suitable to launch script without external config files, since it's using 1.2 API command execute.

Yes, you're correct in this understanding.

if I want to run the local code on interactive cluster (for fast dev iteration), I'll need a separate job json with "exisiting_cluster_id". then run launch (which is /job/run-now underneath)

I'll provide the functionality to add a default_conf method, which allows you to provide a default configuration inside code for dbx execute-based dev.

@renardeinside
Copy link
Contributor

@dexianta please check the latest version and especially the init_adapter method.

@dexianta
Copy link
Contributor Author

@dexianta please check the latest version and especially the init_adapter method.

@renardeinside thank you very much

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants