Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revisit zingg.sh #260

Closed
navinrathore opened this issue May 16, 2022 · 3 comments
Closed

Revisit zingg.sh #260

navinrathore opened this issue May 16, 2022 · 3 comments
Assignees
Milestone

Comments

@navinrathore
Copy link
Contributor

Description
"zing.conf" file should include other zingg variables as well. It should be central point for defining zingg configuration.
zingg.sh should be changed accordingly

Ref: PR #253, issue #15

@sonalgoyal sonalgoyal added this to the 0.3.4 milestone May 16, 2022
@sonalgoyal
Copy link
Member

There are a few other changes we now need in the script to run the python files. The plan is to have some new phases(assessTraining for eg) run completely through python, and existing ones through java. So, we need to parse the args, check if the phase is a python supported phase, and invoke corresponding phaseName.py file through spark-submit.

To parse the phase, please check https://stackoverflow.com/questions/192249/how-do-i-parse-command-line-arguments-in-bash

To invoke spark-submit with python files, please check https://spark.apache.org/docs/2.4.3/submitting-applications.html. In this case, we will not be calling --main-class. We may also need to send --py-files. You can test with the example python api we have in the repo.

@sonalgoyal
Copy link
Member

@navinrathore please check #243 (comment)

@navinrathore
Copy link
Contributor Author

As of now, spark configuration can be read from the zingg.conf file. Through the same file, environment variables could also be defined.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants