Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add a convention for loading yaml config files from the classpath #6

Merged
merged 6 commits into from May 10, 2019

Conversation

c-horn
Copy link
Collaborator

@c-horn c-horn commented May 2, 2019

Instead of explicitly including the config file with spark-submit like:

spark-submit --file config_file.yaml data-validator.jar --config config_file.yaml

If the yaml file is already on Spark's driver classpath (for instance in a jar compiled from a resources/ directory included in a package added with --packages or --jars), you can reference it:

  --config classpath:/config_file.yaml

Copy link
Collaborator

@dougb dougb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add at 2 tests for reading config off of classpath (success and failure), and update README to document new behavior.

@c-horn
Copy link
Collaborator Author

c-horn commented May 2, 2019

@dougb ^

@dougb
Copy link
Collaborator

dougb commented May 6, 2019

I asked @c-horn to make one more change here to add some text about the classpath config feature.

Copy link
Collaborator

@dougb dougb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@colindean
Copy link
Collaborator

I don't have merge perms on this repo (yet). Can you add me or merge it, @dougb or @phpisciuneri?

@phpisciuneri
Copy link
Contributor

@colindean we were trying to get the CI sorted out. Sorry for the delay.

@dougb @c-horn merging branch master into this PR didn't seem to trigger any action. The status check is still waiting for compile data-validator action. I am just going to merge. We can sort this out with a fresh PR.

@phpisciuneri phpisciuneri merged commit b60e5a0 into target:master May 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants