Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Spark-based Integration Test #76

Closed
KenSuenobu opened this issue May 28, 2018 · 2 comments
Closed

Add Spark-based Integration Test #76

KenSuenobu opened this issue May 28, 2018 · 2 comments
Assignees
@KenSuenobu
Labels
difficult Feature that may be difficult to implement distribution Distributed architecture work documentation General documentation enhancement New feature or request help wanted Extra attention is needed testing Testing for feature complete
Milestone
0.2.0

Comments

@KenSuenobu
Copy link
Owner

Add a Spark integration test to the codebase. Need to create a Workflow in CircleCI so that it starts up an instance of Spark (via Docker) in a separate workflow, then the integration tests submits jobs to the local Spark instance, waiting for jobs to finish.
@KenSuenobu KenSuenobu added enhancement New feature or request documentation General documentation testing Testing for feature complete difficult Feature that may be difficult to implement distribution Distributed architecture work labels May 28, 2018
@KenSuenobu KenSuenobu added this to the 0.2.0 milestone May 28, 2018
@KenSuenobu KenSuenobu added the help wanted Extra attention is needed label May 28, 2018
@KenSuenobu
Copy link
Owner Author

Ugh, I forgot how much of a pain this can be in testing. I have to copy over the entire Scattersphere Jar (fat jar) to the Spark server when I submit a job. This way, it can be run as a test suite. More to do!

@KenSuenobu KenSuenobu self-assigned this Jun 7, 2018
@KenSuenobu KenSuenobu modified the milestones: 0.2.0, 0.2.1 Jun 9, 2018
@KenSuenobu
Copy link
Owner Author

Working. Uses the fat jar to submit jobs to Spark, is working perfectly now with integration tests with the real world code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@KenSuenobu KenSuenobu

Labels
difficult Feature that may be difficult to implement distribution Distributed architecture work documentation General documentation enhancement New feature or request help wanted Extra attention is needed testing Testing for feature complete
Projects
None yet
Milestone
0.2.0
Development

No branches or pull requests
1 participant
@KenSuenobu