[BEAM-11415] In-progress Go ValidatesRunner framework. #13503
Conversation
lostluck
left a comment
lostluck
left a comment
There was a problem hiding this comment.
LGTM. It's pretty clean as is. Looking forward to the developments we've discussed.
sdks/go/test/build.gradle
Outdated
There was a problem hiding this comment.
For now, could we use flinkXlangValidatesRunner which at least covers what is in particularly new coverage about this set up. This comment driven by seeing too many vestigial "new" implementations that don't get fully migrated.
Tangent, it's generally preferable to rename older implementations to "old" than new ones to "new" when possible.
Feel free to disregard if you've already gotten far enough in the consolidation in later commits.
There was a problem hiding this comment.
I'm alright with that, done.
There was a problem hiding this comment.
Deduplicate w/the socket script higher up please.
This commit includes everything I've worked on in the Go ValidatesRunner framework so far, for initial review. This creates a new subdirectory, test/validatesrunner, that is intended to just contain a bunch of tests that run and validate lightweight pipelines. It also contains a script that handles starting job servers and expansion services, similar to run_integration_tests.sh, and one initially tested gradle build rule for running it with Flink. Running with the direct runner also works fine, with the command "go test validatesrunner/... --runner=direct". Currently, the script has only been confirmed to work with Flink and the direct runner, but it should theoretically work with the Portable runner and Spark as well, once appropriate gradle build rules are figured out. Most of the contents here are likely to be transient, chances are this will be merged to be an update to the existing integration directory rather than a whole new directory, and will just replace the existing driver approach to the tests. However, this is still valuable to commit as a starting point to the work.
youngoli
force-pushed
the
validatesrunner2
branch
from
c39910d to
e8a9c7b
Compare
2 participants
This PR includes everything I've worked on in the Go ValidatesRunner framework so far, for initial review. Note that it includes the contents of PR #13502 as the first commit since it's a required for this code to work, but that should be reviewed in its own PR. This PR only covers the second commit.
This PR creates a new subdirectory, test/validatesrunner, that is intended to just contain a bunch of tests that run and validate lightweight pipelines. It also contains a script that handles starting job servers and expansion services, similar to run_integration_tests.sh, and one initially tested gradle build rule for running it with Flink. Running with the direct runner also works fine, with the command "go test validatesrunner/... --runner=direct". Currently, the script has only been confirmed to work with Flink and the direct runner, but it should theoretically work with the Portable runner and Spark as well, once appropriate gradle build rules are figured out.
Most of the contents here are likely to be transient, chances are this will be merged to be an update to the existing integration directory rather than a whole new directory, and will just replace the existing driver approach to the tests. However, this is still valuable to commit as a starting point to the work.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.