Initial Ray runner prototype based on FnApiRunner in Beam #10
Conversation
| @@ -84,6 +85,18 @@ def contains_labels(mi, labels): | |||
| return contains_labels(mi, labels) and mi.urn == urn | |||
|
|
|||
|
|
|||
| class RayFnRunnerTest(fn_runner_test.FnApiRunnerTest): | |||
There was a problem hiding this comment.
Awesome work - thanks Pablo!
It may be a good idea to start capturing some of the TODOs and comments here as issues for others to start picking up.
I was able to get the Ray Runner tests passing with one change related to WatermarkManager initialization. Can you verify whether or not this is expected? I just had a couple other minor comments, but otherwise think this looks ready to merge!
| "ray[data]", "apache_beam" | ||
| ], | ||
| extras_require={ | ||
| 'test': TEST_REQUIREMENTS, |
There was a problem hiding this comment.
Thanks for separating out the test requirements as extras! Can we also add them to requirements_dev.txt with a comment noting that they are only required for test execution? Ideally, I think our install_requires dependencies should also be mirrored in requirements_dev.txt for ease of developer dependency management.
|
LGTM. Pablo, can we have a demo on this later sometime? |
|
thanks Patrick, everyone. I'll try to create new work items for this. @wilsonwang371 I'll be happy to do a demo on this. |
This prototype is hacked together as quickly as possible. I can't speak for its efficiency, and its code cleanliness.
Next steps:
ray_runner_test.pyOthers...