Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Streaming data transfer and python integration #6185

Merged
merged 323 commits into from Dec 10, 2019
Merged

Streaming data transfer and python integration #6185

merged 323 commits into from Dec 10, 2019

Conversation

chaokunyang
Copy link
Member

@chaokunyang chaokunyang commented Nov 18, 2019 •
edited

Why are these changes needed?

This is the minimal implementation of streaming data transfer mentioned in doc, consisting of three parts:

  • writer/reader, implemented with C++ to transfer data between streaming workers
  • streaming queue, the transport layer based on Ray’s direct actor call and C++ Core Worker APIs
  • adaption layer for python, implemented with cython to adapt writer/reader for python

To integrate python with streaming c++ data transfer, following changes are made:

  • We moved python code from python/ray/experimental/streaming/ to streaming/python/ray/streaming,
    and soft link to python/ray/streaming just like rllib.
  • We removed batched_queue and added cython based streaming queue implementation.
  • We moved execution graph related logic from Environment into ExecutionGraph.
  • We refactored operator_instance into a processor, and added a JobWorker actor to execute processors.

The java part will be submitted in following PRs.

Related issue number

#6184

Checks

chaokunyang and others added 30 commits November 7, 2019 14:22
@chaokunyang
refactor streaming python to use queue for communication
50a7daa
@chaokunyang
use a separate thread to process task
d0db596
@chaokunyang
Merge branch 'streaming_data_transfer' of https://github.com/ant-tech…
5320bb0 
…-alliance/ray into streaming_data_transfer
@chaokunyang
return None only when all input channels of task return None
ada98d1
@chaokunyang
fix streaming java bazel gen_maven_deps
9d7c086
@chaokunyang
ignore streaming native lib and generated files
27da3a1
@ashione
adjust comments style
9f6c2c3
@chaokunyang
add flatbuffer deps and fix generated java file package
fb5b15b
@zhijunfu
fix crash with testPassActorAsParameter where the callback for the sa…
de4197f 
…me actor ID are invoked twice
@chaokunyang
cp native_dependencies to jar as resources
f19d336
@chaokunyang
Merge branch 'streaming_data_transfer' of https://github.com/ant-tech…
d5108c8 
…-alliance/ray into streaming_data_transfer
Merge branch 'fix-java-build' into streaming_data_transfer
076f848
@ashione
lint
e9a6234
@chaokunyang
load resource file as native lib in java by create tmp file
c2cd3cd
@chaokunyang
Merge branch 'streaming_data_transfer' of https://github.com/ant-tech…
64fe401 
…-alliance/ray into streaming_data_transfer
@chaokunyang
move streaming runtime to runtime package
859d035
@chaokunyang
fix streaming ignore
d7f8a53
@chaokunyang
fix package
b2985e1
@chaokunyang
remove batched_queue
0c04144
@chaokunyang
load native library when needed
c587146
@ashione
transfer consumed tests
4b7ca7e
@chaokunyang
rename jni files name
9afb720
@chaokunyang
Merge branch 'streaming_data_transfer' of https://github.com/ant-tech…
0c1a97e 
…-alliance/ray into streaming_data_transfer
fix jni function name
9d771f5
fix jni compile
cb8edc7
@chaokunyang
streaming cython implementation
965195a
@chaokunyang
Merge branch 'streaming_data_transfer' of https://github.com/ant-tech…
d4ca30f 
…-alliance/ray into streaming_data_transfer
@chaokunyang
add CoreWorker in pxd to expose corker_worker ptr to cython
9d29e81
java worker test
271ed33
fix
079e0d2
@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/19311/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/19312/
Test FAILed.

@chaokunyang
Merge remote-tracking branch 'ray/master' into streaming_data_transfe…
79741a7 
…r_python

# Conflicts:
#	doc/site/blog.html
#	doc/site/community.html
#	doc/site/get_ray.html
#	doc/site/index.html
@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/19325/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/19329/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/19358/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/19359/
Test FAILed.

raulchen
raulchen approved these changes Dec 10, 2019
View reviewed changes
Copy link
Contributor

@raulchen raulchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM

src/ray/ray_exported_symbols.lds Outdated Show resolved Hide resolved
BUILD.bazel Outdated Show resolved Hide resolved
BUILD.bazel Outdated Show resolved Hide resolved
@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/19387/
Test PASSed.

@raulchen raulchen merged commit 6272907 into ray-project:master Dec 10, 2019
@raulchen raulchen deleted the streaming_data_transfer_python branch December 10, 2019 12:33
@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/19391/
Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/19389/
Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/19393/
Test PASSed.

@mehrdadn mehrdadn mentioned this pull request Dec 11, 2019
Merged
3 tasks
@chaokunyang chaokunyang mentioned this pull request Dec 13, 2019
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ericl ericl ericl left review comments

@MissiontoMars MissiontoMars MissiontoMars left review comments

@ashione ashione ashione left review comments

@zhijunfu zhijunfu zhijunfu left review comments

@raulchen raulchen raulchen approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
10 participants
@chaokunyang @AmplabJenkins @ashione @ericl @raulchen @MissiontoMars @zhijunfu @kfstorm @deanwampler @visatish