New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Streaming data transfer and python integration #6185
Merged
raulchen
merged 323 commits into
ray-project:master
from
alipay:streaming_data_transfer_python
Dec 10, 2019
Merged
Changes from all commits
Commits
Show all changes
323 commits
Select commit
Hold shift + click to select a range
50a7daa
refactor streaming python to use queue for communication
chaokunyang d0db596
use a separate thread to process task
chaokunyang 5320bb0
Merge branch 'streaming_data_transfer' of https://github.com/ant-tech…
chaokunyang ada98d1
return None only when all input channels of task return None
chaokunyang 9d7c086
fix streaming java bazel gen_maven_deps
chaokunyang 27da3a1
ignore streaming native lib and generated files
chaokunyang 9f6c2c3
adjust comments style
ashione fb5b15b
add flatbuffer deps and fix generated java file package
chaokunyang de4197f
fix crash with testPassActorAsParameter where the callback for the sa…
zhijunfu f19d336
cp native_dependencies to jar as resources
chaokunyang d5108c8
Merge branch 'streaming_data_transfer' of https://github.com/ant-tech…
chaokunyang 076f848
Merge branch 'fix-java-build' into streaming_data_transfer
e9a6234
lint
ashione c2cd3cd
load resource file as native lib in java by create tmp file
chaokunyang 64fe401
Merge branch 'streaming_data_transfer' of https://github.com/ant-tech…
chaokunyang 859d035
move streaming runtime to runtime package
chaokunyang d7f8a53
fix streaming ignore
chaokunyang b2985e1
fix package
chaokunyang 0c04144
remove batched_queue
chaokunyang c587146
load native library when needed
chaokunyang 4b7ca7e
transfer consumed tests
ashione 9afb720
rename jni files name
chaokunyang 0c1a97e
Merge branch 'streaming_data_transfer' of https://github.com/ant-tech…
chaokunyang 9d771f5
fix jni function name
cb8edc7
fix jni compile
965195a
streaming cython implementation
chaokunyang d4ca30f
Merge branch 'streaming_data_transfer' of https://github.com/ant-tech…
chaokunyang 9d29e81
add CoreWorker in pxd to expose corker_worker ptr to cython
chaokunyang 271ed33
java worker test
079e0d2
fix
2d9bbe4
fix test
3908300
add _raylet.pxd to share Extension types
chaokunyang 049980f
include streaming cython into raylet
chaokunyang 5056090
streaming python flatbuffer generated files rule
chaokunyang c0ad460
Merge branch 'streaming_data_transfer' of https://github.com/ant-tech…
chaokunyang 09f8edf
queue consumer implementation
chaokunyang 24dcdff
refactor execution graph build
chaokunyang ec6a552
refactor DataChannel to get actor id in DataInput/DataOutput
chaokunyang fe8957d
fix cyclic reference serialization error
chaokunyang 7302267
fix serialization
chaokunyang 64c1f87
share ActorID definition
chaokunyang d206909
fix java wordcount
6a28767
Merge branch 'streaming_data_transfer' of github.com:ant-tech-allianc…
526e488
fix build.sh
600ecbe
adapt native queue to python
chaokunyang b56f474
Merge branch 'streaming_data_transfer' of https://github.com/ant-tech…
chaokunyang a3ef084
remove clear
bbabab0
Merge branch 'streaming_data_transfer' of github.com:ant-tech-allianc…
e0648cc
fix cython type declaration
chaokunyang edf4e6b
object QueueID
chaokunyang 22b3718
fix qid_bytes_to_str
chaokunyang c4e5fe7
Merge branch 'streaming_data_transfer' of https://github.com/ant-tech…
chaokunyang 34016db
fix register_custom_serializer exception in local mode
chaokunyang 19cebb1
fix input/output gate
chaokunyang 5b5d3c0
on_streaming_transfer/on_streaming_transfer_sync
chaokunyang c18dadc
set num_workers_per_process_java=1
chaokunyang 676ab7d
fix QueueID and log
chaokunyang ea3b550
fix function descriptor
chaokunyang bf8d9aa
create actor with direct call
chaokunyang 84d470b
add job worker and remove operator instance actor
chaokunyang 0d6f4f0
fix cython compile error
chaokunyang 42daab1
rm doc/streaming-doxy-config
d79d770
revert java code
31a1b56
revert bzl
7a5b521
rm jni
e30ee6a
rm useless code
dafe1f0
add DUMMY meta for __RAY_DUMMY__ positional args
chaokunyang c34f3a6
support bytes in direct call
chaokunyang d4e9e41
add DUMMY argument in streaming queue
chaokunyang 68ae536
fix LocalMemoryBuffer can't be size of 0
chaokunyang 95f3768
add explict and override
chaokunyang aa41229
release gil to avoid direct call deadlock with user thread
chaokunyang 4b8e8e8
convert Buffer to bytes
chaokunyang e492c10
fix reader typo error
chaokunyang 132d0d8
streaming queu remove pull data
bea5a35
remove utility join & split functions
ashione a15c6cb
streaming queu remove get last msg id
3902f03
revert check and remove useless code
bc016ba
remove test_direct_call.py
8d8bdd1
remove useless
50d097c
remove
4a164d5
use cloud pickle to serialize user function
chaokunyang 3775ad7
Merge branch 'streaming_data_transfer_python' of https://github.com/a…
chaokunyang 2c534e4
revert some java changes
71d46b7
Merge branch 'streaming_data_transfer_python' of github.com:ant-tech-…
7c48d1c
revert .gitignore
b15d260
fix streaming python tests and await_finish
chaokunyang b1f1373
Merge branch 'streaming_data_transfer_python' of https://github.com/a…
chaokunyang d3ce797
remove useless interface
96e7b82
Merge branch 'streaming_data_transfer_python' of github.com:ant-tech-…
c99c388
bazel lint
chaokunyang 03f0e89
Merge branch 'streaming_data_transfer_python' of https://github.com/a…
chaokunyang 51fd36c
fix cpp test
71b1a5b
add test in .travis.yml
a7cbb19
fix wait_finish
chaokunyang 3d852fd
Merge branch 'streaming_data_transfer_python' of https://github.com/a…
chaokunyang 00a2ccb
fix .travis.yml
e8e0f0e
Merge branch 'streaming_data_transfer_python' of github.com:ant-tech-…
8de4710
optimize import
chaokunyang 3e8a4f3
Merge branch 'streaming_data_transfer_python' of https://github.com/a…
chaokunyang 29f8682
move queue exception to queue_interface.py
chaokunyang 3f43ee3
move queue to runtime package
chaokunyang 07d1723
rm _streaming.pyx
chaokunyang d4bccde
add cloudpickle dependence
chaokunyang bb7bc9e
Revert "Merge branch 'fix-java-build' into streaming_data_transfer"
8257808
Revert "fix java worker build failure"
c38a31b
fix _CLOSE_FLAG conflict
chaokunyang 1be7f43
Merge branch 'streaming_data_transfer_python' of https://github.com/a…
chaokunyang ea5de84
fix close flag
chaokunyang 1a5d581
fix close flag
chaokunyang 5aa7990
revert useless change
47b9840
Merge branch 'streaming_data_transfer_python' of github.com:ant-tech-…
2632f5f
remove unused config
ashione 9224428
Merge branch 'streaming_data_transfer_python' of https://github.com/a…
ashione 68914f4
fix remove BatchedQueue
chaokunyang e524830
Merge branch 'streaming_data_transfer_python' of https://github.com/a…
chaokunyang 62706a4
streaming queue python test
chaokunyang ac9b906
use CamelCaseStyle instead
ashione 5a45806
fix crash
d34561a
Merge branch 'streaming_data_transfer_python' of github.com:ant-tech-…
a41487a
fix README
chaokunyang 2fcfe8f
add brief intruducation of message & bundle protocol
ashione 09f568c
move streaming.fbs from streaming/src/format to streaming/src
chaokunyang c400d35
refactor some codes
c2ced9d
Merge branch 'streaming_data_transfer_python' of github.com:ant-tech-…
944f8fa
replace streaming flatbuffer by proto buffer
chaokunyang 9f7bc1c
fix streaming python protobuffer
chaokunyang 220dfb0
build python conf
chaokunyang 66ef601
streaming queue pb
bbd7327
namespace
b11e1ed
Update python/ray/_raylet.pyx
chaokunyang d344038
refine streaming config
chaokunyang a238932
clean code & comment
a94f7e0
add comments for queue_interface
chaokunyang c5a6995
refactor ray streaming structure
chaokunyang f99e3c4
fix compile
84b3f5d
clean queue_interface
b291ebd
refactor queue interface, remove QueueWriter QueueReader in queue_man…
042b3b5
fix run_streaming_common_tests
c4f85cc
rm TODO
0107bfe
streaming merger queue to message priority queue
ashione d90f852
rm CreateQueueReader and CreateQueueWriter
b17a784
remove all `streaming` prefix
chaokunyang b450c99
fix queue_interface.h
chaokunyang cdcb54e
add comments
08c778e
refactor streaming common to runtime context
ashione 037e8b2
rebase
ashione 418bb13
refine python directory structure and bazel file
chaokunyang a71a008
fix mock_actor.cc
chaokunyang 0499dbd
replace transfer by channel
ashione 708f423
rename StreamingReader/Writer to DataReader/Writer
chaokunyang 0da2921
remove InitTransfer
chaokunyang 7b7f6d2
fix cython
chaokunyang 30e9bc1
move mock_transfer code to channel to avoid circular deps
chaokunyang b5ea294
google-style python comments
chaokunyang 0fbe421
separate streaming shared library
chaokunyang 37a9496
remove serializable macro from message & message bundle
ashione a0d468f
rm StreamingSerializable
3718fb6
refactor streamingqueue
8b8a7fb
remove useless
e03622c
remove queue_id in interface
a3cd7ed
remove queue_interface
b7ec477
refactor streaming cython
chaokunyang 6fe7928
add comment
059a47b
add comments for data reader & data writer
ashione 2b81da6
fix compile error
ashione 547fe73
channel comments
ashione 37232fe
add comment
8e7296f
remove queue and add streaming python channel
chaokunyang dcf8c76
add comment
226f818
fix error in sys.excepthook
chaokunyang 59a9bed
fix
44677de
use ASCII meta for __RAY_DUMMY__
chaokunyang 609da4a
add comment
d2d642b
Merge remote-tracking branch 'ray/master' into streaming_data_transfe…
chaokunyang 07d8971
Merge remote-tracking branch 'origin/streaming_data_transfer_python' …
chaokunyang 2612c9f
fix bazel
chaokunyang 9da12d6
remove useless comment
081a86a
fix compile
eac1d4f
fix node_manager_port in streaming tests
3fe9da2
test travis
c52c773
test travis
60e5fda
Revert "test travis"
c72cf5c
streaming travis multi os
4ee2a33
Revert "streaming travis multi os"
2236b49
fix ci
cbffdcf
make ray._raylet.Buffer as raw data
chaokunyang fbd2ff0
comment
c361755
fix ci
c4b331f
remove queue concept
chaokunyang 179073c
add pip install
a692a9e
fix
5b747cd
fix cyclic import
chaokunyang 29c9828
fix
3486c04
travis_wait
953c011
fix channel test
chaokunyang 64c9554
lint queue code
4f020e8
fix lint
b8c2f04
fix core_worker.cc compile
4bf93dc
Expose ray related symbols using RTLD_GLOBAL in linux
chaokunyang 0dfeeb7
add streaming python tests
chaokunyang 81724e1
remove proto conf
chaokunyang 9b424f0
add comments
chaokunyang 571f974
polish comments for channel & ringbuffer
ashione abe80f6
remove useless comment in data writer & reader
ashione 727a6be
fix travis ci
chaokunyang 120afa5
fix ci
chaokunyang 4df003d
fix bazel
chaokunyang 7c3b2a4
clean code
chaokunyang 721a946
fix readme
chaokunyang 7d78593
fix readme
chaokunyang a40db1f
change class name
f45cf07
fix ci
chaokunyang aedff91
Hide protobuf symbols to avoid symbols conflict crash
chaokunyang 49a93db
lint
844772a
rename channel.py to transfer.py
chaokunyang 86bafad
fix .gitignore
chaokunyang f9de1be
bazel lint
chaokunyang c720763
lint
chaokunyang b16d974
Merge remote-tracking branch 'ray/master' into streaming_data_transfe…
chaokunyang c2afe43
rename native_channel.pxi to transfer.pxi
chaokunyang 2baa184
revert run_streamingqueue_test.sh
chaokunyang 2544c34
Merge remote-tracking branch 'ray/master' into streaming_data_transfe…
chaokunyang 515e0f5
make direct call buffer to bytes
chaokunyang 13f5e42
add hide symbols comments
chaokunyang 62d5dc7
add exclude symbols comments
chaokunyang 382d607
removed assci meta type
chaokunyang 327364f
fixed streaming conf proto
chaokunyang e7a02f4
refine
fef493a
lint
5f88645
fix typo or polish comments for data reader and writer
ashione ef66487
fix comment
27dbff5
explicit delete local reference in `task_execution_callback` for garb…
kfstorm 7d87d04
Expanded with new pages for getting started, etc. Blog links unchange…
deanwampler 2c281c4
[tune] Fixed bug in PBT where initial trial result is empty. (#6351)
visatish 636e6dd
[Java] Add java exception check in JNI (#6378)
kfstorm 22bd41d
[Java] Fix `instanceof RayPyActor` (#6377)
kfstorm f773a40
remove useless code
chaokunyang bce36e2
use bytes as raw data
chaokunyang 79741a7
Merge remote-tracking branch 'ray/master' into streaming_data_transfe…
chaokunyang 803d7c0
fix deserialize args
chaokunyang b618772
test determine_tests_to_run.py
chaokunyang d166435
use -Wl,--version-script to expose ABI
chaokunyang ab57ef9
fix ray.ldvs
chaokunyang 230a74d
fix symbols conflict
chaokunyang 62b4bf7
ray export symbols
chaokunyang dbe7c0f
remove useless imports
chaokunyang 8b37cff
refine comments
chaokunyang 7335cf8
fix python2 module init function export
chaokunyang c426737
fix lint
chaokunyang File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,70 @@ | ||
# cython: profile=False | ||
# distutils: language = c++ | ||
# cython: embedsignature = True | ||
# cython: language_level = 3 | ||
|
||
from libcpp cimport bool as c_bool | ||
from libcpp.string cimport string as c_string | ||
from libcpp.vector cimport vector as c_vector | ||
from libcpp.memory cimport ( | ||
shared_ptr, | ||
unique_ptr | ||
) | ||
|
||
from ray.includes.common cimport ( | ||
CBuffer, | ||
CRayObject | ||
) | ||
from ray.includes.libcoreworker cimport CCoreWorker | ||
from ray.includes.unique_ids cimport ( | ||
CObjectID, | ||
CActorID | ||
) | ||
|
||
cdef class Buffer: | ||
cdef: | ||
shared_ptr[CBuffer] buffer | ||
Py_ssize_t shape | ||
Py_ssize_t strides | ||
|
||
@staticmethod | ||
cdef make(const shared_ptr[CBuffer]& buffer) | ||
|
||
cdef class BaseID: | ||
# To avoid the error of "Python int too large to convert to C ssize_t", | ||
# here `cdef size_t` is required. | ||
cdef size_t hash(self) | ||
|
||
cdef class ObjectID(BaseID): | ||
cdef: | ||
CObjectID data | ||
object buffer_ref | ||
# Flag indicating whether or not this object ID was added to the set | ||
# of active IDs in the core worker so we know whether we should clean | ||
# it up. | ||
c_bool in_core_worker | ||
|
||
cdef CObjectID native(self) | ||
|
||
cdef class ActorID(BaseID): | ||
cdef CActorID data | ||
|
||
cdef CActorID native(self) | ||
|
||
cdef size_t hash(self) | ||
|
||
cdef class CoreWorker: | ||
cdef: | ||
unique_ptr[CCoreWorker] core_worker | ||
object async_thread | ||
object async_event_loop | ||
|
||
cdef _create_put_buffer(self, shared_ptr[CBuffer] &metadata, | ||
size_t data_size, ObjectID object_id, | ||
CObjectID *c_object_id, shared_ptr[CBuffer] *data) | ||
# TODO: handle noreturn better | ||
cdef store_task_outputs( | ||
self, worker, outputs, const c_vector[CObjectID] return_ids, | ||
c_vector[shared_ptr[CRayObject]] *returns) | ||
|
||
cdef c_vector[c_string] string_vector_from_list(list string_list) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this
if
needed?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
windows use
dll
, and doesn't need to expose symbols, because it always expose symbolsThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
then maybe check the OS instead, in case the file name changes in the future.