[Core worker] Add metadata support in object interface #5031
Conversation
kfstorm
changed the title
kfstorm
changed the title
|
Test FAILed. |
| unready.erase(object_id); | ||
| // TODO (kfstorm): metadata should be structured. | ||
| std::string metadata = object_buffers[i].metadata->ToString(); | ||
| for (auto &error_type : EnumValuesErrorType()) { |
There was a problem hiding this comment.
missing the definition for EnumValuesErrorType?
There was a problem hiding this comment.
EnumValuesErrorType is generated by flatbuffer. PS: This part of code should not be in this PR. It will come with Java worker changes. I'll remove it.
| @@ -8,6 +8,30 @@ | |||
|
|
|||
| namespace ray { | |||
|
|
|||
| /// Object value in ray system. | |||
| class RayObjectValue { | |||
There was a problem hiding this comment.
how about RayObject? or ObjectValue might be also ok
There was a problem hiding this comment.
I'm also considering store object reference in this class in the future, like store OBJECT_ID in metadata field and raw bytes of object ID in data field. Then TaskArg could be totally replaced by this structure. So *Value is not appropriate. Maybe RayObject is good enough. What do you think?
There was a problem hiding this comment.
RayObject sounds good to me.
For storing object ID into this class, is there any requirement for this? In ray we have object IDs and objects, I'm thinking RayObject or ObjectValue is still talking about objects, not the ids.
There was a problem hiding this comment.
One scenario for storing ID into this class is, cross-language invocation against a remote function which returns an object ID.
Suppose we have a Java remote function with signature RayObject<Integer> foo(), and we want to call foo in Java like this: RayObject<RayObject<Integer>> result = Ray.call(SomeClass::foo); Integer intResult = result.get().get();. This works because the return value of foo is serialized by Java and stored in object store, and the caller can deserialize it and get the wrapped Integer type result.
However, if we define foo in Python, then the return value of foo will be serialized by Python. When we try to get the result of foo from Java, deserialization fails because Java can't understand the binary generated by Python.
By encoding object ID into this class, with metadata of OJBECT_ID and data of bytes of the object ID, we have a language-independent way about representing and storing of object ID. We want to deal with actor handle and exception exactly like this to make them language-independent too.
In this PR, this class is just a handful data structure to make the object interface more clean. But it can also be used to help solve cross-language limitations. Now for cross-language we can only call Python from Java, not reversed. And the only acceptable parameter and return type of Python function is byte array. We want to change that.
|
Test FAILed. |
|
Test PASSed. |
|
LGTM, can be merged after the |
|
Test FAILed. |
|
Test FAILed. |
|
Test PASSed. |
|
Test PASSed. |
What do these changes do?
Plasma store supports store both metadata and data for an object ID. But current core worker object interface can only put and get data. Put operation inserts null metadata, and get operation drops metadata. This PR modified the interfaces and wrapped both metadata and data into a struct.
Related issue number
#4850
Linter
scripts/format.shto lint the changes in this PR.