SAMZA-2709: Adding partial update api to Table API#1560
SAMZA-2709: Adding partial update api to Table API#1560mynameborat merged 9 commits intoapache:masterfrom
Conversation
cd68e03 to
22c3c41
Compare
xinyuiscool
left a comment
There was a problem hiding this comment.
Reviewed 1/3. Continue later.
samza-api/src/main/java/org/apache/samza/operators/MessageStream.java
Outdated
Show resolved
Hide resolved
samza-api/src/main/java/org/apache/samza/operators/UpdatePair.java
Outdated
Show resolved
Hide resolved
samza-api/src/main/java/org/apache/samza/table/AsyncReadWriteTable.java
Outdated
Show resolved
Hide resolved
samza-api/src/main/java/org/apache/samza/table/AsyncReadWriteTable.java
Outdated
Show resolved
Hide resolved
samza-api/src/main/java/org/apache/samza/table/remote/TableWriteFunction.java
Outdated
Show resolved
Hide resolved
samza-core/src/main/java/org/apache/samza/operators/impl/SendUpdateToTableOperatorImpl.java
Outdated
Show resolved
Hide resolved
c865bdc to
e4ac3cd
Compare
samza-core/src/main/java/org/apache/samza/operators/impl/SendUpdateToTableOperatorImpl.java
Outdated
Show resolved
Hide resolved
e4ac3cd to
7e4ad6a
Compare
samza-core/src/main/java/org/apache/samza/operators/impl/SendUpdateToTableOperatorImpl.java
Outdated
Show resolved
Hide resolved
samza-core/src/main/java/org/apache/samza/operators/impl/SendUpdateToTableOperatorImpl.java
Outdated
Show resolved
Hide resolved
d14fe2e to
3d34c1f
Compare
samza-core/src/main/java/org/apache/samza/operators/impl/SendUpdateToTableOperatorImpl.java
Outdated
Show resolved
Hide resolved
samza-core/src/main/java/org/apache/samza/operators/impl/SendUpdateToTableOperatorImpl.java
Outdated
Show resolved
Hide resolved
samza-api/src/main/java/org/apache/samza/table/AsyncReadWriteTable.java
Outdated
Show resolved
Hide resolved
3d34c1f to
abfa54f
Compare
xinyuiscool
left a comment
There was a problem hiding this comment.
The batching impl which is trying to cast the update to the value type is not correct. The fundamental problem is that Operation correctly only contains key and value. We need to add a Update there too to reflect the update operation. For put and get, the getUpdate() will return null. For UpdateOperation. we return null in getValue() but return the correct update in the getUpdate().
d409a09 to
ebf1ec3
Compare
samza-core/src/main/java/org/apache/samza/table/batching/UpdateOperation.java
Outdated
Show resolved
Hide resolved
samza-core/src/main/java/org/apache/samza/table/batching/CompactBatch.java
Show resolved
Hide resolved
ebf1ec3 to
e259dfb
Compare
samza-core/src/main/java/org/apache/samza/table/batching/CompactBatch.java
Show resolved
Hide resolved
samza-api/src/main/java/org/apache/samza/table/RecordDoesNotExistException.java
Outdated
Show resolved
Hide resolved
samza-api/src/main/java/org/apache/samza/table/RecordDoesNotExistException.java
Outdated
Show resolved
Hide resolved
e259dfb to
50bca8f
Compare
9e8c6be to
fa906e5
Compare
fa906e5 to
b030b75
Compare
xinyuiscool
left a comment
There was a problem hiding this comment.
Overall looks look. Some minor things in the comments.
samza-api/src/main/java/org/apache/samza/operators/MessageStream.java
Outdated
Show resolved
Hide resolved
aa44105 to
65ef2a2
Compare
samza-core/src/main/java/org/apache/samza/operators/spec/OperatorSpecs.java
Outdated
Show resolved
Hide resolved
samza-core/src/main/java/org/apache/samza/operators/MessageStreamImpl.java
Outdated
Show resolved
Hide resolved
65ef2a2 to
fc8287d
Compare
xinyuiscool
left a comment
There was a problem hiding this comment.
LGTM! Thanks for all the improvements!
samza-api/src/main/java/org/apache/samza/operators/MessageStream.java
Outdated
Show resolved
Hide resolved
samza-api/src/main/java/org/apache/samza/table/AsyncReadWriteTable.java
Outdated
Show resolved
Hide resolved
samza-api/src/main/java/org/apache/samza/table/remote/TableWriteFunction.java
Show resolved
Hide resolved
samza-core/src/main/java/org/apache/samza/operators/impl/SendToTableOperatorImpl.java
Show resolved
Hide resolved
samza-core/src/main/java/org/apache/samza/operators/impl/SendToTableWithUpdateOperatorImpl.java
Show resolved
Hide resolved
samza-core/src/main/java/org/apache/samza/operators/impl/SendToTableWithUpdateOperatorImpl.java
Outdated
Show resolved
Hide resolved
mynameborat
left a comment
There was a problem hiding this comment.
Minor follow up on the comments; Looks good otherwise.
Feature:
Samza Table API currently supports PUT, GET and DELETE. Puts typically write/overwrite the data for a key. Users have frequently requested for the ability to perform partial updates i.e update select fields or a part of the record. This PR intends to add
updateto Table API.Changes:
updateAsyncandupdateAllAsyncmethods toTableWriteFunctionand toAsyncReadWriteTableAsyncReadWriteTablehave been changed to accommodate the API changesendUpdateTomethod toMessageStream. Added corresponding operator spec and operator implementation as wellTests:
API Changes