OSS Delta Sharing Server: Adds api to accept cdf query #135
Conversation
|
Hi @linzhou-db , could you have a standalone PR on https://github.com/delta-io/delta-sharing/blob/main/PROTOCOL.md with the new API's specification? It's better to agree on specification before adding its implementation on the server IMO. |
sg, will do |
Sent out #137 |
server/src/main/scala/io/delta/sharing/server/DeltaSharingService.scala
Outdated
Show resolved
Hide resolved
server/src/main/scala/io/delta/standalone/internal/DeltaErrors.scala
Outdated
Show resolved
Hide resolved
| } | ||
| if (startingVersion.isDefined) { | ||
| try { | ||
| startingVersion.get.toLong |
There was a problem hiding this comment.
Should we validate timestamp?
If it gets validate elsewhere, perhaps add a comment.
There was a problem hiding this comment.
Added a comment, same as the code in our repo, it's validated in the cdc reader.
| ) | ||
| } | ||
|
|
||
| def noStartVersionForCDF: Throwable = { |
There was a problem hiding this comment.
Should we call it noStartParamForCDF?
There was a problem hiding this comment.
I more prefer to keep it the same as DeltaErrors in our code base.
|
@zhuansunxt you still want to take a look? |
|
Why is the concept of table changes coupled with the CDF ? Isn't it possible to extract the info of the difference between a version and another by only knowing which files were added and which were removed ? The API can return this metadata which should be sufficient for the client to use to replicate the changes, correct ? |
|
Hi Basem,
CDF is short for Change Data Feed, it IS the concept of table changes.
Technically yes, while CDF is a well developed, organized, and well maintained feature for this purpose, so good to just use it. |
OSS Delta Sharing Server: Adds api to accept cdf query
DeltaSharedTable.queryCDFand return not implemented exception