feat: Expose methods for ser/de data file

[liurenjie1024]

Required by other systems..

[ZENOTME]

Maybe we can consider to expose method to return a serializable object to make it more general.

[liurenjie1024]

Maybe we can consider to expose method to return a serializable object to make it more general.

I tried to implement sth like:

fn serialize<S: Serializer>(data_file: types::DataFile, s: Serializer) {}

But failed 🤪

[ZENOTME]

I tried to implement sth like:
fn serialize<S: Serializer>(data_file: types::DataFile, s: Serializer) {}
But failed 🤪

Can we directly impl Serialize in in_memory::Datafile, like:

impl Serialize for Datafile{
    // Required method
    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
       where S: Serializer {
           self.try_into().serialize(serializer)
       }
}

[Xuanwo]

Do we need to serialize the data_file into other formats instead of json?

[ZENOTME]

Do we need to serialize the data_file into other formats instead of json?

Not sure. For now maybe json is enogh. But a more general interface may better for compatibilty in future.

[Xuanwo]

But a more general interface may better for compatibilty in future.

I understand your idea, but the DataFile in iceberg is specifically designed to be a JSON format. Therefore, adding a general interface may not provide enough benefits.

[ZENOTME]

I understand your idea, but the DataFile in iceberg is specifically designed to be a JSON format. Therefore, adding a general interface may not provide enough benefits.

Here is the use case, and I think it's not so much relative the original design🤔. (CMIIW):
We seperate the write and commit phase into two process and we need to transfer(e.g. rpc) the write result (data file) to another process reponsible to commit. So We need to serialize the data file. I think in here having a more compactable serialize format is a reasonable requirement. Return a serializable object and let the user decide what format to serialize maybe more friendly.

[Xuanwo]

We seperate the write and commit phase into two process and we need to transfer(e.g. rpc) the write result (data file) to another process reponsible to commit.

It is advisable not to rely on an external data type and attempt to directly serialize/deserialize or transfer it. Instead, downstream systems should have their own data format for this purpose in order to prevent unexpected disruptions.

For a distribution system like risingwave, it may be more important since our service may not be able to upgrade simultaneously.

[ZENOTME]

It is advisable not to rely on an external data type and attempt to directly serialize/deserialize or transfer it. Instead, downstream systems should have their own data format for this purpose in order to prevent unexpected disruptions.

For a distribution system like risingwave, it may be more important since our service may not be able to upgrade simultaneously.

Make sense!