Use Arrow as in-memory data format

[cydrain]

What would you like to be added:

Use Arrow as in-memory data format

Why is this needed:

There are too many data conversion between column-based and row-based.
We can use Arrow format as the internal data format to avoid frequent data conversion.

https://wiki.lfaidata.foundation/display/MIL/MEP+13+--+Support+Apache+Arrow+As+In-Memory+Data+Format

[xiaofan-luan]

Let's only change data format in pulsar. Rest of the format let's leave to later version after 2.0

[cydrain]

Let's only change data format in pulsar. Rest of the format let's leave to later version after 2.0

@xiaofan-luan
Currently, the binlog file written into Minio is with Parquet format wrapped by our data structure, and each RecordBatch (segment) is saved into several files by column.
My suggestion is:

1. one binlog file for one RecordBatch (segment)
2. use pure Parquet file format, which can be written from arrow data directly

[xiaofan-luan]

1. I would prefer to keep multiple binlog files, otherwise if you want to query on disk it will cost too many ios.
   2)Can we make a delegator to read from binlog to arrow rather than change the file format directly? Change the data format on minio will cause incompatibility, how are we gonna to handle this?

[cydrain]

1. I would prefer to keep multiple binlog files, otherwise if you want to query on disk it will cost too many ios.
   2)Can we make a delegator to read from binlog to arrow rather than change the file format directly? Change the data format on minio will cause incompatibility, how are we gonna to handle this?

@xiaofan-luan
1). You're right, since binlog files are saved in Minio not local disk, it can save a lot of network IO to save binlog files by columns.
2). It's OK to keep current binlog file format un-changed for compatibility.

[cydrain]

After deep investigation, we think Arrow is not suitable for Milvus.
I described the detailed reason in MEP document:
wiki.lfaidata.foundation/display/MIL/MEP+13+--+Support+Apache+Arrow+As+In-Memory+Data+Format

[cydrain]

/close

[sre-ci-robot]

@cydrain: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.