feat(query): add native format in fuse table

[sundy-li]

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

Summary about this PR

1. bump arrow2 version
2. Introduce an experimental native storage_format in fuse

Closes #issue

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Updated
databend	⬜️ Ignored (Inspect)		Dec 19, 2022 at 9:37AM (UTC)

[sundy-li]

how to use:

 create table tmp (a int) ENGINE=FUSE STORAGE_FORMAT='native';

It had great performance improvement in hits dataset:

MySQL [(none)]> select count(1),  max(URL) from hits;
+----------+-----------------------------------------+
| count(1) | max(url)                                |
+----------+-----------------------------------------+
| 99997497 | https://yugra-advert2792270][to]=&input |
+----------+-----------------------------------------+
1 row in set (4.491 sec)

MySQL [(none)]> select count(1),  max(URL) from hits_native;
+----------+-----------------------------------------+
| count(1) | max(url)                                |
+----------+-----------------------------------------+
| 99997497 | https://yugra-advert2792270][to]=&input |
+----------+-----------------------------------------+
1 row in set (0.506 sec)

[sundy-li]

Average 1.85x in hits dataset

[BohuTANG]

Based on s3:

Q7 & Q8 native is faster than parquet, others are not very obvious, and a number of them are even slower than parquet.

Q7: SELECT MIN(EventDate), MAX(EventDate) FROM hits;
Q8: SELECT AdvEngineID, COUNT(*) FROM hits WHERE AdvEngineID <> 0 GROUP BY AdvEngineID ORDER BY COUNT(*) DESC;

[sundy-li]

Q7 & Q8 native is faster than parquet, others are not very obvious, and a number of them are even slower than parquet.

Currently, we don't have prewhere optimization like parquet source. This is an initial implementation that take no bad effect on default parquet-based fuse engine, we can have it later.

I'd like to design another faster page filter index than parquet page index later.

[BohuTANG]

@sundy-li

Some tests are deleted(like 09_fuse_engine/09_0001_remote_insert_v2), this is for?

[sundy-li]

@sundy-li

Some tests are deleted(like 09_fuse_engine/09_0001_remote_insert_v2), this is for?

It's duplicated.

[BohuTANG]

🚀