New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bundle Formats (JSON vs BSON vs Avro vs Parquet) #3

Closed

therne opened this issue Oct 31, 2018 · 2 comments

Labels

discussion

Contributor

therne commented Oct 31, 2018

Bundle 의 요구사항은 다음과 같습니다.

앞에 Header가 위치
번들 안에서 i번째 데이터를 가져올 수 있어야 함

Row-based JSON (Current)

Pros:

Easy to code
Easy to debug
Platform-wide

Cons:

Heavy
Slow
Doesn't support raw byte encoding

BSON

Pros:

Compact
Raw-byte encoding (Binary Protocol!)

Cons:

Doesn't support row-based retrieval
Hard to debug

Apache Avro

Pros:

Row-based Retrieval
Fast
Compact
Enterprise Friendly (Especially Hadoop Infrastructure)

Cons:

Isn't it over-engineering?
Worse at Column-based Retrieval
- e.g) Aggregate only payload / ANID

Apache Parquet

Pros:

Column-based Retrieval
Fast
Compact
Enterprise Friendly (Especially Hadoop Infrastructure)

Cons:

Isn't it over-engineering?
Worse at Row-based Retrieval
- e.g) Retrieve data on ith index

Contributor Author

therne commented Oct 31, 2018

Using Avro in Go: https://github.com/actgardner/gogen-avro

byeongsu-hong modified the milestone: Airbloc Public Alpha

Contributor

byeongsu-hong commented Oct 31, 2018

일단 있는것들로 돌아가는 틀을 만든 후 도입해도 될것같습니다.

therne added the discussion label

therne added this to the Airbloc Public Alpha milestone

byeongsu-hong mentioned this issue

P2P Protocol Layer TODO (frostornge) #37

Closed

6 tasks

therne removed this from the Airbloc Public Alpha milestone

therne closed this as completed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment