We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
分布式计算系统中经常会通过shuffle在节点间传输大量数据.IO瓶颈(磁盘/网络)也shuffle环节最大的挑战. 通过减少传输字节数来提升IO是非常有效的手段.常见做法是高效的序列化器+压缩来提升IO性能.
所有编码器解码器都继承自Encoder<E>且由Encoders类进行引用: 接口设计如下:
Encoder<E>
Encoders
public interface Encoder<E> extends Serializable { public void encoder(E value, DataOutput output) throws IOException; public E decoder(DataInput input) throws IOException; }
The text was updated successfully, but these errors were encountered:
#4 Add record encoder
ccfff64
61cc343
harbby
Successfully merging a pull request may close this issue.
添加 数据行编码器/解码器功能
背景:
分布式计算系统中经常会通过shuffle在节点间传输大量数据.IO瓶颈(磁盘/网络)也shuffle环节最大的挑战.
通过减少传输字节数来提升IO是非常有效的手段.常见做法是高效的序列化器+压缩来提升IO性能.
特点:
设计
所有编码器解码器都继承自
Encoder<E>
且由Encoders
类进行引用:接口设计如下:
兼容性:
效果:
The text was updated successfully, but these errors were encountered: