Repository navigation
Releases
Releases · TFMV/arrowpb
v1.9.0
v1.8.0
v1.7.0
v1.6.0
v1.5.0
v1.4.0
v1.3.0
v1.2.0
v1.1.0
This release of arrowpb represents a substantial enhancement to the existing Arrow-to-Protobuf integration code. It provides greater flexibility, improved performance, and better support for advanced use cases like well-known types (google.protobuf.Timestamp, wrapper types), dictionary-encoded columns (mapped to enums), and parallel record conversion. Below is a high-level summary of the key changes and features:
- Well-Known Types (WKT) Support
Automatic Integration with google.protobuf.Timestamp and google.protobuf.*Value (wrappers) by embedding the necessary proto definitions into a global descriptor set.
Runtime-Generated Descriptors now resolve references to these WKT files (e.g., timestamp.proto, wrappers.proto) when compiling via protodesc.NewFiles. - Dictionary-Encoded Columns => Enums
Optional handling of arrow.DictionaryType as a proto enum.
When enabled (MapDictionariesToEnums = true in ConvertConfig), dictionary-encoded columns generate a nested enum in the containing message, allowing cleaner, more strongly typed representations in Protobuf. - Struct and List Support
Deeply-Nested Structs: arrow.StructType fields become nested proto messages, recursively.
Repeated Fields: arrow.ListType, arrow.FixedSizeListType, and arrow.LargeListType become repeated fields in Protobuf.
The code automatically handles multi-level nesting (lists within structs, structs within lists, etc.). - Configurable Presence & Syntax
Proto2 vs. Proto3: If UseProto2Syntax = true, the generated .proto file descriptor is marked syntax = "proto2".
Wrapper Types (google.protobuf.*Value): Set UseWrapperTypes = true to map scalars like int64 to a wrapper message (Int64Value), thereby preserving presence bits even under proto3. - Parallel Conversion & Descriptor Caching
ConvertInParallel function allows distributing row conversion across multiple goroutines, improving throughput on large Arrow records.
Descriptor Caching (with DescriptorCache sync.Map) avoids recompiling the same schema repeatedly. - Logging and Diagnostics
Uses a zap.Logger for detailed logging of:
Descriptor compilation steps
Duration of record-to-proto conversions
Error paths and fallback strategies - Example Usage & Utilities
CreateArrowRecord: A sample function to build a simple Arrow record.
FormatArrowJSON: Renders Arrow data as pretty-printed JSON for debugging.
ArrowReaderToProtos: Streams over an array.RecordReader and produces a slice of Protobuf messages (one per row).
What's Changed
- Enhancements. V0.2 by @TFMV in #5
- Add Zap Logging by @TFMV in #6
- resolve WKT and dictionary enum references by @TFMV in #7
- Improve Test Coverage by @TFMV in #8
Full Changelog: v0.2.0...v1.1.0