Skip to content

v1.1.0

Pre-release
Pre-release

Choose a tag to compare

@TFMV TFMV released this 01 Feb 14:03
· 28 commits to main since this release
88d2139

This release of arrowpb represents a substantial enhancement to the existing Arrow-to-Protobuf integration code. It provides greater flexibility, improved performance, and better support for advanced use cases like well-known types (google.protobuf.Timestamp, wrapper types), dictionary-encoded columns (mapped to enums), and parallel record conversion. Below is a high-level summary of the key changes and features:

  1. Well-Known Types (WKT) Support
    Automatic Integration with google.protobuf.Timestamp and google.protobuf.*Value (wrappers) by embedding the necessary proto definitions into a global descriptor set.
    Runtime-Generated Descriptors now resolve references to these WKT files (e.g., timestamp.proto, wrappers.proto) when compiling via protodesc.NewFiles.
  2. Dictionary-Encoded Columns => Enums
    Optional handling of arrow.DictionaryType as a proto enum.
    When enabled (MapDictionariesToEnums = true in ConvertConfig), dictionary-encoded columns generate a nested enum in the containing message, allowing cleaner, more strongly typed representations in Protobuf.
  3. Struct and List Support
    Deeply-Nested Structs: arrow.StructType fields become nested proto messages, recursively.
    Repeated Fields: arrow.ListType, arrow.FixedSizeListType, and arrow.LargeListType become repeated fields in Protobuf.
    The code automatically handles multi-level nesting (lists within structs, structs within lists, etc.).
  4. Configurable Presence & Syntax
    Proto2 vs. Proto3: If UseProto2Syntax = true, the generated .proto file descriptor is marked syntax = "proto2".
    Wrapper Types (google.protobuf.*Value): Set UseWrapperTypes = true to map scalars like int64 to a wrapper message (Int64Value), thereby preserving presence bits even under proto3.
  5. Parallel Conversion & Descriptor Caching
    ConvertInParallel function allows distributing row conversion across multiple goroutines, improving throughput on large Arrow records.
    Descriptor Caching (with DescriptorCache sync.Map) avoids recompiling the same schema repeatedly.
  6. Logging and Diagnostics
    Uses a zap.Logger for detailed logging of:
    Descriptor compilation steps
    Duration of record-to-proto conversions
    Error paths and fallback strategies
  7. Example Usage & Utilities
    CreateArrowRecord: A sample function to build a simple Arrow record.
    FormatArrowJSON: Renders Arrow data as pretty-printed JSON for debugging.
    ArrowReaderToProtos: Streams over an array.RecordReader and produces a slice of Protobuf messages (one per row).

What's Changed

  • Enhancements. V0.2 by @TFMV in #5
  • Add Zap Logging by @TFMV in #6
  • resolve WKT and dictionary enum references by @TFMV in #7
  • Improve Test Coverage by @TFMV in #8

Full Changelog: v0.2.0...v1.1.0