Decode RawValues without ANY/interface{}

[hhoughgg]

I am using rows.RawValues() and rows.FieldDescriptions() to get the raw values and the codec for each type. I then do codec.DecodeValue(). The issue is that codec.DecodeValue returns any so always causes a heap allocation. I copy my data into an arrow array so this causes the memory to ultimately be allocated three times. Is it possible to do something like codec.(*BoolCodec).DecodeBool() that gets around the DecodeValue interface function?

[jackc]

What is your goal? AFAICT, what you describe is the equivalent of calling rows.Values().

[hhoughgg]

BenchmarkPGXDriverToArrow/PGXDriverToArrowValues-12         	    2478	    459458 ns/op	   22450 B/op	     263 allocs/op
BenchmarkPGXDriverToArrow/PGXDriverToArrowRawValues-12      	    2697	    441613 ns/op	   20176 B/op	     163 allocs/op
BenchmarkPGXDriverToArrow/PGXDriverRawCopyTo-12             	    1728	    675550 ns/op	      64 B/op	       2 allocs/op

I am trying to improve performance as I need to copy 10s of billions of rows per day from Postgres. As far as I understand it each interface{} value from rows.Values will cause a heap allocation. I need to then reserve memory in arrow again to convert from rows to columns. It would be better if I can avoid (any/heap alloc) and convert directly to the real type. The benchmark above is only text columns from postgres and just doing string(bytes) on the raw values instead of DecodeValue. This way since the memory I need is already allocated as an arrow array and it just makes a copy to the existing memory instead of a new allocation.

[jackc]

As far as I understand it each interface{} value from rows.Values will cause a heap allocation.

FWIW, DecodeValue returning any may have some impact, but I don't think it is the primary issue. Interface method calls hindering escape analysis and inlining would be my guess for the chief culprit.

It would be better if I can avoid (any/heap alloc) and convert directly to the real type.

I assume you have tried the simple approach of having one set of the real value types and repeatedly scanning into that? That should minimize allocations. From what I remember of benchmarking and optimizing this path there actually shouldn't be any per row allocations at all. And there are no allocations per value either unless it is a type that requires it like string. But for strings you instead use pgtype.DriverBytes or pgtype.PreallocBytes to avoid the allocation.

[hhoughgg]

I was looking more carefully at how the scan function works after your comment. I will try directly using row.RawValues and making the conversions outside of Scan() using PlanScan and directly appending each value to the Arrow array. Based on what you have mentioned I am hoping this will avoid any allocations per row and the escape analysis will be smart enough to understand the value is not used beyond each Next() call.

Thank you for your help!