What's the best way to measure the size of arrow.Record in Golang?

[Yifeng-Sigma]

Describe the usage question you have. Please include as many useful details as possible.

I want to do some record splitting/merging based on the size, but didn't find a reliable way to estimate the size.
There are two ways:

	for _, col := range rec.Columns() {
		for _, buffer := range col.Data().Buffers() {
			if buffer != nil {
				bytes += uint64(buffer.Cap())
			}
		}
	
}

or

func computeColumnSize(col arrow.Array) uint64 {
	switch colType := col.DataType().(type) {
	case arrow.BinaryDataType:
		switch arr := col.(type) {
		case *array.String:
			return uint64(arr.ValueOffset(arr.Len()) - arr.ValueOffset(0))
                      // ...
		}
	}
}

I'm wondering what's the recommended way to compute the size of arrow.Record.

Component(s)

Go

[zeroshade]

My recommendation would be a variation on the first one:

func calcSize(arr arrow.ArrayData) (sz uint64) {
    if arr == nil {
        return
    }

    for _, b := range arr.Buffers() {
        sz += uint64(b.Len())
    }
    for _, c := range arr.Children() {
        sz += calcSize(c)
    }
    sz += calcSize(arr.Dictionary())
    return
}

That would be my recommendation, it might be reasonable to add this as a utility into the arrow library directly via a PR

[Yifeng-Sigma]

My recommendation would be a variation on the first one:

func calcSize(arr arrow.ArrayData) (sz uint64) {
    if arr == nil {
        return
    }

    for _, b := range arr.Buffers() {
        sz += uint64(b.Len())
    }
    for _, c := range arr.Children() {
        sz += calcSize(c)
    }
    sz += calcSize(arr.Dictionary())
    return
}

That would be my recommendation, it might be reasonable to add this as a utility into the arrow library directly via a PR

Thanks, opened #38839