[Java] DenseUnionVector has no way to set offset/validity directly

[asfimport]

You can set the type ID manually, but you cannot set the offset or validity directly. Ideally, we'd have an API like Python that lets us build it directly from constituent vectors and the offsets/type IDs.

Reporter: David Li / @lidavidm

_{Note: This issue was originally created as ARROW-8666. Please see the migration documentation for further details.}

[asfimport]

Liya Fan / @liyafan82:
@lidavidm Thanks for opening this issue.
Could you please give more details about your scenario?
Currently, we have loadFieldBuffers, does it meet your needs?

[asfimport]

David Li / @lidavidm:
Hi [~fan_li_ya], loadFieldBuffers should do the trick, thanks for pointing it out!

The motivation here is we are constructing a dense union vector where the child arrays are in turn struct vectors. Semantically, this is intended as the union of distinct top-level schemas so that we can pass through different datasets in a single Flight RPC. Ideally we can do this with minimal/no copying or allocation, so being able to just transfer the arrays from the input VectorSchemaRoots and construct the right type ID and offset buffers is perfect.

[asfimport]

Bryan Cutler / @BryanCutler:
@lidavidm can this be closed then, or are there further API improvements needed?

[asfimport]

David Li / @lidavidm:
Closing.

[jarohen]

@lidavidm We've found setOffset to be useful here when constructing DUVs by adding values directly to the underlying vectors - i.e. we do duv.setTypeId(typeId), duv.setOffset(offset), duv.getVectorByType(typeId).set(lng) (we don't do getVectorByType for every row - this is cached).

We've got a commit for it here - would it be something we could donate upstream?

(We've got plenty of end-to-end tests covering it in XTDB, but would also add some unit tests if we raised a PR here.)

[lidavidm]

Sure, I think it makes sense to have still.

[lidavidm]

Issue resolved by pull request 40985
#40985