[Java] ListVector.setNull doesn't update lastSet - performance issue

[jarohen]

Describe the enhancement requested

In ListVector, setNull doesn't update lastSet. This means that, if you set many null values in a row, the offset buffer is unnecessarily re-set for the intervening values - e.g.:

• If I .setNull(3) with lastSet == 2, I set offset 3
• If I then .setNull(4), .setNull(5), .setNull(6), these set 3->4, 3->5, and 3->6 respectively - i.e. O(n²).
• With large and sparse enough vectors, this adds significant time to our internal benchmarks.

Naively, I'd put lastSet = index in setNull in the same way as startNewValue, but aware there might be an implicit/nuanced reason for not doing so?

Cheers,

James

Component(s)

Java

[vibhatha]

@jarohen Thanks for raising this issue. I haven't evaluated this on my own yet, but just curious, planning on making a PR for this?

[jarohen]

Hey @vibhatha 👋 I certainly could, although I'm a little concerned that our fix misses a nuance of how lastSet is intended to be used. If someone with knowledge of the context could confirm we haven't missed anything, yep - more than happy to submit a PR.

[jarohen]

@vibhatha FYI PR opened - #40810

[vibhatha]

Hey @vibhatha 👋 I certainly could, although I'm a little concerned that our fix misses a nuance of how lastSet is intended to be used. If someone with knowledge of the context could confirm we haven't missed anything, yep - more than happy to submit a PR.

I totally understand. Let's see if we can review the PR and make improvements. Thanks for working on this.

[lidavidm]

Issue resolved by pull request 40810
#40810