-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Java] ListVector.setNull doesn't update lastSet - performance issue #40796
Comments
@jarohen Thanks for raising this issue. I haven't evaluated this on my own yet, but just curious, planning on making a PR for this? |
Hey @vibhatha 👋 I certainly could, although I'm a little concerned that our fix misses a nuance of how |
…ctors with lots of nulls, resolves apache#40796
I totally understand. Let's see if we can review the PR and make improvements. Thanks for working on this. |
…ctors with lots of nulls, resolves apache#40796
… in ListVectors with lots of nulls (#40810) Would benefit from someone with knowledge of the context double-checking this doesn't have nuances I'm not aware of - particularly, there's a comment on the field: `the maximum index that is actually set` which one _could_ read to mean 'excluding nulls'? ### Are these changes tested? Yes ### Are there any user-facing changes? No * GitHub Issue: #40796 Authored-by: James Henderson <james@jarohen.dev> Signed-off-by: David Li <li.davidm96@gmail.com>
Issue resolved by pull request 40810 |
… O(n²) in ListVectors with lots of nulls (apache#40810) Would benefit from someone with knowledge of the context double-checking this doesn't have nuances I'm not aware of - particularly, there's a comment on the field: `the maximum index that is actually set` which one _could_ read to mean 'excluding nulls'? ### Are these changes tested? Yes ### Are there any user-facing changes? No * GitHub Issue: apache#40796 Authored-by: James Henderson <james@jarohen.dev> Signed-off-by: David Li <li.davidm96@gmail.com>
Describe the enhancement requested
In
ListVector
,setNull
doesn't update lastSet. This means that, if you set many null values in a row, the offset buffer is unnecessarily re-set for the intervening values - e.g.:.setNull(3)
withlastSet == 2
, I set offset 3.setNull(4)
,.setNull(5)
,.setNull(6)
, these set 3->4, 3->5, and 3->6 respectively - i.e. O(n²).Naively, I'd put
lastSet = index
insetNull
in the same way asstartNewValue
, but aware there might be an implicit/nuanced reason for not doing so?Cheers,
James
Component(s)
Java
The text was updated successfully, but these errors were encountered: