-
Notifications
You must be signed in to change notification settings - Fork 81
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: LayoutBuilder in Numba #2408
Conversation
Codecov Report
Additional details and impacted files
|
As you said in the group meeting today, this PR is now done(ish?) and reviewable, so I'll take it out of draft status and remind myself to review it again. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I reviewed this by checking it out and trying several examples, and they all looked fine. For example,
import numba as nb
import numpy as np
import awkward as ak
import awkward.numba.layoutbuilder as lb
@nb.njit
def doit(builder):
x = 0
y = 0.0
for i in range(10):
subbuilder = builder.begin_list()
x_builder = subbuilder.content("x")
y_builder = subbuilder.content("y")
for _ in range(i):
x_builder.append(x)
y_builder.append(y)
x += 1
y += 1.1
builder.end_list()
builder = lb.ListOffset(
np.int64,
lb.Record([lb.Numpy(np.int32), lb.Numpy(np.float64)], ["x", "y"]),
)
doit(builder)
ak.Array(builder.snapshot()).show(type=True)
produces
type: 10 * var * {
x: int32,
y: float64
}
[[],
[{x: 0, y: 0}],
[{x: 1, y: 1.1}, {x: 2, y: 2.2}],
[{x: 3, y: 3.3}, {x: 4, y: 4.4}, {x: 5, y: 5.5}],
[{x: 6, y: 6.6}, {x: 7, y: 7.7}, {x: 8, y: 8.8}, {x: 9, y: 9.9}],
[{x: 10, y: 11}, {x: 11, y: 12.1}, {...}, {x: 13, ...}, {x: 14, y: 15.4}],
[{x: 15, y: 16.5}, {x: 16, y: 17.6}, {...}, ..., {x: 19, ...}, {x: 20, y: 22}],
[{x: 21, y: 23.1}, {x: 22, y: 24.2}, ..., {x: 26, ...}, {x: 27, y: 29.7}],
[{x: 28, y: 30.8}, {x: 29, y: 31.9}, ..., {x: 34, ...}, {x: 35, y: 38.5}],
[{x: 36, y: 39.6}, {x: 37, y: 40.7}, ..., {x: 43, ...}, {x: 44, y: 48.4}]]
I had to move the Python interface classes (the builder
that has to be created outside of the compiled context) to a public place to be able to get them with
import awkward.numba.layoutbuilder as lb
(This is how I think we should present it to the world. We'll encourage the use of the lb
abbreviation, here and in C++: lb::ListOffset
, etc.)
When the tests finish, I'll merge it. Thanks, @ianna!
BitMaskedType
:append
,extend
,append_null
(0 args),extend_null
(1 int arg)ByteMaskedType
:append
,extend
,append_null
(0 args),extend_null
(1 int arg)EmptyRecordType
:RecordType
must have at least one contentTupleType
must have at least one content- [ ] also remember to remove it from C++- as a separate PREmptyType
IndexedOptionType
:append
,extend
,append_null
(0 args),extend_null
(1 int arg). (append_index
andextend_index
?)IndexedType
:append
,extend
. (append_index
andextend_index
?)ListOffsetType
:begin_list
->content,end_list
(bothmodify offsets)ListType
:begin_list
->content (modify starts),end_list
(modify stops)(both modify offsets)NumpyType
:append
,extend
RecordType
:field
(Literal string) alias,content
(Literal int and string) methodRegularType
:begin_list
->content,end_list
(does nothing?), content property, size 0 is not allowedTupleType
:index
(Literal int) alias,content
(Literal int) methodUnionType
:append_index
(Literal int) updates tags and index -> content, andcontent
(Literal int) methodUnmaskedType
:append
,extend