Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: LayoutBuilder in Numba #2408

Merged
merged 146 commits into from
Jun 29, 2023
Merged

feat: LayoutBuilder in Numba #2408

merged 146 commits into from
Jun 29, 2023

Conversation

ianna
Copy link
Collaborator

@ianna ianna commented Apr 18, 2023

  • BitMaskedType: append, extend, append_null (0 args), extend_null (1 int arg)
  • ByteMaskedType: append, extend, append_null (0 args), extend_null (1 int arg)
  • Remove EmptyRecordType:
  • RecordType must have at least one content
  • TupleType must have at least one content
    - [ ] also remember to remove it from C++ - as a separate PR
  • EmptyType
  • IndexedOptionType: append, extend, append_null (0 args), extend_null(1 int arg). (append_index and extend_index?)
  • IndexedType: append, extend. (append_index and extend_index?)
  • ListOffsetType: begin_list->content, end_list (both modify offsets)
  • ListType: begin_list->content (modify starts), end_list (modify stops) (both modify offsets)
  • NumpyType: append, extend
  • RecordType: field (Literal string) alias, content (Literal int and string) method
  • RegularType: begin_list->content, end_list (does nothing?), content property, size 0 is not allowed
  • TupleType: index (Literal int) alias, content (Literal int) method
  • UnionType: append_index (Literal int) updates tags and index -> content, and content (Literal int) method
  • UnmaskedType: append, extend

@ianna ianna marked this pull request as draft April 18, 2023 11:26
@ianna ianna linked an issue Apr 18, 2023 that may be closed by this pull request
@codecov
Copy link

codecov bot commented Apr 18, 2023

Codecov Report

Merging #2408 (7562075) into main (1f6041e) will increase coverage by 0.40%.
The diff coverage is 88.49%.

Additional details and impacted files
Impacted Files Coverage Δ
src/awkward/_connect/numba/growablebuffer.py 71.89% <0.00%> (ø)
src/awkward/_connect/numba/layoutbuilder.py 87.26% <87.26%> (ø)
src/awkward/numba/layoutbuilder.py 89.51% <89.51%> (ø)
src/awkward/numba/__init__.py 97.81% <100.00%> (ø)

... and 1 file with indirect coverage changes

@ianna ianna temporarily deployed to docs-preview April 18, 2023 11:41 — with GitHub Actions Inactive
@ianna ianna temporarily deployed to docs-preview April 19, 2023 10:23 — with GitHub Actions Inactive
@ianna ianna temporarily deployed to docs-preview April 19, 2023 13:18 — with GitHub Actions Inactive
@ianna ianna temporarily deployed to docs-preview April 20, 2023 14:19 — with GitHub Actions Inactive
@ianna ianna temporarily deployed to docs-preview April 20, 2023 21:58 — with GitHub Actions Inactive
@ianna ianna temporarily deployed to docs-preview May 5, 2023 15:24 — with GitHub Actions Inactive
@pre-commit-ci pre-commit-ci bot temporarily deployed to docs-preview June 28, 2023 14:17 Inactive
@ianna ianna temporarily deployed to docs-preview June 29, 2023 10:00 — with GitHub Actions Inactive
@ianna ianna temporarily deployed to docs-preview June 29, 2023 10:18 — with GitHub Actions Inactive
@jpivarski
Copy link
Member

As you said in the group meeting today, this PR is now done(ish?) and reviewable, so I'll take it out of draft status and remind myself to review it again.

@jpivarski jpivarski marked this pull request as ready for review June 29, 2023 12:53
@jpivarski jpivarski self-requested a review June 29, 2023 12:53
@ianna ianna temporarily deployed to docs-preview June 29, 2023 14:22 — with GitHub Actions Inactive
@ianna ianna temporarily deployed to docs-preview June 29, 2023 15:29 — with GitHub Actions Inactive
@ianna ianna temporarily deployed to docs-preview June 29, 2023 15:46 — with GitHub Actions Inactive
@jpivarski jpivarski temporarily deployed to docs-preview June 29, 2023 18:50 — with GitHub Actions Inactive
@jpivarski jpivarski temporarily deployed to docs-preview June 29, 2023 20:48 — with GitHub Actions Inactive
Copy link
Member

@jpivarski jpivarski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this by checking it out and trying several examples, and they all looked fine. For example,

import numba as nb
import numpy as np

import awkward as ak
import awkward.numba.layoutbuilder as lb

@nb.njit
def doit(builder):
    x = 0
    y = 0.0
    for i in range(10):
        subbuilder = builder.begin_list()
        x_builder = subbuilder.content("x")
        y_builder = subbuilder.content("y")
        for _ in range(i):
            x_builder.append(x)
            y_builder.append(y)
            x += 1
            y += 1.1
        builder.end_list()

builder = lb.ListOffset(
    np.int64,
    lb.Record([lb.Numpy(np.int32), lb.Numpy(np.float64)], ["x", "y"]),
)

doit(builder)

ak.Array(builder.snapshot()).show(type=True)

produces

type: 10 * var * {
    x: int32,
    y: float64
}
[[],
 [{x: 0, y: 0}],
 [{x: 1, y: 1.1}, {x: 2, y: 2.2}],
 [{x: 3, y: 3.3}, {x: 4, y: 4.4}, {x: 5, y: 5.5}],
 [{x: 6, y: 6.6}, {x: 7, y: 7.7}, {x: 8, y: 8.8}, {x: 9, y: 9.9}],
 [{x: 10, y: 11}, {x: 11, y: 12.1}, {...}, {x: 13, ...}, {x: 14, y: 15.4}],
 [{x: 15, y: 16.5}, {x: 16, y: 17.6}, {...}, ..., {x: 19, ...}, {x: 20, y: 22}],
 [{x: 21, y: 23.1}, {x: 22, y: 24.2}, ..., {x: 26, ...}, {x: 27, y: 29.7}],
 [{x: 28, y: 30.8}, {x: 29, y: 31.9}, ..., {x: 34, ...}, {x: 35, y: 38.5}],
 [{x: 36, y: 39.6}, {x: 37, y: 40.7}, ..., {x: 43, ...}, {x: 44, y: 48.4}]]

I had to move the Python interface classes (the builder that has to be created outside of the compiled context) to a public place to be able to get them with

import awkward.numba.layoutbuilder as lb

(This is how I think we should present it to the world. We'll encourage the use of the lb abbreviation, here and in C++: lb::ListOffset, etc.)

When the tests finish, I'll merge it. Thanks, @ianna!

@jpivarski jpivarski enabled auto-merge (squash) June 29, 2023 20:49
@jpivarski jpivarski merged commit 0f1bf1b into main Jun 29, 2023
36 checks passed
@jpivarski jpivarski deleted the ianna/layoutbuilder-in-numba branch June 29, 2023 20:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement LayoutBuilder in Numba
3 participants