feat: add 'dak.backend' and dak.to_list and verify that they overload the ak.* versions

[jpivarski]

The tests need type annotations! That's too much, man!

[martindurant]

Some floating point rounding error?

b          = <Array [[7.21, 5.83, 5.83], [], ..., [5, 4.47, 7]] type='15 * var * float64'>
b_comp     = <Array [[7.21, 5.83, 5.83], [], ..., [5, 4.47, 7]] type='15 * var * float64'>

[jpivarski]

Either that or radically different contents somewhere in b[2:-1] vs b_comp[2:-1].

Only on Python 3.12 on MacOS... On a comparison that was already approximate... Maybe some binary change nudged it over the threshold.

This is also marked as xfail with BAD_NP_AK_MIXIN_VERSIONING. I don't know why it's being marked as a failure if it's xfailed.

[jpivarski]

Since the test failure doesn't seem to be reproduced, just running it again, here's its output, which we can revisit in the future if we see that failure again:

=================================== FAILURES ===================================
____________________________ test_distance_behavior ____________________________

daa_p1 = dask.awkward<from-json-files, npartitions=3>
daa_p2 = dask.awkward<from-json-files, npartitions=3>
caa_p1 = <Array [{points: [{...}, ...]}, ..., {...}] type='15 * {points: var * {x: i...'>
caa_p2 = <Array [{points: [{...}, ...]}, ..., {...}] type='15 * {points: var * {x: i...'>

    @pytest.mark.xfail(
        BAD_NP_AK_MIXIN_VERSIONING,
        reason="NumPy 1.25 mixin __slots__ change",
    )
    def test_distance_behavior(
        daa_p1: dak.Array,
        daa_p2: dak.Array,
        caa_p1: ak.Array,
        caa_p2: ak.Array,
    ) -> None:
        daa1 = dak.with_name(daa_p1.points, name="Point", behavior=behaviors)
        daa2 = dak.with_name(daa_p2.points, name="Point", behavior=behaviors)
        caa1 = ak.Array(caa_p1.points, with_name="Point", behavior=behaviors)
        caa2 = ak.Array(caa_p2.points)
>       assert_eq(daa1.distance(daa2), caa1.distance(caa2))

caa1       = <PointArray [[{x: 6, y: 5}, {...}, {...}], ...] type='15 * var * Point[x: i...'>
            a_comp_form = a_comp.layout.form
            b_comp_form = b_comp.layout.form
            assert a_comp_form == a_form
            assert a_comp_form == b_form
            assert b_comp_form == a_comp_form
    
        if check_divisions:
            # check divisions if both collections
            if a_is_coll and b_is_coll:
                if a.known_divisions and b.known_divisions:
                    assert a.divisions == b.divisions
                else:
                    assert a.npartitions == b.npartitions
    
        # finally check the values
        if isclose_equal_nan:
            assert ak.all(ak.isclose(a_comp, b_comp, equal_nan=True))
        else:
            if convert_to_lists:
                assert a_comp.tolist() == b_comp.tolist()
            else:
>               assert ak.almost_equal(a_comp, b_comp, dtype_exact=True)
E               AssertionError

a          = dask.awkward<distance, npartitions=3>
a_comp     = <Array [[0, 0, 0], [], [0, ...], ..., [0], [0, 0, 0]] type='15 * var * float64'>
a_is_coll  = True
a_tt       = <Array-typetracer [...] type='## * var * float64'>
b          = <Array [[7.21, 5.83, 5.83], [], ..., [5, 4.47, 7]] type='15 * var * float64'>
b_comp     = <Array [[7.21, 5.83, 5.83], [], ..., [5, 4.47, 7]] type='15 * var * float64'>
b_is_coll  = False
b_tt       = <Array-typetracer [...] type='## * var * float64'>
check_divisions = True
check_forms = False
convert_to_lists = False
isclose_equal_nan = False
scheduler  = 'sync'

/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/dask_awkward/lib/testutils.py:99: AssertionError
------------------------------ Captured log setup ------------------------------
DEBUG    fsspec.local:local.py:314 open file: /private/var/folders/n2/pt_35rc53tdgkld9531s2tfh0000gn/T/pytest-of-runner/pytest-0/data0/points_ndjson1.json
DEBUG    fsspec.local:local.py:314 open file: /private/var/folders/n2/pt_35rc53tdgkld9531s2tfh0000gn/T/pytest-of-runner/pytest-0/data0/points_ndjson1.json
DEBUG    fsspec.local:local.py:314 open file: /private/var/folders/n2/pt_35rc53tdgkld9531s2tfh0000gn/T/pytest-of-runner/pytest-0/data1/points_ndjson2.json
DEBUG    fsspec.local:local.py:314 open file: /private/var/folders/n2/pt_35rc53tdgkld9531s2tfh0000gn/T/pytest-of-runner/pytest-0/data1/points_ndjson2.json
------------------------------ Captured log call -------------------------------
DEBUG    fsspec.local:local.py:314 open file: /private/var/folders/n2/pt_35rc53tdgkld9531s2tfh0000gn/T/pytest-of-runner/pytest-0/data0/points_ndjson1.json
DEBUG    dask_awkward.lib.io.json:json.py:122 columns read from disk: ['points.x', 'points.y']
DEBUG    fsspec.local:local.py:314 open file: /private/var/folders/n2/pt_35rc53tdgkld9531s2tfh0000gn/T/pytest-of-runner/pytest-0/data0/points_ndjson1.json
DEBUG    dask_awkward.lib.io.json:json.py:122 columns read from disk: ['points.x', 'points.y']
DEBUG    fsspec.local:local.py:314 open file: /private/var/folders/n2/pt_35rc53tdgkld9531s2tfh0000gn/T/pytest-of-runner/pytest-0/data0/points_ndjson1.json
DEBUG    dask_awkward.lib.io.json:json.py:122 columns read from disk: ['points.x', 'points.y']

</details>

[jpivarski]

The dak.to_list implementation looks good! I can't check ✔️ approved! because I started this PR.

However, I'd say it's ready to be merged.

[jpivarski]

This was here to diagnose the test failure.

But now it looks like the test passed, just having been run a second time. (Usually, not a good thing!)

[martindurant]

(we may come back to check on that array comparison, maybe floating point inaccuracy)

[agoose77]

@jpivarski I wonder, should this be lazy and return a bag of lists?

[martindurant]

We chatted in the meeting and thought not - this is only likely to be used in testing for small data.

[jpivarski]

Yeah, if I called this function, I wouldn't be interested in lazy data. The use-case that came up here was that I wanted to debug a test failure.