Awkward v2 transition #736

lgray · 2022-10-25T22:37:44Z

lgray · 2022-11-06T15:41:38Z

@kratsg hey - what does a form key that looks like:

Gettting ( ad4cd5ec-123e-11ec-92f6-93e3aac0beef/%2FDelphes%3B1/0-25/data/GenJet%2FGenJet.SoftDroppedJet%2C%21load%2C%21content%2C%40instance_version%2C%21item ) : ad4cd5ec-123e-11ec-92f6-93e3aac0beef /Delphes;1 0 25 ['GenJet/GenJet.SoftDroppedJet', '!load', '!content', '@instance_version', '!item']

mean? In particular the @instance_version token? The internal structure of RecordArray forms changed quite a bit in awkwardv2 and nanoevents makes quite a few assumptions.

kratsg · 2022-11-07T16:17:12Z

In particular the @instance_version token? The internal structure of RecordArray forms changed quite a bit in awkwardv2 and nanoevents makes quite a few assumptions

Checking, but I don't recall an instance_version being added...

lgray · 2022-11-07T16:46:24Z

Gotcha - things broke in interesting ways - there's only so much of this code nick/myself actually own and know how to maintain! Thanks for looking.

lgray · 2022-12-07T21:49:30Z

@gordonwatts You should be aware that things have changed a lot with forms under the hood in awkward2. I have gotten your tests to pass again for the auto schema but there is not much confirmation that they work beyond that. @'ing you since it would be worth your time to work the rest of this out.

I'm not sure if the servicex tests are awkward2 compatible yet either.

lgray · 2022-12-07T21:56:40Z

@tejinc awkward2 brings quite some changes to the inner workings of coffea. I have gotten your nanoevents schema (pdune) to pass tests but you may want to check it to ensure proper functionality is kept.

lgray · 2022-12-07T22:45:27Z

@kpedro88 awkward2 brings quite some changes to the inner workings of coffea. I have gotten your nanoevents schema (treemaker) to pass your tests but you may want to check it to ensure proper functionality is kept.

kpedro88 · 2022-12-07T22:59:42Z

@kpedro88 awkward2 brings quite some changes to the inner workings of coffea. I have gotten your nanoevents schema (treemaker) to pass your tests but you may want to check it to ensure proper functionality is kept.

@yimuchen @cmadrid1 @Keane-Tan FYI

lgray · 2022-12-08T18:52:43Z

@nikoladze awkward2 brings quite some changes to the inner workings of coffea. I have gotten your nanoevents schema (physlite) to pass your tests but you may want to check it to ensure proper functionality is kept.

lgray · 2022-12-08T22:40:39Z

@kratsg awkward2 brings quite some changes to the inner workings of coffea. I have gotten your nanoevents schema (delphes) to pass your tests but you may want to check it to ensure proper functionality is kept.

kratsg · 2022-12-08T23:16:31Z

@kratsg awkward2 brings quite some changes to the inner workings of coffea. I have gotten your nanoevents schema (delphes) to pass your tests but you may want to check it to ensure proper functionality is kept.

Checked it and it seemed ok out of the box? It at least works for me for the quick tests I ran so I think it's ok.

EDIT: works on some other older Delphes files I had too that crashed with coffea before... so I think the changes are going to give us better support(?!). Thanks for all the work!

yimuchen · 2022-12-12T17:07:55Z

TreeMakerSchema looks good. But found another bug when loading in a large number events:

events = NanoEventsFactory.from_root("file:mytestfile.root", entry_start=0, entry_stop=100, schemaclass=TreeMakerSchema, treepath='TreeMaker2/PreSelection').events() # This works fine
events = NanoEventsFactory.from_root("file:mytestfile.root", entry_start=0, entry_stop=1000, schemaclass=TreeMakerSchema, treepath='TreeMaker2/PreSelection').events() # This errors

Traceback dump:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "coffeaenv/lib/python3.8/site-packages/coffea/nanoevents/factory.py", line 399, in events
    events = awkward.from_buffers(
  File "coffeaenv/lib/python3.8/site-packages/awkward/operations/ak_from_buffers.py", line 76, in from_buffers
    return _impl(
  File "coffeaenv/lib/python3.8/site-packages/awkward/operations/ak_from_buffers.py", line 121, in _impl
    out = reconstitute(form, length, container, getkey, backend, simplify)
  File "coffeaenv/lib/python3.8/site-packages/awkward/operations/ak_from_buffers.py", line 296, in reconstitute
    contents = [
  File "coffeaenv/lib/python3.8/site-packages/awkward/operations/ak_from_buffers.py", line 297, in <listcomp>
    reconstitute(content, length, container, getkey, backend, simplify)
  File "coffeaenv/lib/python3.8/site-packages/awkward/operations/ak_from_buffers.py", line 274, in reconstitute
    content = reconstitute(
  File "coffeaenv/lib/python3.8/site-packages/awkward/operations/ak_from_buffers.py", line 296, in reconstitute
    contents = [
  File "coffeaenv/lib/python3.8/site-packages/awkward/operations/ak_from_buffers.py", line 297, in <listcomp>
    reconstitute(content, length, container, getkey, backend, simplify)
  File "coffeaenv/lib/python3.8/site-packages/awkward/operations/ak_from_buffers.py", line 274, in reconstitute
    content = reconstitute(
  File "coffeaenv/lib/python3.8/site-packages/awkward/operations/ak_from_buffers.py", line 296, in reconstitute
    contents = [
  File "coffeaenv/lib/python3.8/site-packages/awkward/operations/ak_from_buffers.py", line 297, in <listcomp>
    reconstitute(content, length, container, getkey, backend, simplify)
  File "coffeaenv/lib/python3.8/site-packages/awkward/operations/ak_from_buffers.py", line 148, in reconstitute
    data = backend.nplike.frombuffer(raw_array, dtype=dtype, count=real_length)
  File "coffeaenv/lib/python3.8/site-packages/awkward/_nplikes.py", line 115, in frombuffer
    return self._module.frombuffer(*args, **kwargs)
ValueError: buffer is smaller than requested size

lgray · 2022-12-12T17:16:23Z

@yimuchen thanks! what's the size of the file you're testing against?

yimuchen · 2022-12-12T17:20:20Z

@yimuchen thanks! what's the size of the file you're testing against?

This is a 2GB file containing ~60K events. So about 35KB (?) per event.

coffea/nanoevents/methods/candidate.py

coffea/lookup_tools/lookup_base.py

lgray · 2022-12-12T19:42:09Z

@yimuchen can you try seeing if it's a particular event that causes problems, or if it's just really the size of the event range you want to pick up?

lgray · 2022-12-12T23:24:05Z

Note to self: python 3.11 is a limitation of at least numba

yimuchen · 2022-12-13T13:13:01Z

@yimuchen can you try seeing if it's a particular event that causes problems, or if it's just really the size of the event range you want to pick up?

Tried a few points, it looks like if the event size fails for some slice size (seems fail at about ~400 events in this case), it would consistently fail. So it isn't a case of a single very large event.

lgray · 2022-12-13T14:49:36Z

@yimuchen so it's really just the absolute size of the slice? or where you are in the file? or both? It may be useful to scan through the file 100 events at a time to make sure there's not some pattern that's bugging out.

nsmith- · 2022-12-22T01:46:00Z

coffea/nanoevents/mapping/uproot.py

-                form["contents"][field], prefix + f",{field},!item"
-            )
+            elif field.startswith("@"):
+                # workaround uproot5 bug


Bug is fixed in scikit-hep/uproot5#808

lgray · 2022-12-28T05:35:03Z

Tests will be broken for a bit as I get things daskified.

for more information, see https://pre-commit.ci

lgray · 2023-03-16T13:57:21Z

@nikoladze I could use your help in daskifying PHYSLITESchema. Something is going wrong in zipping the forms but everything looks more or less OK.

I think if that can be fixed up it'll more or less work. You'll need to use uproot main to get it to run in the first place.

lgray · 2023-03-16T14:42:17Z

@kratsg I think I did this correctly. Can you please check the usage of NanoEvents Delphes with dask_awkward? I have no idea what correct use really looks like.

nikoladze · 2023-03-18T12:27:16Z

@nikoladze I could use your help in daskifying PHYSLITESchema. Something is going wrong in zipping the forms but everything looks more or less OK.

I think if that can be fixed up it'll more or less work. You'll need to use uproot main to get it to run in the first place.

Thanks a lot for putting all this effort into this! See #776 for a fix for the form zipping.

fix form zipping for physlite schema

lgray · 2023-03-19T22:57:15Z

@nikoladze there's still some problems in the rendered arrays with the trackParticleLinks - look at the test failures. :-/

Ah - I see it's passing around some dask_awkward stuff within all your offsets functions. I'll get to it in time but it's a fair amount of rather specific code, it would be best for you to figure out where to put in the map_partitions calls and such.

Basically the big change with dask_awkward is that we can only use the factory-reference-through-behavior trick on the client side, so your arrays have to render to something that doesn't require reindexing later.

lgray · 2023-03-25T15:22:01Z

These latest commits require to be-released features in dask-awkward.

lgray · 2023-04-06T00:23:29Z

OK so what we're gonna do is merge this puppy and start release candidates. There is a very short list of things that need doing to get a full release on the coffea side.

@nikoladze @kratsg I will need your help to fix your schemas.

Parquet nanoevents will have to wait for a better implementation of form-remapping in dask-awkward (which should also tidy up some code in uproot). @martindurant @douglasdavis

I will reformulate the task list from this PR into the remaining tasks that need doing, and release an RC for people to start exploring what's new.

lgray · 2023-04-06T00:31:25Z

@btovar @dthain we should talk about workqueue + dask = taskvine? as well, times are a-changin'.

lgray force-pushed the awkward2_dev branch 2 times, most recently from f6d6d30 to 18765f6 Compare November 4, 2022 21:59

lgray force-pushed the awkward2_dev branch 2 times, most recently from 3147db7 to 3927dd3 Compare December 7, 2022 20:33

agoose77 reviewed Dec 12, 2022

View reviewed changes

coffea/nanoevents/methods/candidate.py Outdated Show resolved Hide resolved

coffea/lookup_tools/lookup_base.py Outdated Show resolved Hide resolved

coffea/lookup_tools/lookup_base.py Outdated Show resolved Hide resolved

lgray force-pushed the awkward2_dev branch 3 times, most recently from 2103840 to 2bde16c Compare December 12, 2022 21:52

lgray force-pushed the awkward2_dev branch 2 times, most recently from f347ada to 38dd522 Compare December 13, 2022 01:02

jpivarski mentioned this pull request Dec 14, 2022

Support for dask-awkward arrays scikit-hep/vector#301

Closed

nsmith- reviewed Dec 22, 2022

View reviewed changes

lgray and others added 4 commits March 15, 2023 01:56

starting to work on daskifying treemaker

7fabd3f

[pre-commit.ci] auto fixes from pre-commit.com hooks

a539c33

for more information, see https://pre-commit.ci

get the dask-nanoevents interface looking similar to the old one

5e3ce4d

Weights is daskified; various rebase cruft for jetmet tools

ff251c7

lgray force-pushed the awkward2_dev branch from 224e00b to ff251c7 Compare March 15, 2023 01:05

lgray added 4 commits March 16, 2023 11:11

more efficient distribution strategy for heavy lookups, first try

4ce7c3f

faster tokenization of corrections in local mode

e650361

treemaker is daskified

0791991

pin to awkward > 2.1.0; switch to calver

81cc052

lgray added 2 commits March 16, 2023 14:58

getting closer with PHYSLITE

d3a65a5

delphes is probably daskified

36bf4c6

fix form zipping for physlite schema

a924d7d

lgray added 2 commits March 18, 2023 18:03

Merge pull request #776 from nikoladze/awkward2_dev-fix-physlite-zip

0380f88

fix form zipping for physlite schema

Merge branch 'master' into awkward2_dev

43b3ea7

lgray added 2 commits March 24, 2023 14:29

parquet nanoevents + work on physlite

6c6d5a1

ressurect parquet nanoevents and associated tests

192ee34

lgray added 5 commits April 4, 2023 12:22

now sunder for array context

91c2038

xfail tests that need it for now

6131b48

2023.4.rc0

4e958e4

Merge branch 'master' into awkward2_dev

b059111

more xfails, these are heisenbugs...

d17510b

lgray merged commit 966c94d into master Apr 6, 2023
13 checks passed

lgray deleted the awkward2_dev branch June 26, 2023 16:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Awkward v2 transition #736

Awkward v2 transition #736

lgray commented Oct 25, 2022 •

edited

lgray commented Nov 6, 2022

kratsg commented Nov 7, 2022

lgray commented Nov 7, 2022

lgray commented Dec 7, 2022

lgray commented Dec 7, 2022 •

edited

lgray commented Dec 7, 2022 •

edited

kpedro88 commented Dec 7, 2022

lgray commented Dec 8, 2022 •

edited

lgray commented Dec 8, 2022 •

edited

kratsg commented Dec 8, 2022 •

edited

yimuchen commented Dec 12, 2022

lgray commented Dec 12, 2022

yimuchen commented Dec 12, 2022 •

edited

lgray commented Dec 12, 2022

lgray commented Dec 12, 2022

yimuchen commented Dec 13, 2022

lgray commented Dec 13, 2022

nsmith- Dec 22, 2022

lgray commented Dec 28, 2022

lgray commented Mar 16, 2023 •

edited

lgray commented Mar 16, 2023

nikoladze commented Mar 18, 2023

lgray commented Mar 19, 2023 •

edited

lgray commented Mar 25, 2023

lgray commented Apr 6, 2023 •

edited

lgray commented Apr 6, 2023

Awkward v2 transition #736

Awkward v2 transition #736

Conversation

lgray commented Oct 25, 2022 • edited

lgray commented Nov 6, 2022

kratsg commented Nov 7, 2022

lgray commented Nov 7, 2022

lgray commented Dec 7, 2022

lgray commented Dec 7, 2022 • edited

lgray commented Dec 7, 2022 • edited

kpedro88 commented Dec 7, 2022

lgray commented Dec 8, 2022 • edited

lgray commented Dec 8, 2022 • edited

kratsg commented Dec 8, 2022 • edited

yimuchen commented Dec 12, 2022

lgray commented Dec 12, 2022

yimuchen commented Dec 12, 2022 • edited

lgray commented Dec 12, 2022

lgray commented Dec 12, 2022

yimuchen commented Dec 13, 2022

lgray commented Dec 13, 2022

nsmith- Dec 22, 2022

Choose a reason for hiding this comment

lgray commented Dec 28, 2022

lgray commented Mar 16, 2023 • edited

lgray commented Mar 16, 2023

nikoladze commented Mar 18, 2023

lgray commented Mar 19, 2023 • edited

lgray commented Mar 25, 2023

lgray commented Apr 6, 2023 • edited

lgray commented Apr 6, 2023

lgray commented Oct 25, 2022 •

edited

lgray commented Dec 7, 2022 •

edited

lgray commented Dec 7, 2022 •

edited

lgray commented Dec 8, 2022 •

edited

lgray commented Dec 8, 2022 •

edited

kratsg commented Dec 8, 2022 •

edited

yimuchen commented Dec 12, 2022 •

edited

lgray commented Mar 16, 2023 •

edited

lgray commented Mar 19, 2023 •

edited

lgray commented Apr 6, 2023 •

edited