Continue parsing after an using ijson.items on an array

[jnturton]

Description

I want to parse an array using ijson.items, stop on StopIteration, and then proceed to parse trailing content using the underlying ijson.parse object. In the code below, the final line fails. If I replace while True with for i in range(3) then I parse all of arr without a StopIteration and I can go on to parse the trailing content. This doesn't help me though, because I don't know how long arr is...

Thank you for you help.

import io
import ijson

parse_events = ijson.parse(io.BytesIO(b'''
{
  "leading": "hi",
  "arr": [ 1, 2, 3 ],
  "trailing": "bye"
}
'''))

# not shown: iterate over parse_events to process "leading"

arr_iter = ijson.items(parse_events, 'arr.item')

try:
    while True:
        print(next(arr_iter))
except StopIteration:
    print('Caught StopIteration')

next(parse_events) # I want to read "trailing" now but get StopIteration

[rtobar]

@dzamo what you are asking is basically not possible with the code you came up with.

When the ijson generators raise a StopIteration exception it means they exhausted their input, which in your case means the parse_events generator was exhausted, which in turn means the input data was fully parsed already.

I think your confusion comes from a wrong (but common) expectation as well: you probably think arr will appear only once in your document, and therefore expected ijson.items to finish doing its work once the array finished. In JSON documents keys are allowed to be repeated, and therefore ijson can't stop processing the document just before the key containing the array you are iterating over has finished. For example:

$> echo '{"arr": [1, 2, 3], "arr": [1, 2, 3]}' | python -m ijson.dump -m items -p arr.item
#: value
--------
0: 1
1: 2
2: 3
3: 1
4: 2
5: 3

There are more reasons, but those are enough I think to explain the situation.

To implement what you need you'll have to do all your parsing with the results from items.parse. To construct the objects within your arr array you can take inspiration on how items is inspired (see here) and adapt it to process a single array.

[jnturton]

@rtobar: thank you for the pointer, your suggestion works nicely. I've now got a generator called items_once which is the same as ijson.items but it only parses the next encountered occurrence of a prefix. Perhaps this would be useful addition to the ijson API? Either way, I'll leave my code here in case other users have a similar need.

def _items_once(event_stream, prefix):
    '''
    Generator dispatching native Python objects constructed from the ijson
    events under the next occurrence of the given prefix.  It is very
    similar to ijson.items except that it will not consume the entire JSON
    stream looking for occurrences of prefix, but rather stop after
    completing the next encountered occurrence of prefix.
    '''
    current = None
    while current != prefix:
        current, event, value = next(event_stream)

    while current == prefix:
        if event in ('start_map', 'start_array'):
            object_depth = 1
            builder = ObjectBuilder() # imported from ijson.common
            while object_depth:
                builder.event(event, value)
                current, event, value = next(event_stream)
                if event in ('start_map', 'start_array'):
                    object_depth += 1
                elif event in ('end_map', 'end_array'):
                    object_depth -= 1
            del builder.containers[:]
            yield builder.value
        else:
            yield value

        current, event, value = next(event_stream)    

[rtobar]

Thanks @dzamo for putting the code up, I'm sure somebody else will find it useful too.

I'm reluctant to add something like this to ijson. Not because it's a bad idea on itself, but because it's a bit of a niche use case, and adding it (and maintaining it in the longer term) is a bit more than copy-pasting your code. In particular, the yajl2_c extension re-implements everything in C, so there's always that duplication to take care of.

If more and more people are finding this to be useful we could re-evaluate; otherwise having the code available here is good enough I think. Thanks again!