ijson.parse iter_lines() returns error too many values to unpack (expected 2)

[raeldor]

Me again, sorry. I'm trying to stream data from a web call directly into ijson. However, if I use...

parser=ijson.parse(cellset_response.iter_lines())
for prefix, event, value in parser:
pass

I get 'too many values to unpack (expected 2)' error. If I use iter_content() instead of iter_lines() I get 'not enough values to unpack (expected 2, got 1).

It seems I can't win. :D

[raeldor]

Just to add a bit more detail, iter_lines() seems to give the entire stream as a binary string since the JSON is not formatted with newline characters. iter_content() gives the data in a stream format returning what looks like 1 or 2 binary characters at a time.

I thought ijson was able to handle a stream. Surely that stream doesn't need to be formatted?

[rtobar]

@raeldor my previous advice was to build something around iter_content, not to pass it (or iter_lines) directly with ijson. As mentioned before, ijson works with file-like objects; that is, objects with a read method, so you'd need to wrap the response to make it look like a file that can be read.

Something like this:

import ijson
import requests

class ResponseAsFileObject:
    def __init__(self, url):
        response = requests.get(url, stream=True)
        self.data = response.iter_content(chunk_size=65536)
    def read(self, n):
        if n == 0:
            return b''
        return next(self.data, b'')

URL = 'your-url'
for prefix, event, name in ijson.parse(ResponseAsFileObject(URL)):
    print(prefix, event, name)

[rtobar]

@raeldor please let me know if this worked so we can close this.

[rtobar]

There hasn't been a response in a week, and the solution worked for me, so I'm closing this.

[raeldor]

Apologies. I meant you get round to closing this and totally forgot. Really appreciate your help. Thank you. Ray

…

On Sep 19, 2021, at 23:33, rtobar ***@***.***> wrote:  There hasn't been a response in a week, and the solution worked for me, so I'm closing this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.