Default JSONEncoder won't serialize iterators, can't easily be extended.

[etrepum]

http://code.google.com/p/simplejson/issues/detail?id=88

Reported by i...@simplegeo.com, Jan 17, 2011
What steps will reproduce the problem?

1. simplejson.dumps(iter(range(10)))

What is the expected output? What do you see instead?
The iterator should generate a JSON serialized list.

This is a somewhat thorny issue. I have a large dataset which is lazily loaded in via a generator. I'd like to avoid materializing the entire dataset before serializing, ideally by having SimpleJSON consume from my input generator and write to a FD via dump().

The extension mechanism via `default=some_method' doesn't work, because I need to yield individual elements from the seq, not a materialized seq. The existing _iterencode_list() method is totally capable of serializing lazy seqs, but _iterencode() lacks the necessary conditional to invoke it for those types — and the number of possible types is extremely large. Every method in itertools, for example, returns a different type.

I was able to make this work with stdlib's json module, though it was not pretty. I had to subclass JSONEncoder, copy-paste _iterencode() in, and add the necessary conditional to send lazy seqs through _iterencode_list(). This approach is significantly more painful with SimpleJSON, since _iterencode() and friends are now hidden inside _make_iterencode(), where they can't be touched — I'd have to copy-paste the entire method (and its inner methods) to add one more conditional.

I understand that there's going to be some level of heuristic to detect these types, due to the lack of a shared base type for lazy seqs, but is there not some better way to handle this?

[etrepum]

https://github.com/simplejson/simplejson/tree/iterable_as_array-gh1

Does this branch solve the problem sufficiently? Note that you'll still end up materializing all of the output before returning... unless you create a JSONEncoder and call iterencode(), which will use a slower Python implementation but give you "flow control".

[pferreir]

+1

[etrepum]

Does your "+1" mean that you've tried the branch and it solves a problem you have?

[pferreir]

No, it was +1 for the issue, but I will give it a try.

[etrepum]

Since nobody seems to care, I'm going to retire this issue and leave the branch in limbo.

[nickbabcock]

I realize that this issue is very old and has been closed for a long time, but I tried out the branch and it worked swimmingly. Combining iterencode with iterable_as_array=True, allowed for me to stream json output of several gigabytes while using under 20KB of memory without resorting to the Stackoverflow hack that I'm uncomfortable with (an iterator masquerading as a list!)

[etrepum]

@nickbabcock Would you be interested in porting this branch back to the current master for a pull request?

[nickbabcock]

Yes, no problem. I'll create a pull request for review when I'm done.