[MRG+1] Add set serialization to ScrapyJSONEncoder #2058
I think this change is fine.
The problem with set serialization is that you get a list back, which has a different time complexity at lookups. In your example after duplicate check will run in O(1) before the change, and in O(N) after. But for items export it looks fine.
@redapple Yes, just haven't had much spare time recently.
@kmike On a more general sense there's no way to serialize to JSON and get a set back when deserializing, so that's probably why this feature is not already in the json module. I added this PR because using a set to eliminate duplicates is a common python pattern and thinking of items export as the intended use case.