[WIP] Split large files with ijson #366
Conversation
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
…e into 154-fix-buenos-aires Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
…e into 154-fix-buenos-aires Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
…e into 154-fix-buenos-aires
| package = self.get_package(f_package, array_field_name) | ||
|
|
||
| for number, items in enumerate(util.grouper(ijson.items(f_list, '{}.item'.format(array_field_name)), size), 1): | ||
| package[array_field_name] = [item for item in items if item is not None] |
There was a problem hiding this comment.
In what case is an item None? You can also do filter(None, items).
There was a problem hiding this comment.
@yolile Can you answer this question? Otherwise, PR looks good.
There was a problem hiding this comment.
Sorry, I forgot about this one! the grouper method returns nulls values to complete the number of items set at 'size' parameter
kingfisher_scrapy/util.py
Outdated
| import itertools | ||
| import json | ||
| from decimal import Decimal | ||
|
|
||
| from ijson import utils, ObjectBuilder |
There was a problem hiding this comment.
Run isort -w 119 to fix the test failure.
There was a problem hiding this comment.
Also, don't forget to add ijson>=3 to requirements.in and run:
pip-compile; pip-compile requirements_dev.in
There was a problem hiding this comment.
I got errors running pip-compile using pip 20.1, the current pip-compile version doesn't support pip>=20, the 5.5 version does. Besides, can we document these development processes somewhere?
There was a problem hiding this comment.
See https://ocds-standard-development-handbook.readthedocs.io/en/latest/coding/index.html#id1
I upgraded pip-tools in a new commit.
There was a problem hiding this comment.
great! I thought that only the documentation about the standard development was there. Should we add a reference to that in kingfisher somewhere?
There was a problem hiding this comment.
I opened an issue to discuss what to do with that content: open-contracting/standard-development-handbook#225
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
ref #154