feat: allow fetching other datasets (obf, opff, opf) #223
Conversation
raphodn
force-pushed
the
product-dataset-flavor
branch
from
f7ab588
to
2d113d0
Compare
| }, | ||
| Flavor.obf: { | ||
| DatasetType.jsonl: "openbeautyfacts-products.jsonl.gz", | ||
| DatasetType.csv: "en.openbeautyfacts.org.products.csv", |
There was a problem hiding this comment.
the obf, opff & opf datasets don't seem to have .csv.gz files (getting 404).
so I used plain .csv instead
There was a problem hiding this comment.
obf, opff, opf use an old version of Product Opener so this is expected. The migration to a more recent version is in progress (but it needs quite a lot of work).
raphodn
added
✨ enhancement
🐍 Python
| @@ -57,7 +73,12 @@ def get_dataset( | |||
|
|
|||
|
|
|||
| class ProductDataset: | |||
| def __init__(self, dataset_type: DatasetType = DatasetType.jsonl, **kwargs): | |||
| def __init__( | |||
There was a problem hiding this comment.
slight breaking change, as the documentation gives an explicit usage example with ProductDataset("csv"), which will not work anymore
// instead
ProductDataset(dataset_type="csv")
// or
from openfoodfacts.types import Flavor
ProductDataset(flavor=Flavor.off, dataset_type="csv")
raphodn
force-pushed
the
product-dataset-flavor
branch
from
2d113d0
to
7862959
Compare
|
|
My bad, it's now functional! |
Description
The current
ProductDatasetimplementation only allows to fetch the OFF (food) dataSolution
ProductDatasetto allow fetching other datasets: OBF, OPFF, OPFRelated issue(s)