-
-
Notifications
You must be signed in to change notification settings - Fork 353
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: USDA API import to a list of products missing in OFF #9083
Open
rkiddy
wants to merge
9
commits into
openfoodfacts:main
Choose a base branch
from
rkiddy:usda_import_via_api
base: main
Could not load branches
Branch not found: {{ refName }}
Could not load tags
Nothing to show
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 5 commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
a1d543f
Start something. From a list of products missing in OFF, start to cre…
rkiddy 252169f
Merge branch 'main' of github.com:openfoodfacts/openfoodfacts-server …
rkiddy 34ae37a
Tests include one created food.
rkiddy 3d6a96a
Merge branch 'main' of github.com:openfoodfacts/openfoodfacts-server …
rkiddy b7599e3
First version of tool to check USDA importability.
rkiddy 71bf631
chore: add usda - off fields correspondance as json
alexgarel 00551b6
Merge branch 'main' of github.com:openfoodfacts/openfoodfacts-server …
rkiddy e90b30f
Including requested barcodes as potential tests and adding column dat…
rkiddy 0ee12e4
Resolve differences in usda_to_off_fields.json
rkiddy File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,240 @@ | ||
| fdc_category,number_of_products,category,To create | ||
| "Popcorn, Peanuts, Seeds & Related Snacks",15869,en:Snacks, | ||
| Candy,15053,en:Confectioneries, | ||
| Cheese,13788,en:Cheeses, | ||
| Ice Cream & Frozen Yogurt,10990,en:Frozen desserts, | ||
| Cookies & Biscuits,10239,en:Biscuits, | ||
| Breads & Buns,8479,en:Breads, | ||
| "Pickles, Olives, Peppers & Relishes",8417,en:Salted snacks, | ||
| "Chips, Pretzels & Snacks",8203,en:Snacks, | ||
| "Fruit & Vegetable Juice, Nectars & Fruit Drinks",7870,en:Plant-based beverages, | ||
| Chocolate,7038,en:Chocolate candies, | ||
| Cereal,7026,en:cereals, | ||
| "Snack, Energy & Granola Bars",6731,en:Snacks, | ||
| "Cakes, Cupcakes, Snack Cakes",6717,en:Cakes, | ||
| "Seasoning Mixes, Salts, Marinades & Tenderizers",6145,en:Condiments, | ||
| Soda,6132,en:Sodas, | ||
| Yogurt,5977,en:Yogurts, | ||
| "Ketchup, Mustard, BBQ & Cheese Sauce",5937,en:Sauces, | ||
| Pasta by Shape & Type,5561,en:pastas, | ||
| Dips & Salsa,5440,en:Dips, | ||
| Salad Dressing & Mayonnaise,5395,en:Sauces, | ||
| Other Snacks,4974,en:Snacks, | ||
| Wholesome Snacks,4970,en:Snacks, | ||
| Water,4741,en:Waters, | ||
| Frozen Dinners & Entrees,4263,en:Frozen foods, | ||
| Canned Vegetables,4061,en:Canned vegetables, | ||
| Pre-Packaged Fruit & Vegetables,4028,en:Fruits and vegetables based foods, | ||
| "Pepperoni, Salami & Cold Cuts",3884,en:Prepared meats, | ||
| Frozen Appetizers & Hors D'oeuvres,3877,en:Frozen foods, | ||
| Frozen Vegetables,3596,en:Frozen vegetables, | ||
| Vegetable & Cooking Oils,3574,en:Vegetable oils, | ||
| "Sausages, Hotdogs & Brats",3408,en:Sausages, | ||
| Milk,3367,en:Milks, | ||
| Crackers & Biscotti,3318,en:Biscuits and cakes, | ||
| Pizza,3176,en:Pizzas, | ||
| Prepared Pasta & Pizza Sauces,3099,en:Sauces, | ||
| Baking Decorations & Dessert Toppings,3057,en:Baking decorations,en:Baking decorations | ||
| Other Drinks,2983,en:Beverages, | ||
| Canned Fruit,2941,en:Canned fruits, | ||
| "Oriental, Mexican & Ethnic Sauces",2907,en:Sauces, | ||
| Powdered Drinks,2878,en:Dehydrated beverages, | ||
| Canned & Bottled Beans,2752,en:Canned common beans, | ||
| "Jam, Jelly & Fruit Spreads",2727,en:Fruit preserves, | ||
| Frozen Fish & Seafood,2726,en:Frozen seafood, | ||
| Other Soups,2656,en:Soups, | ||
| Tomatoes,2599,en:Tomatoes, | ||
| "Cake, Cookie & Cupcake Mixes",2533,en:Cake mixes, | ||
| "Croissants, Sweet Rolls, Muffins & Other Pastries",2231,en:Pastries, | ||
| Nut & Seed Butters,2199,en:Vegetable fats, | ||
| Canned Soup,2181,en:Canned soups, | ||
| Frozen Fruit & Fruit Juice Concentrates,2002,en:Fruits based foods, | ||
| Other Cooking Sauces,1783,en:Sauces, | ||
| Iced & Bottle Tea,1674,en:Iced teas, | ||
| Herbs & Spices,1654,"en:Plant-based foods, en:Condiments", | ||
| Other Meats,1630,en:Meats, | ||
| Vegetable and Lentil Mixes,1562,en:Vegetable mixes,en:Vegetable mixes | ||
| Butter & Spread,1560,en:Fats, | ||
| Rice,1484,en:Rices, | ||
| Syrups & Molasses,1452,en:Simple syrups, | ||
| "Granulated, Brown & Powdered Sugar",1444,en:Sugars, | ||
| Other Deli,1427,, | ||
| "Energy, Protein & Muscle Recovery Drinks",1417,en:Beverages, | ||
| Milk Additives,1395,en:Coffee creamers,en:Coffee creamers | ||
| Tea Bags,1355,en:Tea bags, | ||
| Other Frozen Desserts,1347,en:Frozen desserts, | ||
| Mexican Dinner Mixes,1266,en:Mexican Dinner Mixes,en:Mexican Dinner Mixes | ||
| Non Alcoholic Beverages Ready to Drink,1254,en:Non-Alcoholic beverages, | ||
| Pasta Dinners,1237,en:Pasta dishes, | ||
| Crusts & Dough,1222,en:Pie dough, | ||
| "Bacon, Sausages & Ribs",1218,en:Prepared meats, | ||
| Cream,1198,en:Creams, | ||
| Biscuits/Cookies,1187,en:Biscuits, | ||
| Honey,1155,en:Honeys, | ||
| Chewing Gum & Mints,1136,en:Confectioneries, | ||
| Plant Based Milk,1129,en:Plant milks, | ||
| Chili & Stew,1122,en:Stews, | ||
| Canned Seafood,1105,"en:Canned foods, en:Seafoods", | ||
| Snacks,1099,en:Snacks, | ||
| Puddings & Custards,1040,en:Desserts, | ||
| Biscuits/Cookies (Shelf Stable),1030,en:Biscuits, | ||
| Flours & Corn Meal,1026,en:Cereals and their products, | ||
| "Entrees, Sides & Small Meals",969,en:Meals, | ||
| Confectionery Products,934,en:Confectioneries, | ||
| Frozen Patties and Burgers,934,en:Frozen meats,en:Meat patties | ||
| Bread & Muffin Mixes,922,en:Cooking helpers, | ||
| Deli Salads,921,en:Prepared salads, | ||
| Other Grains & Seeds,898,en:Seeds, | ||
| Frozen Bread & Dough,886,, | ||
| Canned Tuna,885,en:Canned tunas, | ||
| Canned Condensed Soup,882,en:Canned soups, | ||
| "Frozen Pancakes, Waffles, French Toast & Crepes",881,, | ||
| Processed Cereal Products,878,en:Cereals and their products, | ||
| Fish & Seafood,864,en:Seafood, | ||
| Prepared Soups,803,en:Soups, | ||
| Flavored Rice Dishes,799,en:Rice dishes, | ||
| All Noodles,791,en:Noodles, | ||
| Meat/Poultry/Other Animals Prepared/Processed,781,en:meats, | ||
| Cooked & Prepared,769,en:meals, | ||
| "Frozen Breakfast Sandwiches, Biscuits & Meals",767,en:meals, | ||
| Gravy Mix,746,en:dehydrated sauces,en:Gravy Mixes | ||
| "French Fries, Potatoes & Onion Rings",732,, | ||
| Eggs & Egg Substitutes,709,en:eggs,en:Egg-based products | ||
| Flavored Snack Crackers,709,en:crackers, | ||
| Canned Meat,708,en:Canned meats, | ||
| Baking Additives & Extracts,701,en:Cooking helpers, | ||
| Liquid Water Enhancer,677,, | ||
| Prepared Subs & Sandwiches,670,en:Sandwiches, | ||
| Baking/Cooking Mixes/Supplies,660,en:Cooking helpers, | ||
| "Gelatin, Gels, Pectins & Desserts",660,, | ||
| "Poultry, Chicken & Turkey",651,en:Poultries, | ||
| Dough Based Products / Meals,592,,en:Dough Based Products | ||
| Alcohol,586,en:Alcoholic beverages, | ||
| Lunch Snacks & Combinations,573,en:snacks, | ||
| Plant Based Water,573,en:Plant-based beverages, | ||
| Sport Drinks,561,en:Sweetened beverages, | ||
| Fish Unprepared/Unprocessed,546,en:Fishes, | ||
| Sauces/Spreads/Dips/Condiments,520,en:sauces, | ||
| Frozen Prepared Sides,517,en:Frozen foods, | ||
| Cereals Products - Ready to Eat (Shelf Stable),511,en:Cereals and their products, | ||
| Meat/Poultry/Other Animals Unprepared/Unprocessed,506,en:meats, | ||
| Vegetables Prepared/Processed,499,en:Prepared vegetables, | ||
| Shellfish Unprepared/Unprocessed,494,en:Seafood,en:Shellfish | ||
| Yogurt/Yogurt Substitutes,481,en:Yogurts, | ||
| Other Condiments,473,en:Condiments, | ||
| Pastry Shells & Fillings,447,en:Pastry helpers, | ||
| Pizza Mixes & Other Dry Dinners,429,, | ||
| "Frozen Bacon, Sausages & Ribs",417,en:frozen meats, | ||
| Cereal/Muesli Bars,409,en:Cereal bars, | ||
| Savoury Bakery Products,374,en:Breads, | ||
| Sweet Bakery Products,366,en:Pastries, | ||
| Prepared Wraps and Burittos,354,en:Sandwiches, | ||
| Chips/Crisps/Snack Mixes - Natural/Extruded (Shelf Stable),296,en:Snacks, | ||
| Stuffing,291,en:stuffing, | ||
| "Frozen Poultry, Chicken & Turkey",280,en:Frozen poultry, | ||
| Sushi,265,en:Sushis, | ||
| Coffee,259,en:Coffees, | ||
| Vegetable Based Products / Meals,249,en:Vegetables based foods, | ||
| "Frozen Sausages, Hotdogs & Brats",240,en:frozen meats, | ||
| Vegetable Based Products / Meals - Not Ready to Eat (Frozen),227,en:Frozen plant-based foods mixes, | ||
| Pies/Pastries - Sweet (Shelf Stable),216,en:Sweet pies, | ||
| Non Alcoholic Beverages Not Ready to Drink,209,en:beverages, | ||
| Pasta/Noodles,209,en:pastas, | ||
| Desserts/Dessert Sauces/Toppings,207,en:desserts, | ||
| Ready-Made Combination Meals,188,en:meals, | ||
| Vegetarian Frozen Meats,181,en:Meat analogues,en:Frozen meat analogues | ||
| Grain Based Products / Meals,177,, | ||
| Meat/Poultry/Other Animals Sausages Prepared/Processed,174,en:sausages, | ||
| "Breakfast Sandwiches, Biscuits & Meals",160,, | ||
| Specialty Formula Supplements,159,, | ||
| Coffee/Tea/Substitutes,158,, | ||
| Cakes - Sweet (Frozen),154,en:Frozen cakes and pastries, | ||
| Other Frozen Meats,152,en:Frozen meats, | ||
| Fruit - Prepared/Processed (Shelf Stable),150,, | ||
| "Pancakes, Waffles, French Toast & Crepes",131,, | ||
| Prepared/Preserved Foods Variety Packs,113,, | ||
| Baby/Infant Foods/Beverages,110,en:Baby foods,en:Baby foods ? | ||
| Herbs/Spices/Extracts,100,, | ||
| Meal Replacement Supplements,96,, | ||
| Grains/Flour,90,, | ||
| Cheese/Cheese Substitutes,89,en:cheeses, | ||
| Drinks Flavoured - Ready to Drink,88,, | ||
| Bread,78,en:Breads, | ||
| Weight Control,70,, | ||
| Nuts/Seeds Prepared/Processed,69,, | ||
| Butter/Butter Substitutes,68,, | ||
| Sandwiches/Filled Rolls/Wraps,65,en:Sandwiches, | ||
| Sweet Spreads,59,en:Sweet spreads, | ||
| Breakfast Drinks,53,en:Beverages, | ||
| Fruit Prepared/Processed,49,, | ||
| Health Care,41,en:Open Products Facts, | ||
| Confection & Snacks,38,, | ||
| Baking/Cooking Supplies (Shelf Stable),35,, | ||
| Ice Cream/Ice Novelties (Shelf Stable),30,, | ||
| Dairy/Egg Based Products / Meals,29,, | ||
| Oils Edible,28,, | ||
| Alcoholic Beverages,26,en:Alcoholic beverages, | ||
| Green Supplements,20,, | ||
| Nuts/Seeds - Prepared/Processed (Shelf Stable),20,, | ||
| Pies/Pastries/Pizzas/Quiches - Savoury (Frozen),19,, | ||
| Breakfast Foods,16,, | ||
| Cereals Products - Not Ready to Eat (Shelf Stable),16,, | ||
| Baking,15,, | ||
| Bread/Bakery Products Variety Packs,15,, | ||
| Popcorn (Shelf Stable),13,, | ||
| Baking Accessories,13,, | ||
| Digestive & Fiber Supplements,13,, | ||
| Milk/Milk Substitutes,13,, | ||
| Herbal Supplements,12,, | ||
| Sugars/Sugar Substitute Products,12,, | ||
| Miscellanious,10,, | ||
| Fruit/Nuts/Seeds Combination,9,, | ||
| Vegetables - Prepared/Processed (Shelf Stable),9,, | ||
| "Condiments, Oils & Dressing",8,, | ||
| "Fruits, Vegetables & Produce",8,, | ||
| Dough Based Products / Meals - Not Ready to Eat - Savoury (Shelf Stable),8,, | ||
| Yogurt/Yogurt Substitutes (Perishable),7,, | ||
| Food/Beverage/Tobacco Variety Packs,6,, | ||
| Baking/Cooking Mixes (Shelf Stable),6,, | ||
| Home Decoration & Supplies,6,, | ||
| Oral Hygiene Products,6,, | ||
| 21.99,6,, | ||
| Grains,5,, | ||
| Vitamins,5,, | ||
| Aquatic Invertebrates/Fish/Shellfish/Seafood Combination,5,, | ||
| Baking/Cooking Mixes (Perishable),5,, | ||
| Fats Edible,5,, | ||
| Pickles/Relishes/Chutneys/Olives,4,, | ||
| Fish Prepared/Processed,4,, | ||
| Egg Based Products / Meals - Not Ready to Eat (Frozen),4,, | ||
| Kitchen Supplies,4,, | ||
| Antioxidant Supplements,3,, | ||
| 18.18,3,, | ||
| Dressings/Dips (Shelf Stable),3,, | ||
| Children's Nutritional Supplements,3,, | ||
| Beef - Prepared/Processed,3,, | ||
| Pork Sausages - Prepared/Processed,3,, | ||
| Soups - Prepared (Shelf Stable),3,, | ||
| 16.9,3,, | ||
| Grain Based Products / Meals - Not Ready to Eat - Savoury (Shelf Stable),2,, | ||
| Pork - Prepared/Processed,2,en:Pork, | ||
| Meat Substitutes,2,, | ||
| Pasta/Noodles - Not Ready to Eat (Frozen),2,, | ||
| Dried Breads (Shelf Stable),2,, | ||
| Vegetables - Prepared/Processed (Frozen),2,, | ||
| 3.66,2,, | ||
| Fatty Acid Supplements,2,, | ||
| Vegetables Unprepared/Unprocessed (Frozen),2,, | ||
| Childcare,2,, | ||
| Shellfish Prepared/Processed,2,, | ||
| Vinegars/Cooking Wines,2,, | ||
| Dough Based Products / Meals - Not Ready to Eat - Savoury (Frozen),1,, | ||
| Skin Care Preparations,1,en:Open Beauty Facts, | ||
| Children's Natural Remedies,1,, | ||
| Baking/Cooking Mixes/Supplies Variety Packs,1,, | ||
| Flour - Cereal/Pulse (Shelf Stable),1,, | ||
| Cakes - Sweet (Shelf Stable),1,, | ||
| Beer,1,en:Beers, | ||
| Amino Acid Supplements,1,, | ||
| Processed Cheese & Cheese Novelties,1,, | ||
| Sauces - Cooking (Shelf Stable),1,en:Sauces, | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| """ | ||
| Read configuration values from .env files. This does not seem | ||
| very useful at the moment, but it costs nothing and we may | ||
| do something smarter in the future. | ||
| """ | ||
| from dotenv import dotenv_values | ||
|
|
||
|
|
||
| def cfg() -> dict: | ||
| d = dotenv_values(".env") | ||
| return d | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,89 @@ | ||
| { | ||
| "sources_fields": { | ||
| "org-database-usda": { | ||
| "fdc_category": "Other Snacks", | ||
| "fdc_data_source": "LI", | ||
| "fdc_id": 1115329, | ||
| "available_date": "0000-00-00", | ||
| "modified_date": "2020-09-12", | ||
| "published_date": "2020-11-13" | ||
| } | ||
| }, | ||
| "code": "0619128673216", | ||
| "code_tags": [ | ||
| "code-13", | ||
| "0619128673XXX", | ||
| "061912867XXXX", | ||
| "06191286XXXXX", | ||
| "0619128XXXXXX", | ||
| "061912XXXXXXX", | ||
| "06191XXXXXXXX", | ||
| "0619XXXXXXXXX", | ||
| "061XXXXXXXXXX", | ||
| "06XXXXXXXXXXX", | ||
| "0XXXXXXXXXXXX" | ||
| ], | ||
| "ingredients_text": "CHILE CHICKPEAS, CHILE PEANUTS, PEPITAS, SMOKEHOUSE HICKORY SMOKED ALMONDS, CHILE POWDER, CITRIC ACID, VEGETABLE OIL & SALT.", | ||
| "ingredients_text_en": "CHILE CHICKPEAS, CHILE PEANUTS, PEPITAS, SMOKEHOUSE HICKORY SMOKED ALMONDS, CHILE POWDER, CITRIC ACID, VEGETABLE OIL & SALT.", | ||
| "categories_tags": [ | ||
| "en:Snacks" | ||
| ], | ||
| "serving_size": "28.0 g", | ||
| "serving_quantity": 28, | ||
| "nutriments": { | ||
| "proteins_value": 25, | ||
| "proteins_serving": 25, | ||
| "proteins_100g": 0.25, | ||
| "proteins_unit": "G", | ||
| "fat_value": 50, | ||
| "fat_serving": 50, | ||
| "fat_100g": 0.5, | ||
| "fat_unit": "G", | ||
| "carbohydrates_value": 67.9, | ||
| "carbohydrates_serving": 67.9, | ||
| "carbohydrates_100g": 0.679, | ||
| "carbohydrates_unit": "G", | ||
| "energy_value": 571, | ||
| "energy_serving": 571, | ||
| "energy_unit": "KCAL", | ||
| "sugars_value": 3.57, | ||
| "sugars_serving": 3.57, | ||
| "sugars_100g": 0.035699999999999996, | ||
| "sugars_unit": "G", | ||
| "fiber_value": 7.1, | ||
| "fiber_serving": 7.1, | ||
| "fiber_100g": 0.071, | ||
| "fiber_unit": "G", | ||
| "calcium_value": 143, | ||
| "calcium_serving": 143, | ||
| "calcium_100g": 14.3, | ||
| "calcium_unit": "MG", | ||
| "iron_value": 2.57, | ||
| "iron_serving": 2.57, | ||
| "iron_100g": 0.257, | ||
| "iron_unit": "MG", | ||
| "sodium_value": 679, | ||
| "sodium_serving": 679, | ||
| "sodium_100g": 67.9, | ||
| "sodium_unit": "MG", | ||
| "vitamin-a_value": 0, | ||
| "vitamin-a_serving": 0, | ||
| "vitamin-a_unit": "IU", | ||
| "vitamin-c_value": 4.3, | ||
| "vitamin-c_serving": 4.3, | ||
| "vitamin-c_100g": 0.43, | ||
| "vitamin-c_unit": "MG", | ||
| "cholesterol_value": 0, | ||
| "cholesterol_serving": 0, | ||
| "cholesterol_100g": 0, | ||
| "cholesterol_unit": "MG", | ||
| "trans-fat_value": 0, | ||
| "trans-fat_serving": 0, | ||
| "trans-fat_100g": 0, | ||
| "trans-fat_unit": "G", | ||
| "saturated-fat_value": 7.14, | ||
| "saturated-fat_serving": 7.14, | ||
| "saturated-fat_100g": 0.07139999999999999, | ||
| "saturated-fat_unit": "G" | ||
| } | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@rkiddy ok, so this is an export of https://docs.google.com/spreadsheets/d/1kO8r2OWLLRuqP-NLkAR8gliVELQHqir6HZ-4CIRs4cM/edit#gid=0 right ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That may have been where I got it. Stephane pointed it out to me in a slack thread. I am, by the way, not understanding that part of it yet. So, I am open to any suggestions.