Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🧽 Data quality (tracker) #5538

Open
19 of 93 tasks
Tracked by #5529
, #5537
...
teolemon opened this issue Aug 23, 2021 · 1 comment
Open
19 of 93 tasks
Tracked by #5529
, #5537
...

🧽 Data quality (tracker) #5538

teolemon opened this issue Aug 23, 2021 · 1 comment
Labels
🧽 Data quality https://wiki.openfoodfacts.org/Quality 🧽 Quality facet One of the facets available in Open Food Facts is /quality & allows us to spot products w/ bad data ⏰ Stale This issue hasn't seen activity in a while. You can try documenting more to unblock it.

Comments

@teolemon
Copy link
Member

teolemon commented Aug 23, 2021 •
edited

What

Before-the-fact data quality (Prevent mistakes before they happen)

Data destroying issues in production right now

List of data destroying issues in production right now

Product addition
Beta Give feedback
  1. Prevention - Limit the number of new product pages without barcodes per IP #170
    product addition products without barcodes 🧽 Data quality

Spellcheck
Beta Give feedback
  1. Prevention - Add an ingredients spell checker in the product edit form #937
    Spellcheck 🥗 Ingredients 🧽 Data quality

On-the-fly checks
Beta Give feedback
  1. Prevention - On the fly quality checks in the product edit form (javascript) #1905
    Fixable via Userscript Hacktoberfest JavaScript Outreachy Perl help wanted ✨ Feature 🧽 on-the-fly quality checks 🧽 Data quality
    jnsereko
  2. Prevent people from entering email addresses as values #6539
    ⭐ top issue ⏰ Stale 🧽 Data quality
    MonalikaPatnaik
  3. Create an API route to provide Quality Warnings and Errors on a given nutrition table/ingredients list #9710
    ✨ Feature 🧽 on-the-fly quality checks 🧽 API - Quality

Edit rules
Beta Give feedback
  1. Prevention - New edit rules to ignore or notify edits that match given conditions #798
    Fixed ? automation 🧽 Data quality 🧽 Data quality - edit rules
    stephanegigandet
  2. Prevention - Prevent "Undefined" from being stored as a value #2909
    ⏲️ 5 minute fix 🐛 bug 🧽 Data quality 🧽 Data quality - edit rules
  3. Prevention - Create edit rule for "Loading…" #4362
    Data destroying issue ⏲️ 5 minute fix 🐛 bug 🧽 Data quality 🧽 Data quality - edit rules
  4. Make the 'Edit Rules' ignore notifications in Slack clearer #904
    🧽 Data quality 🧽 Data quality - edit rules

Edit rules - Nutrition
Beta Give feedback
  1. Prevention - Nutrition - Reject edits where all entered nutrition values are 0 #1564
    API WRITE Nutrition facts 🎯 Big Bang for Your Time 🎯 P1 candidate 🧽 Data quality - Nutrition 🧽 Data quality 🧽 Data quality - edit rules
    benbenben2
  2. Prevention - Nutrition - Ignore nutrition values if salt is the only value passed and it is 0 #1759
    Fixed ? Nutrition facts 🐛 bug 🧽 Data quality - Nutrition 🧽 Data quality

Ingredient list blocklist
Beta Give feedback
  1. Implement a blocklist for the ingredient list #796
    ✨ Feature 🥗 Ingredients 🧽 Data quality

Taxonomy special characters
Beta Give feedback
  1. Categories - Pâtés et Pates are merged together #458
    P3 categories 🐛 bug 🧬 Taxonomies 🧽 Data quality
    stephanegigandet

Ingredients consistency (languages)
Beta Give feedback
  1. Quality facet - Ingredients - Check if extracted ingredients are consistant in different languages for a product #422
    🥗 Ingredients 🧽 Data quality

Old data
Beta Give feedback
  1. Warn visitors when the product page they're viewing is older than 18 months #58
    Product Page ✨ Feature 🧽 Data quality

Verification
Beta Give feedback
  1. Allow editors to verify products and mark them as checked #1438
    Checked products Fixed ? 🧽 Data quality

Making quality more visible
Beta Give feedback
  1. Quality - Distribution - Show quality issues on a product for admins and users #1407
    Admin tools Fixable via Userscript 🎨 Mockup available 🧽 Data quality

Remediation
Beta Give feedback
No tasks being tracked yet.

Quality facets
Beta Give feedback
  1. Remediation - Introduce a suggested fix for all quality facets #1119
    Fixable via Userscript 🧽 Data quality

Images
Beta Give feedback
  1. Remediation - One click button to revert to previous image revision #1092
    Admin tools 🧽 Data quality

Anonymous edits & UUIDs
Beta Give feedback
  1. Admin button to send Yukaidis associated with a product to the Slack moderation channel #1091
    Admin tools comments 🧽 Data quality
  2. Empty Anonymous User ID pages #3431
    ⏰ Stale ✏️ Editing - anonymous edits 🐛 bug 🧽 Data quality
  3. UUIDs parsing is broken #3849
    🎯 P1 candidate 🐛 bug 🧽 Data quality
  4. Add an issue field to a product #1019
    🧽 Data quality

In place edit

image rotation
Beta Give feedback
  1. Add "Rotate Left" and "Rotate Right" buttons in VIEW mode for logged-in users #998
    Admin tools Product Page inline edit ✏️ Editing - Images ✏️ Editing - drip editing 🎨 Mockup available 🖼️ Images 🧽 Data quality

Tackle variable barcodes
Beta Give feedback
  1. Quality facet - Detect possible duplicates with Same name, same brand, same weight, different barcodes #975
    variable-barcodes 🧽 Data quality

Ingredient analysis (false positives)
Beta Give feedback
  1. Ensure that (blé garanti sans gluten) doesn't create a false positive #944
    Tests - ToDo 🌾 gluten 🎯 P1 candidate 🐛 bug 🥗 Ingredients 🥗🔍 Ingredients analysis 🧽 Data quality

Ingredients/Label coherence
Beta Give feedback
  1. Wipe the vegan label if there's a non vegan ingredient in the ingredient list #913
    labels 🌱 Vegan 🥗 Ingredients 🧽 Data quality

Ingredients
Beta Give feedback
  1. Cut the ingredients list into languages #910
    ingredient-list-cutting ✨ Feature 🌍 Multilingual products 🥗 Ingredients 🧽 Data quality
  2. Quality facet - Ingredients - Check for languages present in the ingredient list #907
    🥗 Ingredients 🧽 Data quality

Golden set
Beta Give feedback
  1. Create a reference dataset to perform quality checks #905
    dataset creation 🧽 Data quality

Search
Beta Give feedback
  1. Search in ingredients (and just in ingredients) #903
    ✨ Feature 🔎 Search 🥗 Ingredients 🧽 Data quality

Producer platform
Beta Give feedback
  1. Quality - Prevention - Producer platform - Do not export Empty products #5352
    ⏰ Stale ✨ Feature 🏭 Producers Platform 🧽 Data quality

Producer - One-off
Beta Give feedback
  1. The GS1/Materne import includes ASC and MSC labels on products that do not contain fish #4876
    ⏰ Stale 🎯 P1 🏭 Producers Platform 🐛 bug 🧽 Data quality
    stephanegigandet

Countries
Beta Give feedback
  1. Scanbot is creating bad country entries #4870
    scan statistics 🌐 i18n 🐛 bug 🧽 Data quality
    stephanegigandet
  2. Some products have a different value for lang and lc #4435
    ⏰ Stale 🌍 Multilingual products 🐛 bug 🧽 Data quality
    stephanegigandet

Improve the Recent changes screen
Beta Give feedback
  1. Make detected quality issues visible to moderators in "Recent changes" #2952
    Recent changes ✨ Feature 🎨 Mockup available 🧽 Data quality
  2. Filter Recent Changes by user group (app, bots…) #2125
    Recent changes ✨ Feature 🧽 Data quality
  3. Create warning for "Not indicated" values #2782
    ✏️ Editing ✨ Feature 🧽 Data quality

Nutrition
Beta Give feedback
  1. Bug - Quality facets - Incorrect flagging because of IU units #2580
    🐛 bug 🧽 Data quality
  2. Quality facet - Nutrition - serving_size="serving" for data-quality #5163
    good first issue osd'22 portions ⏰ Stale ⚖️ Quantity ✨ Feature 🧽 Data quality 🧽 Quality facet
  3. Quality facet - Nutrition - list products where serving size can't be parsed #3029
    Outreachy portions ✨ Feature 🧽 Data quality 🧽 Quality facet
    benbenben2
  4. Quality facet - Nutrition - Compare stated and calculated energy for higher accuracy of product listings and spotting faulty products #2429
    Nutrition facts ✨ Feature 🧽 Data quality 🧽 Quality facet
    stephanegigandet
  5. Quality facet - Nutrition - Check that quantity of nutrients/ingredients match legal product claims #1414
    ⭐ top issue 🍬 Sugar 🧽 Data quality 🧽 Quality facet
    benbenben2
  6. Quality - Prevention - Nutrition - Control _100g fields values as digits only #4582
    ⏰ Stale 🐛 bug 🧽 Data quality
  7. Feature request: add data-quality-error: insoluble/soluble fiber greater than total fiber #9729
    ✨ Feature 🧽 Data quality

Nutrition - Averages
Beta Give feedback
  1. Knowledge panel - Salt - Add a warning on products that contain more than the WHO daily salt recommandation #1413
    📖 Knowledge Panels 🧂 Salt 🧽 Data quality - Nutrition 🧽 Data quality

Absence of values
Beta Give feedback
  1. Quality - Prevention - Add a checkbox "Fiber is not specified on the product" #1938
    frontend ✨ Feature 🎯 P1 candidate 🚦Nutri-Score 🧽 Data quality
    stephanegigandet

Photo/Data discrepancy
Beta Give feedback
  1. Quality - Facet - Nutrition photo has changed, but not the nutrition data itself #1910
    Nutrition facts ✨ Feature 🧽 Data quality - Nutrition 🧽 Data quality
  2. Quality - Facet - Show a special prompt when the ingredient list photo has changed, but not the ingredient list itself #1909
    status system ✨ Feature 🧽 Data quality

Grasp opportunities for improvement

Ingredients
Beta Give feedback
  1. Quality facet - Ingredients - Create a facet whenever a new ingredient image would improve the ingredient recognition score #1875
    slack-notifications ✨ Feature 🧽 Data quality

Nutrition
Beta Give feedback
  1. Quality - Nutrition - Exclude prepared nutritional values from graphs and stats #1859
    old-products 📊 Charts 🧽 Data quality

Checkbot
Beta Give feedback
  1. Quality - Distribution - Notifications - Revamp proposal #1606
    checkbot 🧽 Data quality
    CharlesNepote
  2. To be completed query can select invalid records #1603
    P4 🧽 Data quality

Top-scan excellence (slack warnings)
Beta Give feedback
  1. Quality - Distribution - Add a slack warning when one of the top 100 popular products (by scan) is edited #1213
    scan statistics slack-notifications 🧽 Data quality

Data augmentation to detect specific products
Beta Give feedback
  1. Extract alcohol or fat % from title #1189
    Can be done by Robotoff automatic-data-augmentation 🧽 Data quality
    stephanegigandet

Data quality measurement

Tasks
Beta Give feedback
No tasks being tracked yet.

General
Beta Give feedback
  1. Quality Facets - Create a quality facet with detected errors #912
    🧽 Data quality 🧽 Quality facet

User reports
Beta Give feedback
  2. Add button to signal issue with product #1015
    Fixable via Userscript Product Page good first issue mockup-required 👮 Moderation 🧽 Data quality
  3. Add a flag button/API to put up a product for review when you're in a hurry #1408
    API WRITE Admin tools Fixable via Userscript P2 🎨 Mockup available 🧽 Data quality
  4. Let users request deletion of images #2891
    ⭐ top issue Admin tools ✨ Feature 🎯 P1 candidate 👍 Top 10 Issue! 👮 Moderation 🖼️ Images 🧽 Data quality
  5. Add button that sends the barcode of a product to delete to a Slack channel #1014
    barcodes good first issue ✨ Feature 👮 Moderation 🧽 Data quality
  6. Collect feedback from the OCR mobile button "Too bad, wipe" #3752
    Can be done by Robotoff OCR ⏰ Stale ✨ Feature 🤳🥫 blocking mobile apps 🧽 Data quality

NSFW and vandalism detection
Beta Give feedback
  1. Quality facet - Add a "forbidden" words quality rule #1025
    🎯 Big Bang for Your Time 🎯 P1 candidate 🧽 Data quality 🧽 Data quality - edit rules 🧽 Quality facet

Barcodes
Beta Give feedback
  1. Quality Facet - Barcodes - Detect any mismatching barcode in pictures associated to a barcode #1020
    barcodes 🧽 Data quality 🧽 Quality facet
  2. Quality facets - barcodes that have a strange number of digits, or a bad last digit key #1806
    🧽 Data quality 🧽 Quality facet

Ingredients
Beta Give feedback
  1. feat: Try to detect foreign languages in ingredient lists (2019 PR on fresh code) #4818
    Stale ⏰ Stale 💥 Merge Conflicts 🥗 Ingredients 🧽 Quality facet
    teolemon
  2. Quality facet - Ingredients - Consistency check between ingredients and meat used claims #3097
    ingredients analysis ✨ Feature 🥗 Ingredients 🧽 Quality facet
  3. Quality facet - Nutrition/Ingredients - Add quality tests for min/max % of ingredients/nutrients on regulated categories #2862
    ✨ Feature 🧽 Data quality 🧽 Quality facet
  4. Quality facet - Ingredients - Flag products with only additives as ingredients that are not categorized as additives #2242
    🐛 bug 🧽 Quality facet
  5. Quality facet - ingredient % over 100 #2168
    ⭐ 🐛 top bug Hacktoberfest Outreachy Perl 🐛 bug 🥗 Ingredients 🧽 Data quality 🧽 Quality facet
    matheus-de
  6. Quality facet - Check whether ingredients match exactly accross languages #1911
    ✨ Feature 🥗 Ingredients 🧽 Data quality 🧽 Quality facet
  7. Quality facet - On organic products, check that non organic ingredients are ≤ 5% #1419
    organic products 🥗 Ingredients 🧽 Data quality 🧽 Quality facet
  8. Quality facet - Fishing methods in ingredients #1192
    🐠 Fishing 🧽 Data quality 🧽 Quality facet
  9. Quality Facet - Products who have an origin in the ingredient list, but no origin of ingredients filled #1142
    🌱 Eco-Score 📍 Origins 🧽 Data quality 🧽 Quality facet

Packaging
Beta Give feedback
  1. Quality facets - packagings #4501
    ⏰ Stale ✨ Feature 📦 Packaging 🧽 Quality facet

Duplicates
Beta Give feedback
  1. Quality facet - Nutrition - Flag products where 3 or more nutritional values are identical #3554
    ✨ Feature 🧽 Data quality - Nutrition 🧽 Data quality 🧽 Quality facet

Products stored in the wrong language
Beta Give feedback
  1. Quality facet - Products that should not be in French, based on their brand #2954
    ✨ Feature 🧽 Quality - foreign-products-stored-in-french 🧽 Quality facet

Countries
Beta Give feedback
  1. Quality facets - Products with a given barcode country prefix which don't have that country in countries #1088
    barcodes 🧽 Data quality 🧽 Quality facet

Quantity
Beta Give feedback
  1. Quality facet - "quantity-contains-e" has many false positives #2037
    ⚖️ Quantity 🐛 bug 🧽 Data quality 🧽 Quality facet

Facets

Labels
Beta Give feedback
  1. Quality facet - Organic additives #1807
    Outreachy organic products 🧪 additives 🧽 Data quality 🧽 Quality facet

Common name
Beta Give feedback
  1. Quality facet - if common first name like "Paul" is in the product name #1203
    🧽 Data quality 🧽 Quality facet

Image freshness
Beta Give feedback
  1. Quality Facets - freshness of images #1908
    data-freshness ✨ Feature 🖼️ Images 🧽 Data quality 🧽 Quality facet

Data quality remediation

Tools to fix

Ingredients
Beta Give feedback
  1. Quality - Remediation - Add a button to quickly add / remove underscores around allergens in ingredient list #902
    🥗 Ingredients 🧽 Data quality

Getting contributors to own their mistakes
Beta Give feedback
  1. Quality - Distribution - Get checkbot to send an email to people about their product defect #49
    checkbot ✨ Feature 📨 Emails 🕹️ Gamification 🧽 Data quality
    CharlesNepote

Other projets / Non-Food products
Beta Give feedback
  1. Quality facet - find cosmetics in Open Food Facts #1027
    help wanted 🧴 Open Beauty Facts 🧽 Data quality 🧽 Quality facet
    CharlesNepote benbenben2
  2. Quality facets - find petfood in Open Food Facts #1026
    help wanted 🧽 Data quality 🧽 Quality facet
    CharlesNepote benbenben2
  3. Quality - Automation - Wipe nutrition values when a product is transferred to Open Beauty Facts #924
    Fixable via Userscript Nutrition facts 🧴 Open Beauty Facts 🧽 Data quality
  4. Create barcode "blocklists" #933
    barcodes ✨ Feature 🐾 Open Pet Food Facts 📸 Open Products Facts 🧴 Open Beauty Facts 🧽 Data quality
    stephanegigandet

Additives detection
Beta Give feedback
  1. Additives detection - Tricky list of additives: di, tri, polyphosphate de sodium #804
    🐛 bug 🥗 Ingredients 🧪 additives 🧽 Data quality
    stephanegigandet
  2. Additives detection - E262ii detection issues #774
    🐛 bug 🧪 additives 🧽 Data quality

Hunger games
Beta Give feedback
  1. Allow to flag a product for moderation in hungergames openfoodfacts-hungergames#270

Part of
@teolemon teolemon mentioned this issue Aug 23, 2021
Open
1 task
@teolemon teolemon added 🧽 API - Quality 🧽 Data quality https://wiki.openfoodfacts.org/Quality 🧽 Quality facet One of the facets available in Open Food Facts is /quality & allows us to spot products w/ bad data and removed 🧽 API - Quality labels Aug 23, 2021
This was referenced Aug 23, 2021
Open
Open
Open
@teolemon teolemon mentioned this issue Sep 3, 2021
Open
This was referenced Sep 10, 2021
Closed
Open
Closed
Open
Open
Open
Open
Closed
Open
@teolemon teolemon mentioned this issue Sep 21, 2021
Closed
1 task
This was referenced Oct 11, 2021
Open
Open
Closed
Closed
Open
Closed
Open
Open
Open
Closed
Closed
@teolemon teolemon mentioned this issue Dec 10, 2021
Open
1 task
This was referenced Jan 3, 2022
Open
Open
This was referenced Mar 28, 2022
Closed
Closed
@teolemon teolemon mentioned this issue Apr 12, 2022
Open
This was referenced Jul 8, 2022
Open
Open
Open
@teolemon teolemon mentioned this issue Oct 12, 2022
Open
@teolemon teolemon mentioned this issue Oct 25, 2022
Closed
@teolemon teolemon mentioned this issue Nov 22, 2022
Open
@teolemon teolemon mentioned this issue Dec 13, 2022
Closed
4 tasks
@teolemon teolemon mentioned this issue Apr 24, 2023
Open
@teolemon teolemon mentioned this issue May 29, 2023
Open
This was referenced Jun 19, 2023
Closed
Closed
@teolemon teolemon changed the title Data quality (tracker) 🧽 Data quality (tracker) Aug 16, 2023
@teolemon teolemon mentioned this issue Aug 24, 2023
Open
@teolemon teolemon mentioned this issue Oct 25, 2023
Open
This was referenced Nov 13, 2023
Open
Open
Open
@github-actions GitHub Actions
Copy link
Contributor

This issue has been open 90 days with no activity. Can you give it a little love by linking it to a parent issue, adding relevant labels and projets, creating a mockup if applicable, adding code pointers from https://github.com/openfoodfacts/openfoodfacts-server/blob/main/.github/labeler.yml, giving it a priority, editing the original issue to have a more comprehensive description… Thank you very much for your contribution to 🍊 Open Food Facts

@github-actions github-actions bot added the ⏰ Stale This issue hasn't seen activity in a while. You can try documenting more to unblock it. label Jan 23, 2024
This was referenced Apr 1, 2024
Open
Open
Open
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
🧽 Data quality https://wiki.openfoodfacts.org/Quality 🧽 Quality facet One of the facets available in Open Food Facts is /quality & allows us to spot products w/ bad data ⏰ Stale This issue hasn't seen activity in a while. You can try documenting more to unblock it.
Projects
🍊 Open Food Facts Server issues
Status: To discuss and validate
🧽 Ensuring Data Quality
Status: To do
Milestone
No milestone
Development

No branches or pull requests
1 participant
@teolemon