Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: mongodb _id must be a string #7255

Merged
merged 4 commits into from
Aug 29, 2022
Merged

fix: mongodb _id must be a string #7255

merged 4 commits into from
Aug 29, 2022

Conversation

stephanegigandet
Copy link
Contributor

There was an issue for barcode changes, the mongodb _id field got a number long type instead of a string.

This:

  • fixes the issue so that _id is always stored as a string
  • adds a --fix-non-string-ids command to the update_all_products.pl script

Once deployed, we need to:

run ./update_all_products.pl --fix-non-string
issue the MongoDB command indicated at the end of the script run:

products stored in MongoDB with a non string _id have been reloaded from .sto files (if the products still exist) and stored with a string _id.
products with non string ids can now be deleted from MongoDB with this command:
db.products.remove({_id : { $type : "long" }})

Fixes #4667

@sonarcloud
Copy link

sonarcloud bot commented Aug 26, 2022

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
No Duplication information No Duplication information

# see bug #1077 - https://github.com/openfoodfacts/openfoodfacts-server/issues/1077
# make sure that code is saved as a string, otherwise mongodb saves it as number, and leading 0s are removed
$product_ref->{code} = $product_ref->{code} . '';
$product_ref->{_id} .= '';
$product_ref->{code} .= '';
$products_collection->replace_one({"_id" => $product_ref->{_id}}, $product_ref, { upsert => 1 });
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should'nt we refactor and have an upsert_product function that ensure this kind of thing ? (Instead of using replace_one directly). This may help avoiding this error and maybe others ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea. It could be a good issue for open source day. cc @yuktea
I filed a bug for it: #7271

Copy link
Member

@alexgarel alexgarel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@stephanegigandet stephanegigandet merged commit 1b14745 into main Aug 29, 2022
@stephanegigandet stephanegigandet deleted the mongodb_ids branch August 29, 2022 15:25
@CharlesNepote CharlesNepote added the 🧽 Data quality https://wiki.openfoodfacts.org/Quality label Sep 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🧽 Data quality https://wiki.openfoodfacts.org/Quality Products
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

Some products in MongoDB are stored with an _id (code) stored as a number instead of a string
3 participants