Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: gzip all JSON OCRs when saving OCR file on disk #8320

Merged
merged 1 commit into from
May 17, 2023
Merged

Conversation

raphael0202
Copy link
Contributor

@raphael0202 raphael0202 commented Apr 14, 2023

To save space, all OCR files were gzipped, but new generated files are still saved as plain text JSON files.
Also add a created_at field in the OCR JSON file containing the timestamp of generation of the OCR file. This is useful to know if we should generate again old OCR files.

@codecov-commenter
Copy link

codecov-commenter commented Apr 14, 2023

Codecov Report

Merging #8320 (2bfb407) into main (a0cfac9) will increase coverage by 0.03%.
The diff coverage is 82.60%.

@@            Coverage Diff             @@
##             main    #8320      +/-   ##
==========================================
+ Coverage   48.46%   48.50%   +0.03%     
==========================================
  Files         114      114              
  Lines       21268    21295      +27     
  Branches     4768     4773       +5     
==========================================
+ Hits        10308    10329      +21     
- Misses       9677     9679       +2     
- Partials     1283     1287       +4     
Impacted Files Coverage Δ
lib/ProductOpener/Import.pm 30.73% <0.00%> (-0.04%) ⬇️
lib/ProductOpener/Images.pm 10.37% <66.66%> (+0.22%) ⬆️
lib/ProductOpener/Packaging.pm 75.00% <75.00%> (ø)
lib/ProductOpener/Test.pm 40.75% <85.71%> (+3.19%) ⬆️
tests/unit/send_image_to_cloud_vision.t 100.00% <100.00%> (ø)

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@alexgarel
Copy link
Member

@raphael0202 broken test: (tests/unit/send_image_to_cloud_vision.t)

To run locally : make unit-test test=send_image_to_cloud_vision.t

2023-04-14T08:49:53.5274088Z malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "\x{1f}\x{fffd}\b\x{0}...") at tests/unit/send_image_to_cloud_vision.t line 50.
2023-04-14T08:49:53.5286080Z # Tests were run but no plan was declared and done_testing() was not seen.
2023-04-14T08:49:53.5292832Z # Looks like your test exited with 255 just after 3.
2023-04-14T08:50:02.4969181Z tests/unit/send_image_to_cloud_vision.t ....... 

@raphael0202
Copy link
Contributor Author

@alexgarel I haven't set up Product Opener locally, will give it a try ;)

- gzip all JSON OCRs when saving OCR file on disk
- add new `created_at` field to save the timestamp of OCR generation
@sonarcloud
Copy link

sonarcloud bot commented May 12, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
No Duplication information No Duplication information

@raphael0202
Copy link
Contributor Author

@alexgarel it should be good now!

@raphael0202
Copy link
Contributor Author

I checked locally, the generated gzipped JSON file saved correctly.

Copy link
Member

@alexgarel alexgarel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great @raphael0202 !

@alexgarel alexgarel merged commit 45df380 into main May 17, 2023
14 checks passed
@alexgarel alexgarel deleted the gzip-ocr-file branch May 17, 2023 08:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants