Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

generator already executing #121

Closed
L3o-pold opened this issue Jan 13, 2020 · 8 comments · Fixed by #126
Closed

generator already executing #121

L3o-pold opened this issue Jan 13, 2020 · 8 comments · Fixed by #126

Comments

@L3o-pold
Copy link

@L3o-pold L3o-pold commented Jan 13, 2020

Describe the bug
ValueError: generator already executing

To Reproduce
Make a lot of conversion maybe?

Expected behavior
No error

Desktop (please complete the following information):

  • OS: Docker python:3.7-stretch
  • Version 1.11.0

Code:

Where pdf var is a typing.IO[bytes] stream

pdf_content = pdf.read() 
images = convert_from_bytes(pdf_content)

Output:

File "/usr/local/lib/python3.7/site-packages/****, line ****, in ****
images = convert_from_bytes(pdf_content)
File "/usr/local/lib/python3.7/site-packages/pdf2image/pdf2image.py", line 269, in convert_from_bytes
 paths_only=paths_only,
 File "/usr/local/lib/python3.7/site-packages/pdf2image/pdf2image.py", line 144, in convert_from_path
thread_output_file = next(output_file)
ValueError: generator already executing
@Belval

This comment has been minimized.

Copy link
Owner

@Belval Belval commented Jan 14, 2020

I think I will need more information in order to help with this issue.

Can you provide a snippet of the code used? This seems to indicate a weird race-condition but I hardly see how it could happen.

@L3o-pold

This comment has been minimized.

Copy link
Author

@L3o-pold L3o-pold commented Jan 14, 2020

It came from this project https://github.com/algoo/preview-generator/blob/develop/preview_generator/preview/builder/pdf__pypdf2.py#L56

On top of that I have a multithreaded webserver.

CRITICAL:root:generator already executing
Traceback (most recent call last):
  File "script.py", line 70, in get_thumbnail
    image_path, height=payload.get('height', 200), width=payload.get('width', 200)
  File "/usr/local/lib/python3.7/site-packages/preview_generator/manager.py", line 176, in get_jpeg_preview
    mimetype=mimetype,
  File "/usr/local/lib/python3.7/site-packages/preview_generator/preview/builder/pdf__pypdf2.py", line 56, in build_jpeg_preview
    result = convert_pdf_to_jpeg(output_stream, size)
  File "/usr/local/lib/python3.7/site-packages/preview_generator/preview/builder/image__wand.py", line 20, in convert_pdf_to_jpeg
    images = convert_from_bytes(pdf_content)
  File "/usr/local/lib/python3.7/site-packages/pdf2image/pdf2image.py", line 269, in convert_from_bytes
    paths_only=paths_only,
  File "/usr/local/lib/python3.7/site-packages/pdf2image/pdf2image.py", line 144, in convert_from_path
    thread_output_file = next(output_file)
ValueError: generator already executing
@Belval

This comment has been minimized.

Copy link
Owner

@Belval Belval commented Jan 14, 2020

Does it only happen with a specific PDF or with any PDF?

@L3o-pold

This comment has been minimized.

Copy link
Author

@L3o-pold L3o-pold commented Jan 14, 2020

I'll not say any PDF, but a lot.

@L3o-pold

This comment has been minimized.

Copy link
Author

@L3o-pold L3o-pold commented Jan 16, 2020

By the way I see a lot of process that consuming a lot of memory and never ends to execute:

****      3537  0.3  0.0 124532 65744 ?        S    14:34   0:10 pdftoppm -r 200 /tmp/**** -f 1 -l 1
****      3755  0.3  0.0 111372 52884 ?        S    14:36   0:08 pdftoppm -r 200 /tmp/**** -f 1 -l 1
****      4087  0.0  0.0 161448 102204 ?       S    14:40   0:02 pdftoppm -r 200 /tmp/**** -f 1 -l 1
****      4101  0.0  0.0  83104 24196 ?        S    14:40   0:00 pdftoppm -r 200 /tmp/**** -f 1 -l 1
****      4119  0.0  0.0 103892 44852 ?        S    14:40   0:00 pdftoppm -r 200 /tmp/**** -f 1 -l 1
****      4143  0.0  0.0  82148 23052 ?        S    14:40   0:00 pdftoppm -r 200 /tmp/**** -f 1 -l 1
****      4164  0.0  0.0 123084 51272 ?        S    14:41   0:01 pdftoppm -r 200 /tmp/**** -f 1 -l 1
```
@Belval

This comment has been minimized.

Copy link
Owner

@Belval Belval commented Jan 17, 2020

Hmm I am not able to reproduce the issue on my side, what webserver are you using?

@L3o-pold

This comment has been minimized.

Copy link
Author

@L3o-pold L3o-pold commented Jan 17, 2020

@Belval I just switched to Tornado and the problem seems fixed.
I also added a timeout in communicatemethod in your pdf2image lib.

@L3o-pold L3o-pold closed this Jan 17, 2020
@Belval

This comment has been minimized.

Copy link
Owner

@Belval Belval commented Jan 17, 2020

The timeout is a good idea, if the problem is reported again I will consider adding one by default or at least making it a parameter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

2 participants
You can’t perform that action at this time.