Libreoffice Concurrence

[jrios2006]

Hello I have a doubt about a libreoffice concurrence when I try to convert an office documento to pdf.

In my linux computer when I try to execute the conversion program I run this comand
libreoffice --headless --convert-to pdf filename.doc

If this program may take 40 seconds to run for example, then in this time if there is another request to execute another conversion like that
libreoffice --headless --convert-to pdf anotherfilename.doc

The second request executes but the results is bad (fails to generate the pdf) becuase the libreoffice fails.

Is it possible add to the program that before execuntig the conversion, the program asks if libreoffice is executing and if that ocurrs the program wait to execute another conversion?

For example:

def CheckExecuteLibreOffice():

'''
Function to know if Libreoffice is executing for example
ps -aux | grep soffice | wc -l
this return 1 or >1
if $(ps -aux | grep soffice | wc -l) > 1 then
Aux = True
else
Aux = False
end of if
return Aux
'''

.... Process the data

Variable = CheckExecuteLibreOffice()
While Variable is true:
wait 1 second
Variable = CheckExecuteLibreOffice()
end of while
/* In this moment LibreOffice is not executing */
Execute conversion tool

..... Continue the process

Another question, in your program exists a variable that to wait the execute (waitDelay). this works well to say that the convesion takes this seconds.
I think that the code could repeat the process and adds +5 seconds if the result fails and generate and Error 500?

For example

contador = 1
result = convertpdf() ' convertir the office doc
while result is Error or contador <= 10:
waitDelay = waitDelay + 5
contador = contador + 1
Repeat a new process with the same command with the new waitDelay variable
end of while

Best regards,

``

[gulien]

Hello @jrios2006

Thank you for your report! I'm currently working on the version 6 of Gotenberg, I'll try reproduce your concurrency issue and fix it if possible.

[jrios2006]

I have used a excel file that when I print the document generates 500 pages. This process takes more than 40 seconds (with libreoffice), then I execute the same file to another files destination and libreoffice fails with the second or sucesives docs while the first executios does not finish.

If you need I send you a excel file sample and the test with my machine

[jrios2006]

For example
[jra@AIO tmp]$ time libreoffice --headless --convert-to pdf --outdir /tmp kk.xlsx
And this is the result
real 0m6,611s user 0m0,018s sys 0m0,008s
When I execetu this command

curl --request POST --url http://localhost:3000/convert/office --header 'Content-Type:multipart/form-data' --form files=@kk.xlsx --form landscape=true --form waitTimeout=30 -o result1.pdf

This is the result
{"time":"2019-07-03T15:12:17.921898862Z","id":"","remote_ip":"172.17.0.1","host":"localhost:3000","method":"POST","uri":"/convert/office","user_agent":"curl/7.65.1","status":200,"error":"","latency":17166397010,"latency_human":"17.16639701s","bytes_in":1670697,"bytes_out":1996360}

If the second execution is without stop the first
curl --request POST --url http://localhost:3000/convert/office --header 'Content-Type:multipart/form-data' --form files=@kk.xlsx --form landscape=true --form waitTimeout=30 -o result2.pdf

This fails and the result is
{"time":"2019-07-03T15:12:31.671400285Z","id":"","remote_ip":"172.17.0.1","host":"localhost:3000","method":"POST","uri":"/convert/office","user_agent":"curl/7.65.1","status":500,"error":"code=500, message=unoconv: signal: killed","latency":30008519966,"latency_human":"30.008519966s","bytes_in":1670697,"bytes_out":38}

I have added the parameter --form waitTimeout=30 because this convert fails with time standart.

Regards,

Julio.
kk.xlsx
result2.pdf
result1.pdf

[compleatang]

I've encountered this as well and had to build a queue system in my (go) client service. It adds quite a bit of hefty non-meaningful code to the service. Because chromium and libreoffice have quite different concurrency patterns, I've had to isolate the worker queues as my service converts both html and docx to pdf.

Chromium doesn't seem too bad, although I have observed degradation around 5 concurrent html -> pdf requests. In general, Chromium seems to handle up to 5 concurrent requests without too much of a problem in my k8s cluster (1GB RAM and 1 CPU request; 2GB RAM, 2 CPU limit).

On the other hand, libreoffice has atrocious concurrency patterns and I experience pretty significant degradation whenever I have any concurrent requests on the wire with the same resources noted above. My suspicion is the cold boot time + libreoffice's JVM not being happy with concurrency patterns in a constrained resources environment.

To handle this I've just capped my worker queue to 1 for docx and 5 to html. FWIW, this isn't a network issue as I run my client service in the same pod as the gotenberg container so it's localhost and also allows me to ensure that only one client service is talking to one gotenberg.

Ideally, it would be great to be able to move the worker queues into either gotenberg's go level or even the go client library (which, if I'm not the only one experiencing this, wouldn't help others that are not using the go client library) and if moved into gotenberg then to be able to set a worker queue limit for the chromium and libreoffice workers separately via env vars.

In any event, that's my 2c.

[gulien]

@compleatang thank you for the insights!

I'm planning to do some load testing; it will help me to have a better understanding of the API capacity.

For Libreoffice, I'm using a lock mechanism with mutex as unoconv cannot perform concurrent conversion. So the more concurrent requests, the more time it should take to have a result.

[compleatang]

Figured. I'd be interested to see what your results are cause they seem non linear in our testing, albeit inconsistently so.

[jrios2006]

Just a question it is possible download to test the version 6 with this command?
docker run --rm -p 3000:3000 thecodingmachine/gotenberg:6
This is the result
Unable to find image 'thecodingmachine/gotenberg:6' locally docker: Error response from daemon: manifest for thecodingmachine/gotenberg:6 not found.

Can I download this version from another url to test it?

Regards.

[gulien]

Currently you have to fetch the 6.0.0 branch and build the image locally.

[gulien]

This issue is related with #87

[gulien]

@compleatang @jrios2006

The newer version should work better:

• Gotenberg is now able to handle concurrent Office conversions
• Don't forget to increase the timeout using the waitTimeout form field or the environment variable DEFAULT_WAIT_TIMEOUT. Don't forget to increase the MAXIMUM_WAIT_TIMEOUT if needed.
• Load testing results: https://github.com/thecodingmachine/gotenberg/tree/master/loadtesting

Enjoy 😄

[compleatang]

👍