Conversions takes significantly more time compared to converting in browser

[paulius-petkus]

Tested PDF to Squeeze with 90MB Pdf file:
https://www.convertapi.com/a/api/pdf-to-squeeze#snippet=python
It took 4.5-5s to convert + file upload and download couple seconds. So <10s overall.

With python lib it took >60s:

[laurynas-convertapi]

Hi @paulius-petkus,

could you please measure individual steps (upload, api, download) to help us understand better where the major slowdown is.

You can measure like this:

start_time = time.time()
upload_io = convertapi.UploadIO(open(file, 'rb'))
print(upload_io.file_id)
print("Upload time taken: %s seconds" % (time.time() - start_time))

start_time = time.time()
result = convertapi.convert('squeeze', {'File': upload_io}, from_format='pdf')
print("API time taken: %s seconds" % (time.time() - start_time))

start_time = time.time()
saved_files = result.save_files(tempfile.gettempdir())
print("Download time taken: %s seconds" % (time.time() - start_time))

print("The PDF saved to %s" % saved_files)

[paulius-petkus]

@laurynas-convertapi Hello, sure.

Seems Upload is the biggest problem:

[laurynas-convertapi]

@paulius-petkus thanks! Very helpful.

Can you try adding this to the to of the script and see if it helps to reduce upload time:

from http.client import HTTPConnection
HTTPConnection.__init__.__defaults__ = tuple(
    x if x != 8192 else 64*1024
    for x in HTTPConnection.__init__.__defaults__
)

[paulius-petkus]

@laurynas-convertapi I have added it to the very beginning of the script, but it didn't help. Or should I add it somewhere else?

My current code snippet and results:

[laurynas-convertapi]

Yes, it should be at the top. Strange it didn't have any effect, because I think the issue might be related with python httplib, as described here:
psf/requests#2181 (comment)

Here is another suggestion, maybe this would work?
https://stackoverflow.com/a/39518613

[paulius-petkus]

Hi @laurynas-convertapi, as far as I understand you do not have these problems and conversion takes way less time for you? If the problem does not reproduce on your machine, I think I will investigate what is wrong.

[laurynas-convertapi]

@paulius-petkus yes, I can't see significant difference when uploading file via command line curl (tried both on linux and mac) and running python script (python version 3.9.14).

For me 100MB file upload takes ~15s on 5G connection.

If you have an option to run curl, I suggest testing like this:

time curl -F 'file=@files/100mb.pdf' https://v2.convertapi.com/upload

[paulius-petkus]

It seems everything is fine when I run it with curl. Here is my code that ran in windows command prompt:

@echo off
set start_time=%time%
curl -X POST https://v2.convertapi.com/upload?Secret=my_secret -F "File=@C:/Users/petku/Desktop/TEMP/large1.pdf"
set end_time=%time%
echo Start Time: %start_time%
echo End Time: %end_time%

it took few seconds:

When I do the upload document with request instead of using our library, it also works fast (3-7s instead of 45-55s). My python version: 3.12.1

Here is full code with only secret changed:

import requests
import time
import convertapi
import tempfile

api_secret = 'XXXXXXXXX'

start_time = time.time()
file_path = 'C:/Users/petku/Desktop/TEMP/large1.pdf'
api_url = 'https://v2.convertapi.com/upload'
with open(file_path, 'rb') as file:
    response = requests.post(api_url, files={'file': file}, headers={'Authorization': f'Bearer {api_secret}'})
    upload_io = response.json()
print("Upload time taken: %s seconds" % (time.time() - start_time))

convertapi.api_secret = api_secret
start_time = time.time()
result = convertapi.convert('squeeze', {'File': upload_io['Url']}, from_format='pdf')
print("API time taken: %s seconds" % (time.time() - start_time))

start_time = time.time()
saved_files = result.save_files(tempfile.gettempdir())
print("Download time taken: %s seconds" % (time.time() - start_time))

print("The PDF saved to %s" % saved_files)

[laurynas-convertapi]

Hi @paulius-petkus, very interesting, if the same file upload using requests library directly is significantly faster. Thanks for the example, I will investigate it.

[laurynas-convertapi]

@paulius-petkus could you please try comparing following examples on your system? I don't see the difference on mine, but maybe you'll get different results:

Your adjusted version with files:

import requests
import time

file_path = 'files/test100mb.pdf'
api_url = 'https://v2.convertapi.com/upload'

start_time = time.time()

with open(file_path, 'rb') as file:
    s = requests.Session()
    response = requests.post(api_url, files={'file': file})
    upload_io = response.json()
    print(upload_io['FileId'])

print("Upload time taken: %s seconds" % (time.time() - start_time))

Version with data (as library uses):

import requests
import time

file_path = 'files/test100mb.pdf'
api_url = 'https://v2.convertapi.com/upload'

start_time = time.time()

with open(file_path, 'rb') as file:
    s = requests.Session()
    headers = { 'Content-Disposition': "attachment; filename*=UTF-8''test.pdf" }
    response = s.post(api_url, data = file, headers = headers)
    upload_io = response.json()
    print(upload_io['FileId'])

print("Upload time taken: %s seconds" % (time.time() - start_time))

[paulius-petkus]

Hi, both of these solutions works fast for me. It takes ~2s to upload that 100mb file. It seems the problem occurs on windows, cause me and Tomas both use Windows and we both have this problem.

We are not familiar with python code. But maybe the problem is with those chunks / headers creating in lib? - Maybe we can use most simple upload logic? Example in my previous comment (below curl code) works fast.