Pycurl

[Mister-XY]

Hello, I have a question about Pycurl.
Is it possible that you replace pycurl with something else or do you have an alternative?
Unfortunately you often get this error in Pyload (Debian 9) "(35) gnutls_handshake () failed: An unexpected TLS packet was received."
Recompiling with ssl function is also not the "reasonable" solution.
I also found that restarting Pyload helps and the downloads work normally for a while. After that comes the same mistake again.
Partially for all downloads.

[GammaC0de]

@Mister-XY You should use openssl instead of gnutls.
You can read further here: https://wfuzz.readthedocs.io/en/latest/user/installation.html#pycurl-ssl-bug

[Mister-XY]

That is what i mean with

Recompiling with ssl function is also not the "reasonable" solution.

[vuolter]

Actually I'm investigating if we can use python-requests instead pycurl.

[GammaC0de]

@vuolter I vote against python-requests because of low download speed performance

@Mister-XY If you use a relatively recent Linux distribution, the openssl package should already included so no need for recompile..

[Mister-XY]

I think Debian stretch is very up to date.
Here is my pycurl output:
'PycURL / 7.43.0 libcurl / 7.52.1 GnuTLS / 3.5.8 zlib / 1.2.8 libidn2 / 0.16 libpsl / 0.17.0 (+ libidn2 / 0.16) libssh2 / 1.7.0 nghttp2 / 1.18.1 librtmp / 2.3'

[GammaC0de]

You should install libcurl4-openssl-dev instead of libcurl4-gnutls-dev

[Mister-XY]

The same with libcurl4-openssl-dev

'PycURL/7.43.0 libcurl/7.52.1 GnuTLS/3.5.8 zlib/1.2.8 libidn2/0.16 libpsl/0.17.0 (+libidn2/0.16) libssh2/1.7.0 nghttp2/1.18.1 librtmp/2.3'

In this workaround, you see, that the package pycurl is new/recompiled with the libcurl4-openssl-dev package.

[GammaC0de]

I am sure there is a way without recompiling..

EDIT:
Try this:

pip uninstall pycurl
export PYCURL_SSL_LIBRARY=openssl
pip install pycurl

[Mister-XY]

Ok i will watch about that. But now, it shows openssl instead gnu...

[GammaC0de]

it shows openssl instead gnu..

You mean that pycurl now shows OpenSSL instead GnuTLS?
If so, that is a good thing and your problems should be solved.

Ok i will watch about that

That means no errors so far?

[vuolter]

@GammaC0de Okay, I'll make some test to evaluate the network overhead of python-requests when retrieving big files.

[Mister-XY]

I think the pycurl version installed via pip is different than the one installed via apt.

[GammaC0de]

I think the pycurl version installed via pip is different than the one installed via apt.

For sure it is different, so?

You can do pycurl.version and compare to the previous result #3250 (comment)

[Mister-XY]

Look, here via pip

'PycURL/7.43.0.2 libcurl/7.52.1 OpenSSL/1.0.2l zlib/1.2.8 libidn2/0.16 libpsl/0.17.0 (+libidn2/0.16) libssh2/1.7.0 nghttp2/1.18.1 librtmp/2.3'

an here via apt

'PycURL/7.43.0 libcurl/7.52.1 GnuTLS/3.5.8 zlib/1.2.8 libidn2/0.16 libpsl/0.17.0 (+libidn2/0.16) libssh2/1.7.0 nghttp2/1.18.1 librtmp/2.3'

[GammaC0de]

Exactly the same version except OpenSSL instead of GnuTLS and v7.43.0.2 instead of 7.43.0.

[Mister-XY]

Than, the workaround is to install pycurl via pip and not via apt.

[GammaC0de]

@vuolter

I vote against python-requests because of low download speed performance

See https://github.com/svanoort/python-client-benchmarks

[vuolter]

Interesting... thx for the link @GammaC0de 😉
According those graphs pycurl can handle a significant number of http requests more than python-requests (but... no surprises since pycurl is written in c).
The point is if that "superiority" is relevant for us. I mean, I think no one is using pyload with more than a hundred of parallel connections and in those graphs the measuring starts from 600 requests per second.
Even the cpu drain is calculated according the mininum unit of 10000 http requests.

[lukastribus]

I think this discussion would be more appropriate in #2184