Socks proxy error #1077

maxgrechnev · 2018-04-18T09:28:19Z

Steps to reproduce

The code compares behaviour of two libs: urllib3 and python-telegram-bot:

TOKEN = '<your_api_token>'
SOCKS_URL = 'socks5://<your_sock5_proxy_host>:1080/'
SOCKS_USER = '<your_sock5_proxy_user>'
SOCKS_PASS = '<your_sock5_proxy_password>'

################## urllib3 ##################

import urllib3
urllib3.disable_warnings()
from urllib3.contrib.socks import SOCKSProxyManager

manager = SOCKSProxyManager(
	SOCKS_URL,
	username = SOCKS_USER,
	password = SOCKS_PASS,
)
response = manager.request('GET', 'https://api.telegram.org/bot' + TOKEN + '/getMe')
print response.data

############ python-telegram-bot ############

from telegram.utils.request import Request
from telegram import Bot

bot = Bot(
	TOKEN,
	request = Request(
		proxy_url = SOCKS_URL,
		urllib3_proxy_kwargs = {
				'username': SOCKS_USER,
				'password': SOCKS_PASS,
		},
	)
)
print str(bot.get_me().id)

Expected behaviour

I expect the same behaviour of both libraries: urllib3 and python-telegram-bot.

Actual behaviour

urllib3 works well, but python-telegram-bot raises telegram.error.NetworkError.

Configuration

Operating System:
Linux test 4.14.30-v7+ #1102 SMP Mon Mar 26 16:45:49 BST 2018 armv7l GNU/Linux
Raspbian GNU/Linux 9 (stretch)

Version of Python, python-telegram-bot & dependencies:
python-telegram-bot 10.0.2
certifi 2018.01.18
future 0.16.0
urllib3 1.22
PySocks 1.6.8
Python 2.7.13 (default, Nov 24 2017, 17:33:09) [GCC 6.3.0 20170516]

Logs

Output of the code:

{"ok":true,"result":{"id":584331883,"is_bot":true,"first_name":"Test","username":"TestSocksBot"}}
Traceback (most recent call last):
  File "/usr/lib/test/test.py", line 33, in <module>
	print str(bot.get_me().id)
  File "/usr/local/lib/python2.7/dist-packages/telegram/bot.py", line 60, in decorator
	result = func(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/telegram/bot.py", line 191, in get_me
	result = self._request.get(url, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/telegram/utils/request.py", line 245, in get
	result = self._request_wrapper('GET', url, **urlopen_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/telegram/utils/request.py", line 201, in _request_wrapper
	raise NetworkError('urllib3 HTTPError {0}'.format(error))
telegram.error.NetworkError: urllib3 HTTPError SOCKSHTTPSConnectionPool(host='api.telegram.org', port=443): Max retries exceeded with url: /bot<your_api_token>/getMe (Caused by NewConnectionError('<telegram.vendor.ptb_urllib3.urllib3.contrib.socks.SOCKSHTTPSConnection object at 0x7624d7f0>: Failed to establish a new connection: 0x02: Connection not allowed by ruleset',))

The text was updated successfully, but these errors were encountered:

Eldinnie · 2018-04-18T09:50:38Z

Hi

First of all revoke your token, it's in these logs.
Second. what is the form of SOCKS_URL. It should start with socks:// for Request to recognize a socks connection.

maxgrechnev · 2018-04-18T10:15:32Z

@Eldinnie thanks, token's revoked, but don't think I will suffer from stealing my token :) the bot was created only for reproducing this issue.
Of course my SOCKS_URL = 'socks5://<host>:1080/'. If I use socks://, I get the error:
ValueError: Unable to determine SOCKS version

Eldinnie · 2018-04-18T10:26:28Z

Ok thanks. Could you try another thing.
add a second test similar to the first, but instead of using urllib3 use
telegram.vendor.ptb_urllib3.urllib3.contrib.socks import SOCKSProxyManager
It's our vendored one used internally so we can see if the problem is in the proxymanager or further down the line

maxgrechnev · 2018-04-18T10:34:40Z

Traceback (most recent call last):
  File "/usr/lib/test/test.py", line 17, in <module>
	response = manager.request('GET', 'https://api.telegram.org/bot' + TOKEN + '/getMe')
  File "/usr/local/lib/python2.7/dist-packages/telegram/vendor/ptb_urllib3/urllib3/request.py", line 66, in request
	**urlopen_kw)
  File "/usr/local/lib/python2.7/dist-packages/telegram/vendor/ptb_urllib3/urllib3/request.py", line 87, in request_encode_url
	return self.urlopen(method, url, **extra_kw)
  File "/usr/local/lib/python2.7/dist-packages/telegram/vendor/ptb_urllib3/urllib3/poolmanager.py", line 244, in urlopen
	response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/local/lib/python2.7/dist-packages/telegram/vendor/ptb_urllib3/urllib3/connectionpool.py", line 695, in urlopen
	**response_kw)
  File "/usr/local/lib/python2.7/dist-packages/telegram/vendor/ptb_urllib3/urllib3/connectionpool.py", line 695, in urlopen
	**response_kw)
  File "/usr/local/lib/python2.7/dist-packages/telegram/vendor/ptb_urllib3/urllib3/connectionpool.py", line 695, in urlopen
	**response_kw)
  File "/usr/local/lib/python2.7/dist-packages/telegram/vendor/ptb_urllib3/urllib3/connectionpool.py", line 666, in urlopen
	_stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python2.7/dist-packages/telegram/vendor/ptb_urllib3/urllib3/util/retry.py", line 376, in increment
	raise MaxRetryError(_pool, url, error or ResponseError(cause))
telegram.vendor.ptb_urllib3.urllib3.exceptions.MaxRetryError: SOCKSHTTPSConnectionPool(host='api.telegram.org', port=443): Max retries exceeded with url: /botTOKEN/getMe (Caused by NewConnectionError('<telegram.vendor.ptb_urllib3.urllib3.contrib.socks.SOCKSHTTPSConnection object at 0x763c21d0>: Failed to establish a new connection: 0x02: Connection not allowed by ruleset',))

maxgrechnev · 2018-04-18T10:45:18Z

@Eldinnie I see your vendored fork of urllib3 is not in sync to the current urllib3 original repo.
Can it be a reason for the issue?

Eldinnie · 2018-04-18T11:07:38Z

Yes I believe this is the root of the problem. So I hand it over to @tsnoam

maxgrechnev · 2018-04-18T11:16:15Z

@Eldinnie @tsnoam Sorry for annoying you, but could you explain why you forked it? Why not just link to the original one or add a dependency?

tsnoam · 2018-04-19T12:12:37Z

@maxgrechnev

I've set a socks5 using ssh -D and couldn't recreate your problem, neither with python 2.7 nor python 3.6.
I see that you are using authentication to your socks5 server, so I'll try to set such server and see then.
To your question: we're using a vendored urllib3 because there's a bug which is very hard to recreate which we have a local fix for. We haven't yet managed to get our fix upstream. So far we're pleased with the stability of our vendored version so we see no reason to upgrade.

maxgrechnev · 2018-04-19T12:48:01Z

@tsnoam thank you!
Could you tell about the bug you mention? What can I run into if I use the original urllib3 instead of your vendored version?

tsnoam · 2018-04-19T12:55:25Z

@maxgrechnev
I've set a socks5 server with authentication but still couldn't create the error you're experiencing.

For future reference, this is the setup I've created:
docker image running ubuntu18.04
apt install dante-server
adduser user
passwd user (set password to pass)

edit /etc/danted.conf

logoutput: stderr
internal: 0.0.0.0 port=1080
external: eth0
socksmethod: username
clientmethod: none
user.privileged: root
user.unprivileged: nobody
user.libwrap: nobody
client pass {
	from: 0.0.0.0/0 to: 0.0.0.0/0
	log: connect disconnect error
}
socks pass {
	from: 0.0.0.0/0 to: 0.0.0.0/0
	log: connect disconnect error
}

run dante:

danted

tsnoam · 2018-04-19T12:58:53Z

@maxgrechnev
I looked at the error message you got: Connection not allowed by ruleset
It seems to be PySocks literal for errcode 0x02 for errors received from the server side. Can you please check your socks server for logs? It might have more indication on what's going wrong.

tsnoam · 2018-04-19T13:03:35Z

@maxgrechnev
check issue #533 regrading the bug we fixed in urllib3.
i don't know how relevant it is for socks5 users.

maxgrechnev · 2018-04-20T12:59:29Z

@tsnoam I have contacted the provider of the proxy I use. He noticed the difference between two requests:

urllib3 requests IP address of Telegram Bot API,
your vendored lib requests domain name api.telegram.org.

I don't know which behaviour is correct, maybe both.
But the second request fails because of filtering domain names and passing only direct IP requests on the proxy server side.

tsnoam · 2018-04-20T17:18:55Z

@maxgrechnev Thanks for the clarification. Everything makes much more sense now.
It appears that urllib3 had committed a semantic behaviour fix to properly diffrentiate between socks5 & socks5h ( urllib3/urllib3#1036 ).
This is indeed a good reason to upgrade urllib3 (or at least cherry pick this commit).

On a side note, I would personally recommend that you'd use a proxy provider which supports resolution of hostnames over the proxy. With the latest developments in Russia I would be surprised if they hadn't already tampered with DNS records.

ghost · 2018-04-20T19:51:14Z

How will the ready-made solution for launching python-telegram-bot through a proxy for Russia look like, using the user and password?

tsnoam · 2018-04-21T06:56:02Z

@ruslan1990
I am unable to answer this question, not being from this region myself. I can only assume that you would require an encrypted tunnel of some sort (VPN, proxy, ssh tunnel, etc.). The solution should not be fundamentally different than how you'd connect with your Telegram.
In anycase, this is offtopic for this issue.

ghost · 2018-04-21T07:29:20Z

@tsnoam
I use this code.

request_kwargs = {'proxy_url':'socks5://185.211.245.141:1090/', 'urllib3_proxy_kwargs':{'username':'499962286', 'password':'pass'}}
updater = Updater('TOKEN', request_kwargs=request_kwargs)

But I get this error.


2018-04-21 12:22:40,164 - telegram.vendor.ptb_urllib3.urllib3.connectionpool - WARNING - Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<telegram.vendor.ptb_urllib3.urllib3.contrib.socks.SOCKSHTTPSConnection object at 0x0534D970>: Failed to establish a new connection: 0x01: General SOCKS server failure',)': /botTOKEN/deleteWebhook
2018-04-21 12:22:41,163 - telegram.vendor.ptb_urllib3.urllib3.connectionpool - WARNING - Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<telegram.vendor.ptb_urllib3.urllib3.contrib.socks.SOCKSHTTPSConnection object at 0x0534D9F0>: Failed to establish a new connection: 0x01: General SOCKS server failure',)': /botTOKEN/deleteWebhook
2018-04-21 12:22:42,161 - telegram.vendor.ptb_urllib3.urllib3.connectionpool - WARNING - Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<telegram.vendor.ptb_urllib3.urllib3.contrib.socks.SOCKSHTTPSConnection object at 0x05363510>: Failed to establish a new connection: 0x01: General SOCKS server failure',)': /botTOKEN/deleteWebhook
2018-04-21 12:22:43,158 - telegram.ext.updater - ERROR - Error while bootstrap del webhook: urllib3 HTTPError SOCKSHTTPSConnectionPool(host='api.telegram.org', port=443): Max retries exceeded with url: /bot TOKEN /deleteWebhook (Caused by NewConnectionError('<telegram.vendor.ptb_urllib3.urllib3.contrib.socks.SOCKSHTTPSConnection object at 0x05363CB0>: Failed to establish a new connection: 0x01: General SOCKS server failure',))
2018-04-21 12:22:43,206 - telegram.ext.updater - WARNING - Failed bootstrap phase; try=1 max_retries=-1

These proxy parameters work in the telegram application on the phone.

tsnoam · 2018-04-21T07:32:25Z

@ruslan1990 First of all, replace your password to the socks proxy. You've just posted it cleartext.

Regarding the error you get, it's a server side error. You should check the socks server logs for more insight.

ghost · 2018-04-21T08:49:07Z

@tsnoam
With the same settings, pyTelegramBotAPI works.

tsnoam · 2018-04-21T09:02:59Z

@ruslan1990 The error message is from the server side. Please check the server logs so we can understand what the problem is.

tsnoam · 2018-04-21T17:06:17Z

@maxgrechnev @ruslan1990
We've committed the socks5h fix from upstream urllib3 into our fork.
We would be very grateful if you could test it and see if it solves your issue before we make an official release.

You can get this patched version by running:

git clone --branch=urllib3_socks5h --recursive https://github.com/python-telegram-bot/python-telegram-bot

ghost · 2018-04-21T22:03:37Z

@tsnoam
I installed the patch, but it did not help. The same errors appear.

tsnoam · 2018-04-21T22:05:11Z

@ruslan1990 Have you created a new clone of the git repo exactly as specified in the command I supplied?

ghost · 2018-04-22T11:46:05Z

@tsnoam Yes, I use copy paste.

tsnoam · 2018-04-22T11:49:26Z

@ruslan1990 In that case, I believe that this is a different issue than @maxgrechnev experienced.
Please open a new issue in github. doing so, please contact your socks proxy provider and ask him what is the error on his side, as it is reported as a server error.

tsnoam · 2018-04-22T11:50:25Z

@maxgrechnev
Have you had the chance to try the patch?

ghost · 2018-04-22T12:09:12Z

@tsnoam
I'm sorry, your patch works, I just installed it incorrectly.

tsnoam · 2018-04-22T12:09:40Z

@ruslan1990 Thanks for the update.

maxgrechnev · 2018-04-22T17:47:10Z

@tsnoam
I will do it tomorrow.

maxgrechnev · 2018-04-24T07:36:02Z

@tsnoam Glad to inform you that the fix works well. Thank you!

tsnoam · 2018-04-25T18:02:08Z

@maxgrechnev
Thanks for testing and updating us. We'll merge this to master and try to release a new version asap.

FIxes #1077 Changes semantics of socks proxy behaviour. Until now - socks5:// was actually socs5h:// and we didn't really have socks5://

Eldinnie added the bug 🐛 label Apr 18, 2018

tsnoam added pending reply and removed bug 🐛 labels Apr 19, 2018

tsnoam self-assigned this Apr 19, 2018

tsnoam removed the pending reply label Apr 25, 2018

tsnoam mentioned this issue Apr 25, 2018

Add urllib3 fix for socks5h support #1085

Merged

tsnoam closed this as completed in #1085 Apr 25, 2018

tsnoam added a commit that referenced this issue Apr 25, 2018

Add urllib3 fix for socks5h support (#1085)

1ec1234

FIxes #1077 Changes semantics of socks proxy behaviour. Until now - socks5:// was actually socs5h:// and we didn't really have socks5://

rahiel added the bug 🐛 label Apr 25, 2018

github-actions bot locked and limited conversation to collaborators Aug 23, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Socks proxy error #1077

Socks proxy error #1077

maxgrechnev commented Apr 18, 2018 •

edited

Loading

Eldinnie commented Apr 18, 2018

maxgrechnev commented Apr 18, 2018

Eldinnie commented Apr 18, 2018

maxgrechnev commented Apr 18, 2018

maxgrechnev commented Apr 18, 2018

Eldinnie commented Apr 18, 2018

maxgrechnev commented Apr 18, 2018

tsnoam commented Apr 19, 2018

maxgrechnev commented Apr 19, 2018

tsnoam commented Apr 19, 2018

tsnoam commented Apr 19, 2018

tsnoam commented Apr 19, 2018

maxgrechnev commented Apr 20, 2018 •

edited

Loading

tsnoam commented Apr 20, 2018 •

edited

Loading

ghost commented Apr 20, 2018 •

edited by ghost

Loading

tsnoam commented Apr 21, 2018

ghost commented Apr 21, 2018 •

edited by ghost

Loading

tsnoam commented Apr 21, 2018

ghost commented Apr 21, 2018

tsnoam commented Apr 21, 2018

tsnoam commented Apr 21, 2018

ghost commented Apr 21, 2018

tsnoam commented Apr 21, 2018 •

edited

Loading

ghost commented Apr 22, 2018

tsnoam commented Apr 22, 2018

tsnoam commented Apr 22, 2018

ghost commented Apr 22, 2018

tsnoam commented Apr 22, 2018

maxgrechnev commented Apr 22, 2018

maxgrechnev commented Apr 24, 2018 •

edited

Loading

tsnoam commented Apr 25, 2018

Socks proxy error #1077

Socks proxy error #1077

Comments

maxgrechnev commented Apr 18, 2018 • edited Loading

Steps to reproduce

Expected behaviour

Actual behaviour

Configuration

Logs

Eldinnie commented Apr 18, 2018

maxgrechnev commented Apr 18, 2018

Eldinnie commented Apr 18, 2018

maxgrechnev commented Apr 18, 2018

maxgrechnev commented Apr 18, 2018

Eldinnie commented Apr 18, 2018

maxgrechnev commented Apr 18, 2018

tsnoam commented Apr 19, 2018

maxgrechnev commented Apr 19, 2018

tsnoam commented Apr 19, 2018

tsnoam commented Apr 19, 2018

tsnoam commented Apr 19, 2018

maxgrechnev commented Apr 20, 2018 • edited Loading

tsnoam commented Apr 20, 2018 • edited Loading

ghost commented Apr 20, 2018 • edited by ghost Loading

tsnoam commented Apr 21, 2018

ghost commented Apr 21, 2018 • edited by ghost Loading

tsnoam commented Apr 21, 2018

ghost commented Apr 21, 2018

tsnoam commented Apr 21, 2018

tsnoam commented Apr 21, 2018

ghost commented Apr 21, 2018

tsnoam commented Apr 21, 2018 • edited Loading

ghost commented Apr 22, 2018

tsnoam commented Apr 22, 2018

tsnoam commented Apr 22, 2018

ghost commented Apr 22, 2018

tsnoam commented Apr 22, 2018

maxgrechnev commented Apr 22, 2018

maxgrechnev commented Apr 24, 2018 • edited Loading

tsnoam commented Apr 25, 2018

maxgrechnev commented Apr 18, 2018 •

edited

Loading

maxgrechnev commented Apr 20, 2018 •

edited

Loading

tsnoam commented Apr 20, 2018 •

edited

Loading

ghost commented Apr 20, 2018 •

edited by ghost

Loading

ghost commented Apr 21, 2018 •

edited by ghost

Loading

tsnoam commented Apr 21, 2018 •

edited

Loading

maxgrechnev commented Apr 24, 2018 •

edited

Loading