Wild exception appeared with large area scans with many accounts (480)

[Baklap4]

Trying to map my whole city and a few other towns around it with about 57 square km.
I'm using 480 ptc accounts with this.

I can scan the whole region if i set this up in steps.
1 - scan a smaller region
2 - scan smaller region with more accounts
3 - scan bigger region
4 - scan bigger region with more accounts
etc till i've reached my whole region.

But after a while (say 30 mins) it tries to log them all in again and i'm getting a lot of exceptions.
The same happens if i mark my whole region and all those 480 ptc accounts for the first time.

Some exceptions are:

[2016-08-11 15:00:02,596][worker-128][   ERROR][L 125] A wild exception appeared!
Traceback (most recent call last):
  File "worker.py", line 103, in run
    password=config.ACCOUNTS[self.worker_no][1],
  File "/home/baklap4/pokeminer/src/pgoapi/pgoapi/pgoapi.py", line 162, in login
    self.set_authentication(provider, username=username, password=password)
  File "/home/baklap4/pokeminer/src/pgoapi/pgoapi/pgoapi.py", line 82, in set_authentication
    self._auth_provider.user_login(username, password)
  File "/home/baklap4/pokeminer/src/pgoapi/pgoapi/auth_ptc.py", line 63, in user_login
    r = self._session.get(self.PTC_LOGIN_URL, headers=head)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 487, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 467, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='sso.pokemon.com', port=443): Max retries exceeded with url: /sso/login?service=https%3A%2F%2Fsso.pokemon.com%2Fsso%2Foauth2.0%2FcallbackAuthorize (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f8464206950>: Failed to establish a new connection: [Errno 24] Too many open files',))

OR

[2016-08-11 15:00:02,347][worker-289][   ERROR][L 140] A wild exception appeared!
Traceback (most recent call last):
  File "worker.py", line 135, in run
    self.main()
  File "worker.py", line 177, in main
    map_objects = response_dict['responses'].get('GET_MAP_OBJECTS', {})
TypeError: 'NoneType' object has no attribute '__getitem__'
[2016-08-11 15:00:02,347][worker-184][    INFO][L 253] Server seems to be busy or offline - try again!
[2016-08-11 15:00:02,347][worker-184][    INFO][L 260] Cleanup of request!
[2016-08-11 15:00:02,347][worker-184][    INFO][L 147] Finished RPC login sequence (app simulation)

Or

[2016-08-11 15:01:30,938][ worker-99][   ERROR][L 125] A wild exception appeared!
Traceback (most recent call last):
  File "worker.py", line 103, in run
    password=config.ACCOUNTS[self.worker_no][1],
  File "/home/baklap4/pokeminer/src/pgoapi/pgoapi/pgoapi.py", line 168, in login
    response = self.app_simulation_login()
  File "/home/baklap4/pokeminer/src/pgoapi/pgoapi/pgoapi.py", line 145, in app_simulation_login
    response = request.call()
  File "/home/baklap4/pokeminer/src/pgoapi/pgoapi/pgoapi.py", line 226, in call
    response = request.request(self._api_endpoint, self._req_method_list, self.get_position())
  File "/home/baklap4/pokeminer/src/pgoapi/pgoapi/rpc_api.py", line 131, in request
    response = self._make_rpc(endpoint, request_proto)
  File "/home/baklap4/pokeminer/src/pgoapi/pgoapi/rpc_api.py", line 119, in _make_rpc
    http_response = self._session.post(endpoint, data=request_proto_serialized, timeout=30)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 518, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 403, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 578, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 354, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 324, in _raise_timeout
    if 'timed out' in str(err) or 'did not complete (read)' in str(err):  # Python 2.6
TypeError: __str__ returned non-string (type Error)

Any help would be useful!

EDIT

It seems that when i start with little workers (4,5) and then increase to use all my workers (20,24) it works but i have to kill the app in between :/
But once it decides to sleep and eventually kill the workers i have to stop the app to prevent exceptions (which will come in about 15 minutes)

[SapitoSucio]

My whole area scan is 33 km^2 and I'm using 50 workers and it's fine, maybe you're using so many.

[Baklap4]

@DavidSelem i could do that but in the end i want to map whole the netherlands.
Which eventually will be needed like 3000 logins..
Running 50 accounts does indeed work but is way too slow to map a whole country or even a region.

Besides especially as this readme says the following:

Bulletpoint list of features

• multithreaded, multiple accounts at the same time
• aims at being very stable for long-term runs
• able to map entire city (or larger area) in real time
• gathers Pokemon and Gyms
• data gathering for further analysis
• visualization
• reports for gathered data

Currently i find this not very stable for long term run with many accounts. Max it'll reach is 15 minutes without bugging out

[Aiyubi]

• Definitly too many workers - there is something about the workernumber in the readme. (unless you have a huge scan delay...)

Exceptions:

1. [Errno 24] Too many open files
   There is a limit of open files your OS can have. You can look into increasing this if you really want to. Check ulimit as a keyword for linux
2. TypeError: 'NoneType' object has no attribute 'getitem'
   Yeah there is a check missing - will look into this. This happens when the server says there is nothing to see. This happens when you a) scan too fast, b) there is nothing in that area, c) The server gets too many requests from your IP and just ignores your requests.
   Yes I'm getting this one myself too often but mapping the whole netherlands will definitly need proxys to get around the limits and in my opinion is over the top.
3. That one I have not seen yet. Maybe related to 1. But thats an error in the API

[Baklap4]

@Aiyubi I've check the ulimit. This is default set to 1024.

Furthermore the scanning just works fine when mapping a very big area.
Just when logging in it creates those exceptions.
Logging in 20 workers at the same time is peanuts.
After logging in those 20 workers i can log in all those others 460 accounts with no problem.

So it could be possible scanning the whole netherlands.
About sending too many requests from my ip i'm currently using just one proxy account which works great with these 480 accounts. Except from logging in. After they're logged in it crashes and results every worker to get the status exception or login fail.

[Baklap4]

just ran another test
It seems the proxy i was using was not that great.
I've entered a new proxy and i can easily log in those 480 accounts. I wonder what it does when it reaches the first sleeps/killing states (after 15 minutes) I'll keep ya guys updated

[Baklap4]

Just ran two tests.
One where i've upgraded my ulimit to 40.000 open files. This resulted in instant exceptions. When reverted back i've ran it again. Still getting exceptions after 10-15 minutes.
I can however log on to those 480 accounts the first time i run this script. It takes a while but it works
I'm using sqllite btw. I don't know how to use mysql

[2016-08-11 16:53:06,465][worker-128][   ERROR][L 125] A wild exception appeared!
Traceback (most recent call last):
  File "worker.py", line 103, in run
    password=config.ACCOUNTS[self.worker_no][1],
  File "/home/baklap4/pokeminer/src/pgoapi/pgoapi/pgoapi.py", line 162, in login
    self.set_authentication(provider, username=username, password=password)
  File "/home/baklap4/pokeminer/src/pgoapi/pgoapi/pgoapi.py", line 82, in set_authentication
    self._auth_provider.user_login(username, password)
  File "/home/baklap4/pokeminer/src/pgoapi/pgoapi/auth_ptc.py", line 63, in user_login
    r = self._session.get(self.PTC_LOGIN_URL, headers=head)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 487, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 467, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='sso.pokemon.com', port=443): Max retries exceeded with url: /sso/login?service=https%3A%2F%2Fsso.pokemon.com%2Fsso%2Foauth2.0%2FcallbackAuthorize (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7faa743e19d0>: Failed to establish a new connection: [Errno 24] Too many open files',))
[2016-08-11 16:53:06,473][worker-126][   ERROR][L 125] A wild exception appeared!
Traceback (most recent call last):
  File "worker.py", line 103, in run
    password=config.ACCOUNTS[self.worker_no][1],
  File "/home/baklap4/pokeminer/src/pgoapi/pgoapi/pgoapi.py", line 162, in login
    self.set_authentication(provider, username=username, password=password)
  File "/home/baklap4/pokeminer/src/pgoapi/pgoapi/pgoapi.py", line 82, in set_authentication
    self._auth_provider.user_login(username, password)
  File "/home/baklap4/pokeminer/src/pgoapi/pgoapi/auth_ptc.py", line 63, in user_login
    r = self._session.get(self.PTC_LOGIN_URL, headers=head)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 487, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 403, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 578, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 354, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 324, in _raise_timeout
    if 'timed out' in str(err) or 'did not complete (read)' in str(err):  # Python 2.6
TypeError: __str__ returned non-string (type Error)

[Baklap4]

I'm also getting a lot of these errors:

Traceback (most recent call last):
  File "worker.py", line 103, in run
    password=config.ACCOUNTS[self.worker_no][1],
  File "/home/baklap4/pokeminer/src/pgoapi/pgoapi/pgoapi.py", line 168, in login
    response = self.app_simulation_login()
  File "/home/baklap4/pokeminer/src/pgoapi/pgoapi/pgoapi.py", line 145, in app_simulation_login
    response = request.call()
  File "/home/baklap4/pokeminer/src/pgoapi/pgoapi/pgoapi.py", line 226, in call
    response = request.request(self._api_endpoint, self._req_method_list, self.get_position())
  File "/home/baklap4/pokeminer/src/pgoapi/pgoapi/rpc_api.py", line 131, in request
    response = self._make_rpc(endpoint, request_proto)
  File "/home/baklap4/pokeminer/src/pgoapi/pgoapi/rpc_api.py", line 119, in _make_rpc
    http_response = self._session.post(endpoint, data=request_proto_serialized, timeout=30)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 518, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 403, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 578, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 351, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 814, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connection.py", line 289, in connect
    ssl_version=resolved_ssl_version)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 350, in ssl_wrap_socket
    rd, _, _ = select.select([sock], [], [], sock.gettimeout())
ValueError: filedescriptor out of range in select()

This has to do with more than the 1024 connections what are made with so many accounts.
According to this question: http://stackoverflow.com/questions/14250751/how-to-increase-filedescriptors-range-in-python-select
The way to go is using .poll() instead of select() as this does not suffer from it?

[gunawanputra]

@Baklap4 there is an issue on pyOpenSSL pyopenssl/issues/168 urllib3/urllib3/issues/589

[Baklap4]

@gunawanputra so is there a way to remove the timeout parameter? (to make the workaround work?)

[Baklap4]

Hmmm after starting my laptop today it does not fall into exceptions.
Yesterday i've upgraded my ulimit to 20480. Then did a restart (which was not needed they said.) Well right now it does not crash so i think it is "solved"?

[Baklap4]

Alright another update
I've stopped the worker.py script myself after 3 hours without exceptions.
Now i'm restarting it with no additions to the config or whatsoever. And i'm getting the same exceptions again as i was getting yesterday (shown above). The problem still persists :/
I've tried restarting my pc again to see if this helped it didn't

[Baklap4]

Alright i think i've figured out what the problem is.
There are more than 1024 outgoing connections if i have more than 500 accounts running.
Since linux can't have more than 1024 outgoing connections i'm stuck.
I've tried things like changing the port range and so it didn't work. Any other ideas?

[Baklap4]

Running the programs separate from each other works while exceeding the 1024 limit.

[subzerofun]

You really can run 3000 accounts at the same time? Wow... How do you not get banned when starting the workers up?

I can only safely use around 200 workers (mapping 2 small cities), everything above ~220 just throws LOGIN_FAILED or EXCEPTION. i then wait for a few minutes and reset my vpn service to get a new IP and everything works again - as long as i´m using less than 220-250 workers.

yesterday i tried to fire up 400 accounts, but not a single one logged in... do you use a mix of PTC and google accounts or just PTC?

[Baklap4]

I can safely run up to 480 acvounts on one instance.
The thing you see with 'login bad' or 'login failed' is just saying something went wrong during logging in. if you see 'exception' it could mean youve hit your max limit of open outgoing connections. In my case that would be 1024 per application.

With 480 accounts i reach around 900 'open' connections. Which is less than 1024 so not crashing into exceptions.
Im currently running 2 instances with 480 and 375 ptc accounts all coming from 1 ip address. Its been running fine for 12 hours straight now with a remote mysql database.

Im also noticing rhings with the upload speed.
Since my database is remote i have to upload data to the database. Im on a 2mbps uplink which is not much. Therefore i see sometimes an exception/loginfailure.

[Kobito123]

when i use 330 bots i get a temporary ip bann :O what did u do to not get banned ?