No Pokemon returned due to soft ban

[Matsu16]

No pokemon. 2 minutes ago it was still working. whats going on. no errors ..scans are just fine

[dikkedeur]

it seems there is a problem with having a lot of accounts.
i did a test on a small area on my city, with 2 accounts it all looks okay, so i expand the radius to 2,7KM covering my whole city with 12 accounts.
i see then a lot less pokemons then there should be, some area's are blank.

so it seems that it then goes to fast, a full scan take roughly 4 minutes.

[Matsu16]

It starts up perfectly but it stops showing after like 5 min. Got 240 accounts on 4km

How do i post javascript console?

[dikkedeur]

did another run on a total different location outside my city, no pokemns show up anymore.. it keeps on scanning but nothing pops up.
no error in javascript console ( or i might be looking wrong )

is it possible that the database is corrupt and how to fix it? i dont mind doing a rescan it only takes 5 minutes max..
can i just delete the database file and will it be remade??

[dikkedeur]

no that is not the problem i run 1 instance just port 5000, and i did a new fresh db pull, and that doesnt work either.
it seems that nia is blocking too much requests, but then again it would trow out an error i assume.
and that is not the case.

17.94% of scan.
2016-08-09 14:17:54,012 [ scan] [ INFO] Completed 18.00% of scan.
2016-08-09 14:17:54,575 [ scan] [ INFO] Completed 18.05% of scan.
2016-08-09 14:17:59,757 [ scan] [ INFO] Completed 18.11% of scan.
2016-08-09 14:17:59,832 [ scan] [ INFO] Completed 18.17% of scan.
2016-08-09 14:18:00,285 [ scan] [ INFO] Completed 18.22% of scan.
2016-08-09 14:18:00,718 [ scan] [ INFO] Completed 18.28% of scan.
2016-08-09 14:18:01,174 [ scan] [ INFO] Completed 18.33% of scan.
2016-08-09 14:18:01,547 [ scan] [ INFO] Completed 18.39% of scan.
2016-08-09 14:18:02,146 [ scan] [ INFO] Completed 18.45% of scan.
2016-08-09 14:18:02,279 [ scan] [ INFO] Completed 18.50% of scan.
2016-08-09 14:18:03,011 [ scan] [ INFO] Completed 18.56% of scan.
2016-08-09 14:18:03,204 [ scan] [ INFO] Completed 18.61% of scan.
2016-08-09 14:18:04,175 [ scan] [ INFO] Completed 18.67% of scan.
2016-08-09 14:18:04,743 [ scan] [ INFO] Completed 18.73% of scan.
2016-08-09 14:18:09,924 [ scan] [ INFO] Completed 18.78% of scan.
2016-08-09 14:18:10,200 [ scan] [ INFO] Completed 18.84% of scan.
2016-08-09 14:18:10,499 [ scan] [ INFO] Completed 18.89% of scan.
2016-08-09 14:18:10,884 [ scan] [ INFO] Completed 18.95% of scan.
2016-08-09 14:18:11,339 [ scan] [ INFO] Completed 19.01% of scan.

it just keeps on going.. so either they are sending just null, that can be it?
but how to determine if that is the case?

[dikkedeur]

well what do you know... suddenly something pops up, i dont know for sure if everything is returning right now.. we must wait and let it scan for some time to see if it works again and that it is not a lucky coincidence.

edit: it goes way slower then before... dont know why but i see now pokemons popping up on the map.. its still scanning my area, at a slower rate.

[nborrmann]

I have the assumption that this is in fact be a rate-limit.

In cbea6b3 I pushed a change, which will print a warning if that's the case. Please report back whether or not you get that warning.

I am sometimes getting the warning as well, but it doesn't seem to follow any logic when it occurs. Maybe you guys find something?

[djv2]

i'm 99% sure you got an IP ban. i have the same issue
@nborrmann do you think you can make a notice when you get an IP ban? niatnic returns 0 of everything when it happens

[dikkedeur]

got it indeed..

2016-08-09 14:39:37,030 [ models] [WARNING] Received valid response but without any data. Possibly rate-limited?
2016-08-09 14:39:37,030 [ scan] [ INFO] Completed 0.22% of scan.
2016-08-09 14:39:37,030 [ pgoapi] [ INFO] Attempting login: stinkpoep
2016-08-09 14:39:37,056 [ models] [WARNING] Received valid response but without any data. Possibly rate-limited?
2016-08-09 14:39:37,056 [ scan] [ INFO] Completed 0.28% of scan.
2016-08-09 14:39:37,056 [ pgoapi] [ INFO] Attempting login: eenhooppoep
^X2016-08-09 14:39:37,121 [ models] [WARNING] Received valid response but without any data. Possibly rate-limited?
2016-08-09 14:39:37,122 [ scan] [ INFO] Completed 0.34% of scan.
2016-08-09 14:39:37,122 [ pgoapi] [ INFO] Attempting login: flapperdeflap

[nborrmann]

For me, those errors just disappear after some time.

[dikkedeur]

and now i got less warnings.. so there is when you have too much accounts a possibility that it will sent nothing to you..

[Matsu16]

And if i make the area bigger. Would that solve a rate limit?

[dikkedeur]

not a change... its due more accounts im afraid.. the scanning goes also slower it seems, so it might be that nia is doing their magic and limiting requests, since its slower then before and if you go to fast ( read more accounts from same IP ) then they just limit you more ( assumption im not an expert on this case ^^)

[PhoenixICE]

this would mean that public wifi areas would have max accounts that can connect to them, interesting...

[nborrmann]

I am only getting these errors very occasionally:

[Matsu16]

I just think were pushing to many workers. But als not an expert. Wonder how many accounts will be stable for my area then..

[djv2]

@nborrmann i am getting it a lot with 100 accounts

would it be possible, to in the code, add something that says if 1 account is returning 0 results after like 5 retries, skip that account

[Matsu16]

Djv2, got the same screen u got with 240 accounts on 4km.

[dikkedeur]

get them now also more

d valid response but without any data. Possibly rate-limited?
2016-08-09 15:01:24,645 [ scan] [ INFO] Completed 79.65% of scan.
2016-08-09 15:01:25,146 [ models] [WARNING] Received valid response but without any data. Possibly rate-limited?
2016-08-09 15:01:25,146 [ scan] [ INFO] Completed 79.71% of scan.
2016-08-09 15:01:25,551 [ scan] [ INFO] Completed 79.77% of scan.
2016-08-09 15:01:25,705 [ scan] [ INFO] Completed 79.82% of scan.
2016-08-09 15:01:26,344 [ scan] [ INFO] Completed 79.88% of scan.
2016-08-09 15:01:29,115 [ scan] [ INFO] Completed 79.93% of scan.
2016-08-09 15:01:29,702 [ scan] [ INFO] Completed 79.99% of scan.
2016-08-09 15:01:29,835 [ pgoapi] [ INFO] Server seems to be busy or offline!
2016-08-09 15:01:30,675 [ scan] [ INFO] Completed 80.04% of scan.
2016-08-09 15:01:32,655 [ scan] [ INFO] Completed 80.10% of scan.
2016-08-09 15:01:33,470 [ scan] [ INFO] Completed 80.16% of scan.
2016-08-09 15:01:34,556 [ scan] [ INFO] Completed 80.21% of scan.
2016-08-09 15:01:34,812 [ scan] [ INFO] Completed 80.27% of scan.
2016-08-09 15:01:35,319 [ scan] [ INFO] Completed 80.32% of scan.
2016-08-09 15:01:35,712 [ models] [WARNING] Received valid response but without any data. Possibly rate-limited?
2016-08-09 15:01:35,712 [ scan] [ INFO] Completed 80.38% of scan.
2016-08-09 15:01:35,889 [ models] [WARNING] Received valid response bu

and also

2016-08-09 15:01:29,835 [ pgoapi] [ INFO] Server seems to be busy or offline!

[nborrmann]

This is definitely not an IP-based ban. It seems to be per account.

So adding more accounts should actually resolve this, because the time between requests on the same account will be larger

[johved]

Is there a system in place in choosing which accounts scans what location?

Because Niantec uses soft-bans if an account has travelled extremly fast i.e. jumps from one location to the next instantly. I think soft-bans return zero pokemons and last for something between 0.5-3 hours. I thought it was account specific but perhaps it also is IP specific?. Anyway I guess people should not have locations very far apart if the server doesnt keep the same accounts at separate locations.

[djv2]

@nborrmann it definitely IS an IP-based ban. if i change my IP (via VPN), no more errors during scan

[Matsu16]

Are u saying we need te make more😂

[Matsu16]

Good question @johved

[nborrmann]

If this were IP-based all requests would fail, instead of just a few.

The errors are soft account bans, as @johved describes.

[Matsu16]

Then there should be a system like @johved is talking about

[Matsu16]

Almost every account got a softban now. Just waiting for a solution.

[dikkedeur]

for me its still working, but its better not to scan large area's with alot of accounts form same ip just to be sure

[dikkedeur]

the best would be ( just a theory dunno if its possible) make it that a few accounts are attached to a small area ( so not scan large area in 1 blow ) then multiple instances running in batches ( dont know how to explain it better )

in short, you have if you have 12 accounts 4 small area;s with 3 accounts attached to 1 area, they scan it... after 100% wait 20 seconds before doing rescan..
that could limit it... maybe.

[CuRleGit]

Ah! @PhoenixICE is right.

I just dropped a tracker on an area I knew would not have any Pokemon and suddendly my console filled up with "Received valid response but...".

Then I put a small tracker in a densely populated area which I know WILL have Pokemon and the message does not appear at all.

I've been running my tracker in a semi-rural area, 50% of the lines on my console are "Received valid response but..." so I thought I was getting throttled.

(this is using 20 accounts)

Time to create and add more accounts.

[nborrmann]

@CuRleGit As explained in the FAQ:

In this case the server returns a perfectly valid response, but it contains no map data. In rural areas you might get this error without being blocked, if there just aren't any map objects (pokemon, pokestops, spawnpoints) in the scanned area.

[eXeLe]

My ~1000 was working pretty good for the 24+hrs with sqlite.
18km radius in ~1hr10min

Switched for MariaDB, it works faster, but IP ban again after ~15min =(

Seems like slowdown from DB throttling helped to stay out of Niantic ban-sensors.

[Mickey32111]

Something I've noticed is that the scan will be doing fine not receiving any rate-limited errors, but then it'll slow down, maybe due to bottlenecking on the system, idk, but then I guess it tries to play catchup and the scan speeds up exponentially, throwing tonnes of rate-limited errors out.

[nborrmann]

@Mickey32111 A lot of time is spent on parsing and inserting into the DB. If the response is empty, it won't spend any time there and it will be able to download much faster.

[eugenio412]

Same to me, I get banned after 15 min, than if I stay quiet for an hour I can scan other 15 min, before getting banned again.
I have 1km radius with 100 acc's.
But it seems to be IP related, it works using my phone tethering while banned on my VPS. I guess we need some Tor, or proxies, to get different ips with different threads.

[CuRleGit]

Anyone been able to use 30+ accounts?
I tried bumping my scanner up to 30 (from 20) but got an IP ban pretty quickly.

[hacwa]

I have been scanning 2, 3 and 4km areas using between 15-60 accounts.
I got the least errors using 20 accounts on a 2.5km area, when i get too many errors i change my ip and it instantly corrects it.

[EternisRequiem]

@hacwa How would you suggest doing that? Been having a lot of troubles but I'm not super familiar with VPN and the like.

[labaland]

@hacwa Can you use your pogomap från another public ip When u host it On vpn? I cant connect to my server.. Only local, ports are open On router and vpn service :(

[CuRleGit]

@labaland
What VPN are you using?
I was unable to get it working with PIA. I believe I had everything set up correctly and a port forwarding test site was saying my port was open, but I could not connect to my map.

I ended up trying ngrok. Very easy to setup and I can not connect to my map running through a VPN.

[hacwa]

I am using "Private Internet Access" VPN, excuse the noobness plz. new to this myself.
I use the PIA program that i downloaded from there website and when i get quite alot of errors i disconnect from UK southhampton for example and change to any other area they have available like UK london or US texas and it clears up straight away.

The way i use the map is by having a laptop in the back of the car connected to my phone which is set as an AP, the laptop runs pogom runserver. i connect my tablet to my phone AP and just use the phones IP:port into the tablets browser.

[eugenio412]

News: I dropped my accounts to 15 and now I don't have any ban, seems like now for larger areas we will need some proxied threads.

I've seen that Pgoapi implemented proxy settings in the last pull request, I'm tring to get to the bottom of the question myself but it seems hard, very hard

[labaland]

i get 3 to 5 ptc acc banned every day now, anyone else? (18left atm) !
and just now I got ip banned , passed me almost a week without ban

[Mickey32111]

Getting 10-15 accounts banned whenever the service is running for any longer than an hour. Also a 1 hour IP ban too.

[CuRleGit]

I believe some of my initial accounts have been banned.
Running on fresh ones, now. Doesn't take long to manually create enough, with the help of macros.

[Mickey32111]

It's the IP bans that come with it that make it really difficult

Sent from my iPhone

On 16 Aug 2016, at 22:35, CuRleGit notifications@github.com wrote:

I believe some of my initial accounts have been banned.
Running on fresh ones, now. Doesn't take long to manually create enough, with the help of macros.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[shakes80]

You guys seem to be looking in the wrong place for the throttling setting. Its located in pgoapi.py

class PGoApiWorker(Thread):
THROTTLE_TIME = 10.0

Also, I am pretty sure AuthPtc all the users at once is a bad idea. It's probably one of the reasons banning is happening. Might want to look into spinning up accounts gradually. Calling the AuthPtc in batches of a few and gradually adding users will look less suspicious to Nia. The add_auth_accounts can be easily modified to do this instead of flushing the auth accounts.

Also, it looks a bit how accounts are being used. They seem to be cycling when the workers are called, I'm a bit more grey on this. I need a bit more time to study it. If I were Nia, I would look for this type of activity.

I've read over the code and it needs some tweaking, I am willing to help out but I'm a little strapped for time.

[labaland]

@shakes80, if my all my ptc accounts already has accepted tos (I do it when I create them) can I disable AuthPtc then? If yes how? :D

[EternisRequiem]

@CuRleGit Could you point me in direction of how to do said macros? Been giving myself a huge headache.

[bigblackqueh]

so.. rate-limited error is still widespread, anyone has any solutions? or is this map dead now

[dikkedeur]

what i wonder si, since the new update, does it all still work? or are accounts being banned now quicker? or at least at a faster rate then before?

[Letrab]

@Illydth : Update, apparently Niantic IS banning IP addresses.
News item from my country:
http://uk.businessinsider.com/pokemon-go-proximus-customers-lose-access-2016-8?r=US&IR=T

[dikkedeur]

@Letrab thats old news already.. we know they ban ip temp for 1 to 3 hours when you get that rate limited.
sollution is lower the request, dont scan 24/7.
but is na not blocking accounts now on a faster rate since in the update it says minor bot fixes.
thats the big question, ip bans are things we can overcome
account bans are a pain in my opinion.

[djv2]

so i don't know how to actually remove/delete accounts from the queue, but what i did to help with this issue is

1. if it is banned (code 3 multiple times), then i will print that username in a local .txt file so i can manually delete or examine for further investigation
2. if it is banned (code 3 multiple times), then i will skip the user in queue by not scanning by just returning out of the function

[ohnight8290]

SAME problem here
Received valid response but without any data. Possibly rate-limited?.......
Is there any solution?

[ghost]

Just wait 2h~ and everything will be fine again. It seens to be an temporary IP ban for me, because i created a new account and the problem persisted.

[RvV1979]

I have been running pogom with 40 accounts on a small area for over a month without any problem. This morning all accounts gave the 'Possibly rate limited' warning - no pokemon being scanned anywhere. I stopped the script and restarted periodically to see if the problem persists (changing the ip through vpn does not help).

It has now already been more than 9 hours and no improvement whatsoever. Anyone have similar experiences lately; did NA change their softban policy (i.e. longer wait times)?

[ghost]

I'm having the same problem since yesterday, @RvV1979, have been using for over a month on a small area too.
Any workarounds?

[eXeLe]

Probably because of captcha.
Check #413