Add Gym and Pokéstop scanning from DB and requested file dumps

[SiteEffect]

Implemented the ability to dump data of gyms and Pokéstops directly to a json file. Additionally a scan mechanic was introduced to scan forts (i. e. gyms and Pokéstops) based on the skip-empty scan method since a real spawnpoint scanning is not relevant when there is no timer. Gym scanner are now better to track by vision, adding a old and never merged PR feature which increases scanned location radius on map.

Description

The implementation was done with respect to the existing features of dumping spawnpoints as well as the skip empty cell scans. This resulted in the introduction of a new scan flag --fort-scan (-fs) for so called "gym scanners". Hereby, valid gyms and Pokéstops inside the step-limit are pulled from the DB and stored inside a file if --dump-gymsor --dump-pokestops is specified together with a file. If not file is added for scanning or dumping, the new scan sub-class HexSearchFort pulls all otherwise dumped forts from the DB directly and the generated hex area is thinned out when no fort is found inside each cell.
All these features are best used with the --no-pokemon (-np) flag to increase the cell size from standard Pokémon size of 70 m to a respectable gym scan size of 450 m.
To make the bigger cells when using -np finally visible I implemented PR #861 to todays DB structure. All credits to @wesleytsai.

Motivation and Context

This feature has been requested various times since gym only scanners are often used to expand the gym map while not increasing ressource costly Pokémon scan areas. Most lately the highly voted request of DevKat and in Issue #1487, #1321 and #1154

How Has This Been Tested?

On my personal map.

Types of changes

• New feature (non-breaking change which adds functionality)

Note: This feature can be extended with Lured Pokéstop support - finding a lure with a fort scanner locks scan-position for one worker and switches scan-delay to each 3 min. to retrieve the newly spawned Pokémon until the lure expires.

Checklist:

• My code follows the code style of this project.
• I have updated the documentation accordingly. (To-Do)

[bbdoc]

Hi Arengo,

Looks like a very interresting feature. Anyway, I'm a bit surprise this wan run with only -np set. I thought you can scan with a 450m range only if running -np -nk ?

Could you also give more details on how this would work ? Will this PR do some kind of initial scanning to discover gyms in the same way -speed is doing it and then automatically switch to skip-empty ? What if gyms have been scanned already ? How will workers be assigned to different hives ? In other word, will this work with a very big -st as workers will automatically calculate ideal path the closest gyms ? I fear as distances can be quite big in gym-only scanning my workers might start travelling 100km to get to a gym on the other side of the map.

Thanks

[SiteEffect]

@bbdoc -np deactivates the 70m search cell and activates the 450m search cell since quite some time now. (-nk is not needed). You can compare this on your phone, when travelling above speed limit of 38 km/h that Pokémon will not appear anymore, sightings remain empty but you still see Gyms and Pokéstops.
Regarding to this ability to travel above speed limit, people have already implemented Gym scanners for a while. I know metropolitan gym scanners where worker travel with 700 km/h without loss in functionality or banned accounts over time. Therefore, as viable technique to scan forts, --skip-empty was chosen - skipping all cells without any fort in it improves the general load of your workers. You use it on an existing database, where forts are already all known. This can be achieved during regular Pokémon scanning with speed or -np standard HexScan. When you collected your forts you can switch to -fs gyms.json/pokestops.json and either --dump-gyms/pokestops them once into a file for re-use (dumping was generally requested due to other reasons, too) and get your scan-informations of where the forts to scan are or you just leave it the scan method to grab them for each scan from the DB what is a slight increase of load. Compare this behaviour with the old -ss from file or directly from the DB.
The scan mechanism itself is just HexScan with skipped cells because we don't have to account timing of spawnpoints, since forts can change anytime, and we don't care about an optimal worker speed related path since we have technically not a real speedlimit. With this, all needs are already fulfilled while still having a efficient way to cover as much area as needed/possible. I wan't to point out that it would be great to address leap frogging of multiple worker in HexScan in the future - preventing a start from one hex corner at once but distributing the area initially somewhat equally between the available worker.

[bbdoc]

ok so my workers will simply teleport between the different hives that contain a gym ? How do you define the time between the scans for each gym ? Is there some sort of formula to know how many workers you need based on the number of hives that contain some gyms ? Btw, will status screen report on the number of gyms found and number of hives that need to get scanned ?

[SiteEffect]

It's fully configuration dependant. You simply decide with -sd how often scans happen.

[SubandiK]

Works like a charm thanks so much!

[michikrug]

you messed up the brackets in this one.
should be:

if geopy.distance.distance(
                    center, (pokestop['lat'], pokestop['lng'])).meters <= step_distance:

[SiteEffect]

Oh, you're right. Thank you, will fix it! Flake8 persuaded me to write shitty code :-P

[moropg]

It seems that this PR updates the db to a newer schema version. Beware of that while checking this out.

For the rest, this PR is quite gamechanging while scanning gyms. It has issues, e.g. with massive step distances (like st 50 or 100) it takes forever to fire up the threads (if it ever starts), and I still get some "empty" scan. This thing asks for a step up, something like clustering the dumped points and feeding them in a -ss like fashion. A massive improvements from a regular hex nonetheless!

[SiteEffect]

@leccarelepalle When it comes to huge -st, RocketMap struggles in general. This is not related to this PR.
The empty scans might come from complete soft-bans when you really travel with enormous speed like in a -st 100 with -w 100and -np so you travel +45 km in one single -sd 15. This is probably too much and asks for an additional speed-limitation implemented into the code (I do travel with around max. 600 km/h without any issues for forts, though). In your case, the exact numbers of your configurations would be interesting, when you get these full-empty responses.
Regarding clustering, I do not know if this would really improve that scanning mechanism at all since we do not have a specific time to scan a gym because we do not know when it changes and the areas are already filtered. What can happen is, that let's say 2 gyms are close to each other but fall into two different cells and no other gym would be neglected when only these two are covered by a "clustered" scan. It happens sometimes, but I do not think this is a game changer like clustering Pokémon spawns, where the additional time information is the main reason why to cluster.

[moropg]

@aRengo about empty scans, I'll tell you more once I did more testing, I already have a reasonable -sd though . I've yet to fire up a st100, because the hex schedulers gets stuck for ages. I think the problem is in the skip-empty algorithm which needs to calculate which cells are empty. I agree that clustering is totally optional, but a better approach for wide areas would be the aforementioned "-ss like" approach in which only non-empty points are used (and directly fed by the DB/Json). This way the whole "which cells are empty" calculations are avoided (worth mentioning that some of those routines are also pretty broken and unreliable for distances like a st100 gym scanner) and you may actually go big with gym scans. Using this approach the scanning points clustering (just spatial clustering) may be really nice to avoid to scan many times the same area if they are full of forts. Would also be a pretty trivial modification of the already existing tool.

[SiteEffect]

@leccarelepalle from a computational point of view, I totally agree. It will be much resource lighter in the start up. In this case, clustering makes sense instead multiply scan one area with multiple forts. Thanks for your feedback! Let me hear about your findings regarding the empty scans.

[BeriBGood]

I have been using this for 2 days+ now, working fine and no bug or exceptions detected for now.

[moropg]

@aRengo empty scans seems quite sporadic, and we don't have any on smaller step sizes. Using st < 20 is quite fast to fire up (seems faster pulling directly from DB instead of Json), even when dinamically changing location. I can use up to st 40ish with a reasonable startup time, anything above seems to get stuck forever...Can anything be done about it? I would love to scale up to st 100 hexes

[bbdoc]

Am I the only one who cannot get this to work ? I'm getting Schedule creation had an Exception: UnboundLocalError("local variable 'fort_locations' referenced before assignment",) on startup. I started with a fresh clone of PGM and only included this PR to make sure it was not linked to a dependency with something else, but still the same error. Did anyone else tested this PR using beehives ? Thanks

[SiteEffect]

@bbdoc I guess beehives probably produces the error. While this should be addressed by me to support all viable scanning methods, it is not recommended to use beehives when you can cover the same area without -bh in this PR.

[moropg]

The problem is, @aRengo , I can't really exceed st 40ish or the threads never start, so I too could make some use of the -bh flag

[SiteEffect]

Close/Open to restart Travis.

[bbdoc]

Any chance to get an update working with -bh ? ;-)

[bbdoc]

I'm giving this PR a try without -bh at the moment ;-)

Anyway, getting a lot of those error messages :

Exception message: UnboundLocalError("local variable 'abandon_loc' referenced before assignment",)
Traceback (most recent call last):
  File "/usr/src/app/pogom/search.py", line 987, in search_worker_thread
    scan_date, account, account_sets)
  File "/usr/src/app/pogom/models.py", line 1967, in parse_map
    if abandon_loc:
UnboundLocalError: local variable 'abandon_loc' referenced before assignment

No big deal as probably linked to some accounts being banned, but should not throw this error...

[michikrug]

As I researched on those errors, it seems that this is a problem with old / left-over code that should not be there anymore. I removed those parts (line 1962-1978) and it is working fine without errors.

[bbdoc]

@michikrug So you removed lines that were added by this PR on models.py ? Would prefer to let @aRengo check this will not result in any side effect.

[EDIT] Removed those lines, but now getting another error : 2017-05-05 12:24:36,817 [ search-overseer][ search][ ERROR] Schedule creation had an Exception: TypeError("unsupported operand type(s) for +: 'dict' and 'list'",).

[michikrug]

@bbdoc since I have merged a lot of PRs I can not really tell, what is also changed in my code in comparison to this current PR
But your error seems strange and more likely a mistake on your side?!

[bbdoc]

I didn't merge any other PR. Fresh clone of dev + merged only this PR + deleted those lines from models.py. I'll wait for @aRengo to remove those lines and push another commit if he confirms they are not needed and will try again.

[SubandiK]

I have the same problem as bbdoc after getting a more recent version of this, with the original it worked flawlessly, didn't change anything else.

[SubandiK]

Actually, I just "solved" it for me by inserting
abandon_loc = False at #1953
because the error was basically complaining that it wasn't set. Seems to run again, dunno about long-term consequences though.

[SiteEffect]

Will probably rebase that soon and change a lot. Let's use the deprecated -ss #2010 or -speed for this. Closing it for now.

[rpotpogo]

@aRengo - Any chance of this PR coming back? I need this for my gym only scanner.

[michikrug]

@rpotpogo I have it still maintained in my new-api and raids branch. But I have to say, there is also a lot of customized code and content included.

[SiteEffect]

Yes, chance for a re-write with -ss for raids and --skip-empty for forts is there.

[SkOODaT]

I know this is closed but I need to say THANK YOU for this, I adapted it to work with my map and I have the ability to lure stops on my map, so now i have a dedicated lure placement scanner .....my community thanks you :)

[ghost]

@SkOODaT How did you manage that?