Add Detailed Gym Data

[maddhatter]

Description

Collection detailed information about which Pokemon are in a gym, along with the gym name and description. It's an opt-in feature (using the -gi switch), so no additional API hits unless you explicitly want them.

This pairs really well with #748 (already merged) to create a gym-only scanner, and would allow future development on a local gym leaderboard using the information collected (and sent out via webhook).

Motivation and Context

#605

It's nice to have the option to see detailed gym information about who's in which gyms, the gym's prestige level, the gym's name, etc.

How Has This Been Tested?

Backend tested on Windows 10 with SQLite and a MariaDB (running on a CentOS VM), in various scanning locations (Mine and @FlarumHostingCo's location, namely)
Frontend tested on Windows 7/10 on Firefox/Chrome, and on Android Chrome

Screenshots (if appropriate):

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.

[totoberg123]

Can you provide the right frontend code to show all users in the gym?

[Zerotask]

Can you post some screenshots?

[jnovack]

last_scanned is better English than updated_at. Preference.

[jzju]

What do you think about this database layout? It should make gathering statistics easier but may be too complicated/slow.

[jnovack]

If you are increasing the model, you should consider metric and security updates as well.

EVERYTHING should have a last_modified.

I would LIKE to see everything have a modified_by (or similar naming convention) for every row added of the PTC/Google login name of the scanner. For people with distributed scanners, a bad scanner can corrupt your DB, and by tagging each entry made, bad data can easily be carved out.

[maddhatter]

@Johanju I'm not sure how PokeGym.endTime would be determined - if the scanner were running slowly, you could potentially miss entire gym capture/re-captures, etc. I like the added stats/reporting that schema would allow though...

@jnovack I agree, but I think that maybe should be a separate PR?

Added front-end code as well, here's a screenshot: https://i.imgur.com/CRgQQI2.png

[jnovack]

@maddhatter Don't forget to add a basic webhook of sending the current data.

I suggest a gym hashed array, and trainers/mons array in JSON. level[0] would be the first defender, etc.

Suggestion;

{
  "id": string,
  "long": string,
  "lat": string,
  "team": id
  "name": string, /* soon */
  "url": string, /* soon */
  "level": [
    {
      "trainer": string,
      "pokemon": string,
      "pokemon_id": int,
      "cp": int
    }
  ]
}

[maddhatter]

@jnovack I don't have anything set up for webhooks to test, but I threw in a quick implementation. If I get some time later, I'll stand up something to receive the hooks and just dump to console.

[jnovack]

https://requestb.in/

Free, simple, easy.

[maddhatter]

Well, that's handy. Other than me copy/paste failing on a property name (fixed now), looks like it's working.

[jnovack]

Why is the Gym Name in gym_details, and not gym?

The long, lat, id and name should be in the same table, those are static and unchanging.

In fact, team_id should not even be in gym. It should be in gym_details since it is mutable. That's a breaking change, so I'd like some feedback.

[maddhatter]

My thinking was since gym detail scanning is optional (--gym-info / -gi), stuff that can only be discovered by doing a get_gym_details call would go into GymDetails. The gym info that's returned by the "normal" scanner's get_map_objects call was already going into Gym, so I left it there.

[jnovack]

• gym_name and gym_picture - I understand, but I would rather it be in gym, the column(s) can be empty if you aren't running -gi.
• team_id - That's a breaking change that should be discussed by the collaborators. I have no say.

Paging @invisiblek @chuyskywalker

[maddhatter]

Might as well squash GymDetails into Gym completely really - the only thing left on it would be the last_scanned column, and if Gym's getting NULLs for the other values anyway, could add gym.last_detailed_scan or something to it.

That would probably make it easier for non-SQL people to play with anyway, since there'd be no joins to access the data. I'll play around with updating it later on tonight (getting offline for now)

[jnovack]

You are probably right, good point. Perhaps last_inspected?

Or, for semtentics sake, the 900km glance should be last_skimmed, and the inspection should be last_scanned. But that's just me being punch-drunk and silly at this point.

[FlarumHostingCo]

I'm getting undefined rather then data

Tables for gymmembers and gymdetails are both empty too

[maddhatter]

@FlarumHostingCo Did you run it with -gi? It adds a lot of API calls, so it's opt-in.

The undefined's not good either way... I'll update that to be an empty string instead.

[FlarumHostingCo]

No missed that being required @maddhatter How can I add it to javascript or config.ini to default yes? I like to just use python runserver.py and go :)

[FlarumHostingCo]

@maddhatter turning on gyms in menu, map goes blank. Console traceback is

016-08-15 21:12:30,824 [ search_worker_6][ auth_ptc][ INFO] Request PTC Access Token...
2016-08-15 21:12:30,942 [ search_worker_2][ search][ ERROR] Exception in search_worker: (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '))' at line 1"). Username: xxxxxbxxx
Traceback (most recent call last):
File "/home/map/pogom/search.py", line 324, in search_worker_thread
parse_gyms(gym_details)
File "/home/map/pogom/models.py", line 525, in parse_gyms
DeleteQuery(GymMember).where(GymMember.gym << gym_details.keys()).execute()
File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 3274, in execute
return self.database.rows_affected(self._execute())
File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2656, in _execute
return self.database.execute_sql(sql, params, self.require_commit)
File "/usr/local/lib/python2.7/dist-packages/playhouse/shortcuts.py", line 203, in execute_sql
sql, params, require_commit)
File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 3492, in execute_sql
self.commit()
File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 3316, in exit
reraise(new_type, new_type(*exc_args), traceback)
File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 3485, in execute_sql
cursor.execute(sql, params or ())
File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 161, in execute
result = self._query(query)
File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 317, in _query
conn.query(q)
File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 837, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1021, in _read_query_result
result.read()
File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1304, in read
first_packet = self.connection._read_packet()
File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 983, in _read_packet
packet.check_error()
File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 395, in check_error
err.raise_mysql_exception(self._data)
File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 120, in raise_mysql_exception
_check_mysql_exception(errinfo)
File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 112, in _check_mysql_exception
raise errorclass(errno, errorvalue)
ProgrammingError: (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '))' at line 1")
2016-08-15 21:12:30,949 [ search_worker_2][ search][ INFO] Search step 3 beginning (queue size is 4)

[maddhatter]

@FlarumHostingCo What version of MariaDB are you using?

[FlarumHostingCo]

@maddhatter 10.1.16 On CentOS7.2

[FlarumHostingCo]

@maddhatter I installed MySQL5.7 on another server. Gyms don't load and console gets flooded with this..

gym_id) REFERENCESgym(gym_id))')... Retrying 2016-08-16 03:58:38,554 [ search_worker_0][ models][ WARNING] (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (map.gymdetails, CONSTRAINTgymdetails_ibfk_1FOREIGN KEY (gym_id) REFERENCESgym(gym_id))')... Retrying 2016-08-16 03:58:38,645 [ search_worker_0][ models][ WARNING] (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (map.gymdetails, CONSTRAINTgymdetails_ibfk_1FOREIGN KEY (gym_id) REFERENCESgym(gym_id))')... Retrying 2016-08-16 03:58:38,735 [ search_worker_0][ models][ WARNING] (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (map.gymdetails, CONSTRAINTgymdetails_ibfk_1FOREIGN KEY (gym_id) REFERENCESgym(gym_id))')... Retrying 2016-08-16 03:58:38,826 [ search_worker_0][ models][ WARNING] (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (map.gymdetails, CONSTRAINTgymdetails_ibfk_1FOREIGN KEY (gym_id) REFERENCESgym(gym_id))')... Retrying 2016-08-16 03:58:38,916 [ search_worker_0][ models][ WARNING] (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (map.gymdetails, CONSTRAINTgymdetails_ibfk_1FOREIGN KEY (gym_id) REFERENCESgym(gym_id`))')... Retrying

[gatomiaurex]

Hi, I have been trying to set a gym only scanning map with the developer branch after seeing this awesome feature, I already set everything and the map is running however it shows the previous version of the gyms, as in it only shows prestige, location and color (it does update it in real time), runserver seems to find the info tho.

This is the configuration I am testing

"C:\Python27\python.exe" runserver.py -gi -a ptc -u ******* -p ******** -l "21.49080150475335,-104.8851978778839" -H 0.0.0.0 -P 5001 -k *********

I already ran the command

pip install -r requirements.txt --upgrade

But it stays the same, I am using MySQL.

Am I doing something wrong?

Thanks in advance.

[Guile3home]

Too many SQL varibales warning is obtained after opening gym-info argument.

[maddhatter]

@gatomiaurex Sorry, I could have sworn I replied yesterday... try running npm install and Ctrl+F5 your browser. It looks like your frontend assets haven't been updated. Even if you don't use -gi, the prestige level on the gym popup is slightly different after this (XXX/YYY to show how close to prestiging it is).

@Guile3home That's something that really should be handled by the DB abstraction layer, but unfortunately isn't. What's happening is too many rows are trying to be inserted at once. I submitted #1091 to update the max # of rows being upserted.

EDIT: Updated with the PR I just opened.

[djv2]

is there a way to only display 1 team's gyms on map? like gym filter?

[gatomiaurex]

@maddhatter don't worry, I fixed it by deleting everything and starting from downloading the map again and it worked

thanks!

[maddhatter]

@djv2 No, but that would be kind of cool (esp with something like #782)- you should write something up and send a PR :)

[djv2]

@maddhatter unfortunately i couldn't get https://github.com/PokemonGoMap/PokemonGo-Map/pull/782 to work with your detailed gym info. did you get it to work (without any problems)?

[maddhatter]

I actually hadn't merged it (I keep looking at it and wanting to, but hadn't) until now. Looks like it's working without issues...

[djv2]

got https://github.com/PokemonGoMap/PokemonGo-Map/pull/782 to work, i just forgot to grunt build it lol

i ended up just putting an if statement for a filter of what to display. no UI change so no PR but it's simply just this in the map.js
if (item['team_id'] !== 2 || prestige1 > 2500 || gymLevel1 < 5) { return }

[gatomiaurex]

Is there a way to make it show the nickname of the people in the gyms?

[nunopitbull]

http://prntscr.com/ce0t24

Can you please apply that change already to your code? So it brings the file changed already? Or put it in the instructions!