[Feature Request] CREST Pilot Tracking #84

Closed
vivace-io opened this Issue Jan 19, 2016 · 16 comments

Projects

None yet

6 participants

@vivace-io

This falls in line with #83 which is asking for SSO. This could be extended further to use the new CREST endpoint, detailed in this dev blog. The advantage of this would be that the users do not need to keep the IGB open to track their location, and the cache time is a short 10 seconds. Thoughts? I'm hoping that once I know my way around I can help with this :)

@exodus4d
Owner

Yeah it is on my "TODO" list... ;)

@exodus4d
Owner

So this is currently in work.

Done so far:

  • Added Login process over CCP´s SSO.
  • Added oAuth2.0 functions for all relevant basic actions (getAccessToken(), refreshAccessToken(), walkEndpoint(), showEndpoints(),...)
  • Added refresh function that automatically refreshes" online" characters and verify their position/system and some other data.

There are some design decisions to made now. Im not 100% sure how what is the best way to do some things. The current (online) state:

  • Each user can have infinite API Keys,
  • Each API Key can handle up to 3 Characters
  • Each "character" owned by a "user" can access all the privat maps from that user. This means, "private" maps are bound to a user, and not to a character. The advantage of this is, that all "alts" can edit/see all of your maps.

With the CREST API, some things will change:

  • It is not possible to "grap" all characters from a user (person). I can just access the single character who logged on. Therefore a person will not be able to "see" all private maps of all his characters!
  • This leads us to the fact, that "privat" maps can not be "bind" to a "user" (as present), but to a "character".

CCPs SSO does not provide a way to log in multiple characters at once. The only way to "ship around" that, is to still create "users" on Pathfinder and let them manually log in multiple characters, which is not very user friendly.

@exodus4d exodus4d self-assigned this Jan 27, 2016
@exodus4d exodus4d added this to the v 1.0.0RC2 milestone Jan 27, 2016
@Zumochi
Zumochi commented Jan 27, 2016

Perhaps it would be possible to register a user, and then link characters to said user by logging in through SSO.

Users who don't want an account can stick to just using SSO.

@vivace-io

Does CREST destroy the old token or disable the token from being refreshed if you attempted to authorize a second character on the same account? If it does not, you could do basically what Zumochi said and have SSO for authentication and then a second "layer" allowing them to log in the individual characters after which they will not have to do it again.

EDIT: Also, can you push this work somewhere so I can take a loot at it?

@exodus4d
Owner

No CCP does not "destroy" any valid access_key. The only thing I have to take care of, is to "refresh" any access_key when the 20min expire time expires.
So in theory it is possible to log in a user, and then refresh the "access_key" by its "refresh_token" until your server or CCPs API shuts down for more than 20min (which can take month).

The "second layer" is probably the easiest solution for now. If there is no "user" created and associated with a character, Ill create a new user. If there is already a user logged in, Ill connect the new character to that user.... There is still one problem: If the CREST API is offline, no one can access or edit the maps. Ill not provide multiple ways of logging in (with API, without API) that is for sure :)

Ill push my work to the public develop branch if there is a stable state available which can take some weeks.

@Zumochi
Zumochi commented Jan 29, 2016

There'll be more issues when the CREST API is offline, so that'd be an entirely understandable time for the mapping tool to be unavailable as well. In that case, the user can always fall back to using IGB until that is removed as well.

Maybe a feature branch separate from develop can be created btw?

@exodus4d exodus4d modified the milestone: v 1.0.0RC3, v 1.0.0RC2 Feb 6, 2016
@exodus4d exodus4d added a commit that referenced this issue Feb 6, 2016
@exodus4d closed #90 (IGB map not working) jsPlumb v2.0.5 downgraded to v1.7.6
#84 renamed "landingpage" view to "login" view
#84 added some basic CREST API data mappers
#84 added "isNPC" flag t Corporation table (WIP)
#84  added CREST API settings to environment.ini
27c0698
This was referenced Feb 6, 2016
@exodus4d
Owner
exodus4d commented Feb 6, 2016

New feature branch is now publicly available "Branch #84". It is still a very early WIP state and can not be used in "production". But If you are intrested in checking the code or implementation. feel free... :)

Some required information for testing:

  1. Register your App by CCP to get API access ( Client_ID and Secret_key) which is required to use the (not private) CREST API (UR: https://developers.eveonline.com/applications) (check screenshot)
  2. Add the required CREST "Scopes" you want to have access (check screenshot)
  3. Add the callback URL (check screenshot)

dev_setup
4. Add your Client_ID and Secret_key data to environment.ini
5. Thats it!

How it works

  • All the "magic" happens in here: ccpssocontroller.php
  • There is a small "SSO login" link next to the login form login.html L148
  • When clicking the link you will get redirected to CCP s Auth-URL (HTTP 302) (see official docu.)
  • Login with your Acc (the required Scopes set up for your App can be seen):
    dev_login
  • After login, your App should get a a valid access_token and refresh_token whith will be used from now on for any API request.
  • The access_token is valid for 20min (CCP default). I store them in session (remember -> session data is saved to DB table session) with a ttl of 1200s (20min)
  • All the API caching is done by the framework itself (by there Cache-Control response Header). But i found a bug in F3 which is not yet fixed "Issue 908"
    All CREST API paths are "dynamic" and are fetched automatically by recursive API calls "walkEndpoint(); L309" (which corresponds to the official documentation). As an example of that, check out the getCharacterData(); function which "walks" down the CREST API root path, until the requested "leaf" is reached and returns the data "L390"
  • All the JSON date coming from the CREST API is mapped by a OOP implementation of _PHP_s "RecursiveArrayIterator()" class (checkout: mapper classes )
  • Some "static" tables are automaticalls filled/updated (characte, corporation, alliance)...

With the current state we can access any CREST API data after successful login by a very generic CREST controller. All the caching is done automatically by the framework and we don´t have to care about it).

Next steps

  • Code a nice intuitive UI for SSO login
  • "Bringing everything together" :) Make sure all the CREST data is "in sync" with the DB, update, add, remove location data for logged in characters.....
  • This will massively reduce the AJAX response timings which are done by each client to update the map data for each client.
  • In addition to that, i have added a second DB-Server which is running a database replication of MariaDB for "read* queries. This is of course not required... and was just a test with a huge success. -> The DB response time could be reduced by ~ 150% !

Feel free to comment your ideas or any suggestions for improvement. Maybe we could arrange a Teamspeak talk or something like that...


Official CREST Docu: http://eveonline-third-party-documentation.readthedocs.org/en/latest/sso/intro/

@VivianMeally

Can it be allready used? Or is it only for testing? Dont see this SSO Login login at https://www.pathfinder-w.space/

@Tupsi
Contributor
Tupsi commented Mar 11, 2016

Trying out branch #84 atm and got the whole auth process working, but instead of a "real login" I only get a bunch of text back after authorization, in the form of

array(10) { ["created"]=> string(19) "2016-03-11 10:45:18" ["updated"]=> string(19) "2016-03-11 10:45:18" ["active"]=> bool(true) ["name"]=> string(6) "McTest" ["corporationId"]=> array(9) { ["created"]=> string(19) "2016-03-10 18:44:28" ["updated"]=> string(19) "2016-03-10 18:44:28" ["active"]=> bool(true) ["name"]=> string(25) "Dumbo Corp" ["shared"]=> bool(false) ["isNPC"]=> bool(false) ["corporationCharacters"]=> NULL ["mapCorpo... and so on.

but not a working pathfinder behind it. Can only hit reload which gets me back to the login screen.

Is this an intended behavior currently, because its just basic sso testing, or should this already be going further into the app?

@exodus4d exodus4d added a commit that referenced this issue Mar 12, 2016
@exodus4d - #84 CREST Login (WIP)
- New CREST controller
- Database restructuring
- improved type-casting for some controller functions
- New login process
- Fixed some bugs during the setup process (/setup root)
- Added CREST request caching by response headers
7e94ec4
@exodus4d exodus4d added a commit that referenced this issue Apr 24, 2016
@exodus4d closed #131 new "route search" algorithm, added current map systems t…
…o live search, added refresh/update functionality for each found route, added bulk route refresh function, added "meta map" route search (search on multiple maps), added route "filters" (restrict search on "stargates", "wormholes", "jumpbridges"), added route "filter" for wormholes (reduced/critical wormholes)

closed #89 fixed "loop connections" on same system
#84 added error messages for "invalid" CREST "Client ID"
added "bootboxjs" (customized styled checkboxes/radio buttons) CSS only
"Font Awesome" version upgrade 4.4.0 -> 4.61
"Bootbox.js" version upgrade 4.3.0 -> 4.4.0
fixed "system dialog" (added responsive layout)
ac1a746
@exodus4d exodus4d added a commit that referenced this issue May 2, 2016
@exodus4d - #84, #138 improved "character selection" on login page (expired coo…
…kies are deleted, character panel layout improvements)

- added new "Server info panel" to the login page
- added new cronjob to delete expired cookie authentication data
8900276
@Tupsi
Contributor
Tupsi commented May 13, 2016

please see my issue #147 I just opened. Seems you need to escape the names you get from ccp a bit more.

@exodus4d
Owner

#84 branch is deprecated and will be removed soon. Developer branch is
up2date and a working deployment of it can be found at
https://www.dev.pathfinder-w.space .this is a new Test Server for the
upcoming CREST Release.
I'll check the escaping too. Thanks

Ugurtan notifications@github.com schrieb am Fr., 13. Mai 2016, 15:58:

please see my issue #147
#147 I just opened. Seems
you need to escape the names you get from ccp a bit more.


You are receiving this because you were assigned.
Reply to this email directly or view it on GitHub
#84 (comment)

@Tupsi
Contributor
Tupsi commented May 14, 2016

thanks, and you can ignore my old remark to #84 as I am already using your dev branch.

@Tupsi
Contributor
Tupsi commented May 14, 2016

the nginx error log is filling up with lots of messages like this one and the result is, that the user creating these is hit with a server error message in the browser and needs to reload the pathfinder website.

2016/05/14 15:30:59 [error] 11392#11392: *344797 FastCGI sent in stderr: "PHP message: method: 'GET', url: 'https://crest-tq.eveonline.com/characters/[characterID]/location/'" while reading response header from upstream, client: xxx.xxx.xxx.xxx, server: myserver.net, request: "POST /api/map/updateUserData HTTP/2.0", upstream: "fastcgi://127.0.0.1:9000", host: "myserver.net", referrer: "https://myserver.net/map"

@Tupsi
Contributor
Tupsi commented May 15, 2016

here is the one leading to a 500 server error with an "emergency shutdown" from pathfinder.

2016/05/15 19:20:52 [error] 3459#3459: *804671 FastCGI sent in stderr: "PHP message: method: 'GET', url: 'https://crest-tq.eveonline.com/characters/366150322/location/'
PHP message: [app/main/lib/web.php:128] Base->error(500,'method: 'GET', url: 'https://crest-tq.eveonline.com/characters/366150322/location/'')
PHP message: [app/main/controller/ccp/sso.php:516] Lib\Web->request('https://crest-tq.eveonline.com/characters/366150322/location/',array('timeout'=>3,'method'=>'GET','user_agent'=>'PATHFINDER - v1.0.0RC3 | https://github.com/exodus4d (pathfinder.tdsin.net)','header'=>array('Authorization: Bearer 63Wcfxxxxxxxg97eUth9tfbsqYVtwia_KKmRnFHkV2-c-dI0xxxxxx3SpPi-75yP3SxxxxxxxVgJUiqMYw2','Host: login.eveonline.com','Host: crest-tq.eveonline.com')),array('suppressTimeoutErrors'=>true))
PHP message: [app/main/controller/ccp/sso.php:555] Controller\Ccp\Sso->getEndpoint('63Wcfp0Qxxxxxxia_KKmRnFHkV2-c-dI0t0qIyyyyySpPi-75yP3SSmP5ucswGVVgJUiqMYw2','https://crest-tq.eveonline.com/characters/366150322/location/',array('suppressTimeoutErrors'=>true))
PHP message: [app/main/controller/ccp/sso.php:556] Controller\Ccp\Sso->walkEndpoint('63Wcfp0xxxxxxUth9tfbsqYVtwia_KKmRnFHkV2-c-dI0t0qxxxxxSpPi-xxxxxxxxxxUiqMYw2',array('standings'=>array('href'=>'https://crest-tq.eveonline.com/standings/366150322/'),'bloodLine'=>array('href'=>'https://crest-tq.eveonline.com/bloodlines/1/','id'=>1,'id_str'=>'1'),'gender_str'=>'1','waypoints'=>array('href'=>'https://crest-tq.eveonline.com/characters/366150322/navigation/waypoints/'),'private'=>array('href'=>'https://crest-tq.eveonline.com/characters/366150322/private/'),'channels'=>array('href'=>'https://crest-tq.eveonline.com/characters/366150322/chat/channels/'),'href'=>'https://crest-tq.eveonline.com/characters/366150322/','accounts'=>array('href'=>'https://crest-tq.eveonline.com/accounts/366150322/'),'portrait'=>array('32x32'=>array('href'=>'http://imageserver.eveonline.com/Character/366150322_32.jpg'),'64x64'=>array('href'=>'http://imageserver.eveonline.com/Char

@Dessix
Contributor
Dessix commented May 21, 2016

This seems to be what I'm getting, as mentioned in #156.

This was referenced May 21, 2016
@exodus4d exodus4d added a commit that referenced this issue Jun 3, 2016
@exodus4d v1.0.0 (#183)
* #84 test data dump from CREST login

* updated "credits" dialog (Google+ link)
fixed login form layout

* updated Cortex Data-Mapper

* - #84 CREST Login (WIP)
- New CREST controller
- Database restructuring
- improved type-casting for some controller functions
- New login process
- Fixed some bugs during the setup process (/setup root)
- Added CREST request caching by response headers

* pathfinder-84 [Feature Request] CREST Pilot Tracking, many smaller Bugfixes

* pathfinder-84 [Feature Request] added develop JS files

* closed #121 fixed wormhole signature type caching

* closed #120 removed map-loading animation for larger maps (same behaviour as IGB)

* closed #119 fixed wormhole signature id count

* closed #114 Added check for already existing system when adding a new one. (fixed PDO 'duplicate entry' error)

* closed #112 fixed DataTables error for missing "status" data (signature table)

* closed #111 fixed convertDataToUTC(); client side date transformation

* closed #109 fixed system TrueSec rounding

* closed #103 fixed system updated timestamp in getData()

* fixed CSS class for secStatus in Routes module

* closed #121 fixed wormhole signature type caching

* changed dateTime format from German to US format
fixed some minor bugs in signatureTable module

* closed #81 fixed "signature type" overwriting by "signature reader" update

* closed #106 added new signature_types form C5/6 wormholes (gas/ore)

* closed #129 fixed parameter hinting

* closed #131 new "route search" algorithm, added current map systems to live search, added refresh/update functionality for each found route, added bulk route refresh function, added "meta map" route search (search on multiple maps), added route "filters" (restrict search on "stargates", "wormholes", "jumpbridges"), added route "filter" for wormholes (reduced/critical wormholes)
closed #89 fixed "loop connections" on same system
#84 added error messages for "invalid" CREST "Client ID"
added "bootboxjs" (customized styled checkboxes/radio buttons) CSS only
"Font Awesome" version upgrade 4.4.0 -> 4.61
"Bootbox.js" version upgrade 4.3.0 -> 4.4.0
fixed "system dialog" (added responsive layout)

* closed #134  fixed db column type DT_INT (8 bytes) to DT_BIGINT

* closed #138 added new cookie based login

* closed #137 fixed javascript errors on trying to establish an "invalid" connection

* - #84, #138 improved "character selection" on login page (expired cookies are deleted, character panel layout improvements)
- added new "Server info panel" to the login page
- added new cronjob to delete expired cookie authentication data

* #138 enables character switching between characters which have same user

* - PHP Framework upgrade 3.5.0 -> 3.5.1 (fixes some issues with CREST cURL caching, and SESSION management)
- #138 added "cookie logout" to "logout" menu entry

* - updated "feature page" with new feature descriptions and label
- added some new images to the "feature gallery"
- removed "beta" status from "magnetizing" feature on map menu
- hide "server status" panel on "mobile" breakpoint

* - #138 clear character authentication data on sold characters

* closed #142 added custom "onsuspect()" session handler

* #142 do not log suspect if no file is defined in pathfinder.ini

* #142 added NullSec Data/Relic sites to C1/2/3 wormholes as signature option

* #144 fixed "Character not found" warning

* #144 fixed "Character not found" warning

* closed #144 fixed broken routes panel in IGB

* updated README.md for upcoming release

* #147 response header validation

* #149 changed comment for 'BASE' framework var

* fixed map  import

* - added minimal SDE dump (EVE Online: Citadel)
- #147 improved CREST API error logging (WIP)
- improved SSO controller (removed access_token from public endpoints)

* closed #154 added alliance maps to CREST API

* - updated Gulp build dependencies
- increased CREST timeout from 3s -> 4s
- added "Accept" Headers for some CREST endpoints

* cloased #147

* - closed #153 added character verification check for getAll(); Signatures Ajax endpoint

* - updated README.md (added Slack developer chat information)

* Bugfix frig holes (#159)

* added missing frigate wormholes and fixed Q003 destination in shattered wormholes

* changed C7 to 0.0 for Q003

* - fixed broken "graph" data for system

* added a  "failover" system  for bad crest requests (HTTP status 5xx,.. )

* Red Gaint => Red Giant (#161)

* closed #163 added CREST endpoint support for "waypoints"

* fixed typo

* closed #160 fixed tooltip container

* - added new features to login page

* closes #154 added alliance map support

* fixed XML path for cronjobs

* fixed a bug with inactive "private" maps

* closes #175 added alternative environment configuration

* - v1.0.0  build
ecd505a
@exodus4d
Owner
exodus4d commented Jun 3, 2016

Issue resolved in master branch

@exodus4d exodus4d closed this Jun 3, 2016
@Tupsi Tupsi referenced this issue in Tupsi/pathfinder Jun 6, 2016
Merged

v1.0.0 (#183) #6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment