EDDI Core: Body, system, & station data details from http://api.eddp.co are not reliable

[Tkael]

EDDI relies on data from http://api.eddp.co for system details, body details, faction details, & more.
http://api.eddp.co is built around an EDDN listener and mirrors the data to http://api.eddp.co for inclusion in these functions.

EDDN passes messages in real time but http://api.eddp.co stores Body/System data for later queries. Consequently, if incorrect data is passed through EDDN it can be stored in http://api.eddp.co & reported incorrectly to later users.

There have been recent issues with some applications passing incorrect / out-of-sync data to EDDN. http://api.eddp.co captures and reports the unfiltered and incorrect data.
See for example Pleiades Sector IR-W d1-55:
Population is 0 on http://api.eddp.co/systems/Pleiades%20Sector%20IR-W%20d1-55
Population is 900,000 on https://eddb.io/system/45103

Either

1. We need obtain & to revise the code jgm has used to populate data on https://api.eddp.co to filter out unreliable sources or
2. We need to switch to another endpoint that already filters for unreliable data (such as EDDB).

[Tkael]

Note: Faction state changes are handled differently in that they are relayed more or less in real time from the EDDN TCP uplink. Though this is bounced through http://api.eddp.co, faction state changes should continue to remain reliable. See https://github.com/EDSM-NET/EDDN/wiki for how this information is transmitted and EddpMonitor.cs for how EDDI receives the data.

[Tkael]

The data currently being pulled from EDDP for body details may also be redirected to pull from EDSM via the EDSM public API, as described here: https://www.edsm.net/en/api-system-v1
The data currently being pulled from EDDP for system details can also be redirected to pull from the EDSM API as described here: https://www.edsm.net/en/api-v1

[Tkael]

EDDP is also used in DataDefinitions/BlueprintMaterials.cs with http://api.eddp.co/_blueprintrequirements, in MaterialMonitor/MaterialMonitor.cs with http://api.eddp.co/_materiallocations and http://api.eddp.co/_materialuses.

[richardbuckle]

Moving to post-2.4

[ghost]

Comment from the "trusted EDDB user" section of the peanut gallery:

You actually have four slightly related issues:

1. There are applications or certain versions of applications that could send EDDN events with the wrong system name in them. The root cause is lag in the cAPI coupled with missing sanity checks in the application, and these have resulted in stations getting duplicated. Shinrarta Dezhra is a good example, it has one station, EDDI reports two, EDDB even had three.
   You can protect yourself against these by looking at the software/version part of the EDDN message and for example ignoring messages from ED Discovery < 8.2.7 and EDMC < 2.4.2.0 if they would create a new station.

2. Not all the data in EDDB comes from EDDN messages. At the moment, there is no automated solution for planetary settlements where a player cannot dock. All planetary settlements have been added manually. The same applies to the population, the system reserve level, and the body correlation (station X orbits body A, outpost Y is on planet B). The population will be part of the journal messages in 2.4, but there is no solution for the settlements.

3. There are no EDDN messages for these manual additions nor the deletions of duplicate stations, so the issues fixed in EDDB will only have an effect on EDDB itself and its nightly dumps. In order to profit from the updates on EDDB, you would have to regularly sync your data with the data from the EDDB dumps, and if the station with ID X is present in the last dump but not present in the current dump, you need to remove it from your data.

I deliberately used the term station ID because station and system names are not unique in game.

4. Systems are renamed by frontier with almost every major patch, and you need to track these changes also. The most obvious changes are in the Colonia area where the Eol Prou names are getting replaced with the names from the CEI, but there are changes in the bubble also. The only way to track these changes is by reading the patch notes, applying the changes to your data manually, and having some kind of translating filter in your input stream that translates new messages with old names into messages with the new names. The sync functions of ED Discovery and the "delay sending until docked" of EDMC can generate these.

[Tkael]

All the more reason to work with another source rather than redoing it all from scratch. Also created pull request #106 to add body reserve level to body objects reported by EDDI.

[Tkael]

Material / blueprint data could be referenced from https://github.com/msarilar/EDEngineer/tree/master/EDEngineer/Resources/Data potentially too.

[Tkael]

EDDP faction updates and static reference files on the server (like blueprint requirements) have now been transitioned to separate issues #177 & #178.

[Tkael]

Another potential endpoint we can use: http://elitebgs.kodeblox.com/api/eddb/docs

[Tkael]

We need to decide which endpoint is most advantageous for us to use before we can move forward with this.

[Tkael]

Which option should we pursue (click the one you like better)?

[Tkael]

The EDDB Unofficial API is more complete. Analysis attached.
EDDI details review.xlsx

[Tkael]

So neither the EDDB unofficial API nor the EDSM API is as complete as I'd like.

The EDSM API lacks

• A faction endpoint that allows searching by factions
• EDDB IDs
• Information on non-landable bodies
• landing pad size information

The EDDB API lacks

• A way to link station data to system data

[Tkael]

• Handle SystemAddress 64bit id value in FSDJump, Location, Docked, StartJump, NavBeaconScan, ApproachBody, and LeaveBody events, and in the transient data we send to the EDSM API via the EDSM Responder.
• Handle MarketID in various events and in the transient data we send to the EDSM API via the EDSM Responder.
• Update EDDN Responder to check system and market ids before sending data

Benefits of using SystemAddress and MarketID data:

1. These values are unique. The name may not be. For example, there are 3 different systems with a station named "Laplace Ring"... Balante, Kongga, & Wolf 529.
2. These can provide an extra sanity check for messages we are going to send to EDDN.
3. These can allow the lookup of remote station and system data in events that reference these values but do not otherwise identify remote stations or systems.

[Loophole62ndFF]

Would it be possible to combine data drawn from both EDDB and EDSM to obtain what is required? The EDDP data, at least in the region of space I spend most of my time in, is largely incorrect.