schemas: Add for FCMaterials data

[Athanasius]

Odyssey Update 12 added a new FCMaterials Journal event and accompanying file. This lists any Fleet Carrier/Bartender orders for buying or selling Odyssey-specific materials (used to upgrade suits and weapons).

There is also data for this in the CAPI /market endpoint.

So, we want a new EDDN schema to allow for relaying data about this.

1. It should use the FCMaterials.json format as its basis.
2. But the README should also document the known CAPI format, and how to mutate it into the Journal/schema format.

Example FCMaterials.json:

{
  "timestamp":"2022-06-08T12:44:19Z",
  "event":"FCMaterials",
  "MarketID":3700710912,
  "CarrierName":"PSI RORSCHACH",
  "CarrierID":"K4X-33F",
  "Items":[
    {
      "id":128961533,
      "Name":"$encryptedmemorychip_name;",
      "Name_Localised":"Encrypted Memory Chip",
      "Price":500,
      "Stock":0,
      "Demand":5
    },
    {
       "id":128961537,
      "Name":"$memorychip_name;",
      "Name_Localised":"Memory Chip",
      "Price":600,
      "Stock":0,
      "Demand":5
    },
    {
      "id":128972290,
      "Name":"$campaignplans_name;",
      "Name_Localised":"Campaign Plans",
      "Price":600,
      "Stock":5,
      "Demand":0
    }
  ]
}

CAPI example (not from the same FC as above):

"onfootmicroresources": {
  "sales": [],
  "purchases":[
    {
      "name":"weaponinventory",
      "locName":"Weapon Inventory",
      "total":30,
      "outstanding":30,
      "price":90000
    },
    {
      "name":"chemicalinventory",
      "locName":"Chemical Inventory",
      "total":30,
      "outstanding":30,
      "price":90000
    },
    {
      "name":"digitaldesigns",
      "locName":"Digital Designs",
      "total":90,
      "outstanding":90,
      "price":120000
    },
    {
      "name":"operationalmanual",
      "locName":"Operational Manual",
      "total":60,
      "outstanding":60,
      "price":70000
    },
    {
      "name":"productionschedule",
      "locName":"Production Schedule",
      "total":60,
      "outstanding":60,
      "price":100000
    },
    {
      "name":"manufacturinginstructions",
      "locName":"Manufacturing Instructions",
      "total":30,
      "outstanding":30,
      "price":90000
    }
  ]
}```
So, it *looks* like the CAPI data 'fulfillment' is inverted, with `outstanding` instead of `Stock` or `Demand`, but this will need confirming.

Also whilst the Journal has the nice separate `CarrierName` and `CarrierID` I'm not sure that's available in the CAPI data.

[Athanasius]

We also need to be 100% sure that the CAPI data is there generally, and not only for the owner of the FC.

[Eahlstan]

We also need to be 100% sure that the CAPI data is there generally, and not only for the owner of the FC.

I'd say it is. My bubble commander doesn't have a carrier and got that data from 3 different carriers (I only tried those 3).

[Eahlstan]

As requested.

market.zip

[Athanasius]

I've had a different idea about Journal vs CAPI.

Rather than wanting to be 99.9% sure of the differences in format, and what data might be in one format and not the other...

... why not just two separate schemas?

Other than PII removals both can then just be "this envelope, plus what's in the file (Journal) or CAPI data".

So, fcmaterials_journal-v1.json and fcmaterials_capi-v1.json.

Especially for the CAPI I have low confidence that Frontier won't suddenly change something about it without pre-notifying us. Keeping it simple, and separate, means that Senders won't need to adjust and Listeners can be as nimble as they feel the need to be.

[Athanasius]

With not transforming either format to the other we will end up with e.g. EDMC doing

1. Player docks at an FC, CAPI queries made, fcmaterials_capi/1 message sent.
2. Player then disembarks and visits the bartender resulting in a possibly 'duplicate' fcmaterials_journal/1 message (but then no more duplicates unless the journal data has changed).
3. In EDMC's case if the player presses 'Update' after docking, and the CAPI-sourced data isn't changed there would also be no duplicate message.

[robbyxp1]

I agree with the above.

But we can have one schema, just have it have an object with the data in, named either "CAPI" or "Journal".

[Athanasius]

I agree with the above.

But we can have one schema, just have it have an object with the data in, named either "CAPI" or "Journal".

1. I'd like to hear the preference of the EDDN Listeners.
2. It'll complicate tying the schema down, if that seems the way to go, with having two different formats. Much easier to do that with separate schemas.

For EDMC I'm happy to support two different schemas, it goes through different code paths anyway.

[Athanasius]

After changing the in-development schema to allow for CAPI data as well a bit of a discussion ensued on Discord as to if we should have a single schema allowing both data sources, or a separate schema for each.

Whilst I was convinced enough to try a single schema, I personally still think separate schemas keep things simpler for both Senders and Listeners. I could easily see some Listeners switching on $schemaRef and expecting to be able to "just go" from that point, rather than needing to check things like "what does data-source say?" or "Is this Items an array or a dict object?".

@clonedArtie appears to be Team One Schema To Rule Them All
@spansh seems to prefer One Schema For Each Source

Any other comments from the likes of @robbyxp1 @AnthorNet @Tkael @themroc5 ?

You can in effect see examples of the data format differences in https://edgalaxydata.space/EDDN/dev/Unknown-2022-08-31.jsonl .

Array (Journal) versus dictionary (CAPI, and then it contains two keys, the value of which can either be an empty array, or a non-empty dict), different cased key names, different form of the (symbolic) name, Journal has Demand/Stock when CAPI only has one (due to that split into two keys).

Listeners are very much going to want to check the data source (be that key or schema name) and then use different code.

No, Senders won't be trying to transform one into the other, that runs into issues if there are changes, whereas each Listener can be much more nimble than waiting for "1. Sender dev codes changes, 2. Sender dev releases new version, 3. Sender users upgrade to new version", which might also call for Listener changes as well.

[robbyxp1]

So,

message": {"CarrierID": "X3F-N5Z", "CarrierName": "WARD'S OLOGIES", "Items":
[{"Demand": 0, "Name": "$compactlibrary_name;", "Price": 10000, "Stock": 0,
"id": 128962598}, {"Demand": 0, "Name": "$aerogel_name;", "Price": 500,
"Stock": 0, "id": 128961524}, {"Demand": 0, "Name": "$accidentlogs_name;",
"Price": 4000, "Stock": 0, "id": 128972282}], "MarketID": 3706433792,
"data-source": "Journal", "event": "FCMaterials", "horizons": true, "odyssey":
true, "timestamp": "2022-08-31T13:43:55Z"}}

Comments: Items is a direct copy of the .json, fine.

"message": {"CarrierID": "X3F-N5Z", "Items": {"purchases": [], "sales":
{"128961524": {"id": 128961524, "name": "aerogel", "price": 500, "stock": 0},
"128962598": {"id": 128962598, "name": "compactlibrary", "price": 10000,
"stock": 0}, "128972282": {"id": 128972282, "name": "accidentlogs", "price":
4000, "stock": 0}}}, "MarketID": 3706433792, "data-source": "CAPI", "event":
"FCMaterials", "timestamp": "2022-08-31T14:54:40Z"}}

Comments: names are different, not a problem, but where is odyssey and horizons flags? And lacking CarrierName but I guess this is CAPI.

But then later you had:

"message": {"CarrierID": "X3F-N5Z", "Items": {"purchases": [], "sales":
{"128961524": {"id": 128961524, "name": "aerogel", "price": 500, "stock": 0},
"128962598": {"id": 128962598, "name": "compactlibrary", "price": 10000,
"stock": 0}, "128972282": {"id": 128972282, "name": "accidentlogs", "price":
4000, "stock": 0}}}, "MarketID": 3706433792, "data-source": "CAPI", "event":
"FCMaterials", "horizons": false, "odyssey": true, "timestamp":
"2022-08-31T15:03:47Z"}}

with them back in.

I'm happy its one schema, and i'm happy there are not complicated name translations at the client end to do.

[Athanasius]

Yes, as I noted on Discord, I initially failed to actually set the horizons and odyssey flags on my CAPI-sourced messages, but later fixed that and at least one went through in that format.

The current fcmaterials/1 in #195 is the result of me experimenting yesterday with what was necessary and possible with a single schema for both data sources. It should absolutely not be taken as the canon of how the schema(s) will end up on the live service. I didn't update the README for yesterday's work.

I'm most interested in hearing from all the major EDDN Listeners now. If they've not spoken up by about this time tomorrow then I'll make the decision myself, which is currently leaning towards:

1. Two schemas, one for each data source.
2. Lock them down as much as current knowledge allows. Despite my worries about Frontier changing things without warning... I can't recall the last time they actually did so in a way that caused EDDN messages to be rejected by the Gateway. Thus I'd much rather lock things down so we have fewer bad-Sender messages make it to Listeners.
3. No augmentations where they're un-necessary (i.e. not leveraging a CarrierName into the CAPI data).
4. I do prefer keeping the Journal names for keys however. So CAPI will have id -> MarketID, and name -> CarrierID. Yes, the CAPI data has name for what is actually the ID of a Fleet Carrier.

[Athanasius]

That 24 hours notice is now up. I've started work to use a schema per data source for this. This includes updating some documentation around that for the future.

[Athanasius]

The PR is now updated for separate schemas, including the _capi README being in first draft form.

These schemas are now up on dev.eddn.edcd.io for testing.

[Athanasius]

One last round of end-to-end testing just now. Results are in https://edgalaxydata.space/EDDN/dev/Unknown-2022-09-02.jsonl from "gatewayTimestamp": "2022-09-02T15:44:17.357422Z" to "gatewayTimestamp": "2022-09-02T16:01:30.891159Z"`.

This covered:

1. No orders of either type.
2. Things being bought: single item, multiple items different counts, the buy orders having been fulfilled and either still there or now gone.
3. Things being sold: single item, multiple items different counts, having been bought but still listed, no longer listed.

My focus was on the CAPI side of things, but having to be at the bar tender to buy/sell meant I also triggered the Journal side for most of this.

All passed through without issues in the EDMC branch, and no rejections/errors on the dev.eddn.edcd.io side.

So, unless anyone can spot a problem by 12:00 UTC tomorrow, I'll merge this and move towards putting it up on the live service.

[Athanasius]

The schemas are now live.