node-list schema and CI

[benhylau]

^{This initial comment is collaborative and open to modification by all.}

Task Summary

🎟️ Re-ticketed from: #
📅 Due date: N/A
🎯 Success criteria: Define new schema and validate with CI for list of consumers.

Define https://github.com/tomeshnet/node-list schema based on consumer (who will parse it) needs and ensure CI catches non-conformant json files and prevent merge.

To Do

• Define schema
• Collect list of consumers (current only map on website, but perhaps a real-time status poll bot, chat bot, or compiled KML assets by CI?)
• Handle images
• Validate CI catches invalid jsons
• Cut new version 0.7 of schema
• Update json-schema spec version to 2019-09 (repo uses draft-04 from 2013), and make sure it still works with CI
  • See two migrations documents: Draft-04 to Draft-07 and Draft-07 to 2019-09
  • Is there value in this? (edit by @benhylau: no we should not do this, see comment)

[benhylau]

I would like some advice on this:

• Collect list of consumers (current only map on website, but perhaps a real-time status poll bot, chat bot, or compiled KML assets by CI?)

What do y'all think are consumers of the data?

[darkdrgn2k]

What do y'all think are consumers of the data?

In nycmesh all conversations with "do you have a node number" if not goto https://www.nycmesh.net/join

This information goes (automatically?) into a google sheets and that informs their website map. Each entry gets a node number that then is used to discuss.

I think this process needs to be defined before we translate it into json (or other source)

[makew0rld]

What do y'all think are consumers of the data?

@benhylau The list you gave (map, bots, KML) covers most things I think. Generation of static maps for promotion and distribution could be another consumer, as I assume the rendering of static maps is a totally separate process than the way the website map works.

[benhylau]

Hmm, would be cool if there is a library for:

• coordinates -> kml
• coordinates -> map PNG

Then we can travis ci run them and output assets to github releases on merge.

I am really busy in August and won't get to investigate these, if anyone feels they want to play with the asset generation side of things feel free to spin a new issue for that. Example github release script and the assets.

[makew0rld]

Yes, that would be great. Potentially there might be existing mapgen tools that work with KML, which could simplify the work. That would be a good thing to look in to.

[makew0rld]

• Validate CI catches invalid jsons

This is already happening in the node-list repo, isn't it?

[makew0rld]

Hmm, would be cool if there is a library for:

• coordinates -> kml
• coordinates -> map PNG

I'd be happy to look into this. I've made a separate issue for it, #67.

[makew0rld]

• Define schema

@benhylau Can you elaborate on what is needed for version 0.7 of the schema?

[darkdrgn2k]

I think its premature to try to permute the file while its in flux.

I think there are many datapoints we need to figure out how and were to store first.

[makew0rld]

Like what? Can we figure them out in this issue?

[darkdrgn2k]

i have no idea. there is a plethora of things that need to be figure out that spans multiple disciplines.

From monitoring, to on-boarding, to running services, to tacking the network.

[darkdrgn2k]

Collect list of consumers

what does this mean?

[makew0rld]

@darkdrgn2k I chimed in about it above. My understanding was the Ben was asking who would be using or "consuming" the JSON files, what software would be using that data as input. I spun this into #67, to address any software we'd need to write ourselves.

[makew0rld]

Does it make sense to work on #67 currently, or should I wait until the new schema has been defined so I don't duplicate work?

Edit: Answered in chat, the new CI schema needs to be defined first.

[makew0rld]

I have created a draft version of schema v0.7 in this PR. Reviews would be helpful.

[benhylau]

See two migrations documents: Draft-04 to Draft-07 and Draft-07 to 2019-09
Is there value in this?

I actually did a Draft-07 one before, but it turns out it's poorly supported across tools. Our CI validation tool, linked in README, does not support Draft-07 and I have noted this in README. There is little actual value to upgrade. What we have is Draft-07 compatible, but not the other way around.