Keeping file size reasonable and schema independance

[rom1504]

Main issue

Currently each version has its own schema.

What would be nice is having a common schema for every version.

Problem is : new things will be added and older versions didn't have as much data.

An other problem : some files, for examples blocks.json contain too many things which is annoying when creating extraction scripts (and possibly when reading it too)

One solution : keep things meaningful, split big files.

For example we might have a blockVariation.json instead of storing all the variations in blocks.json.

That would allow to have one common schema for all the versions. Some version just wouldn't have some of the data files.

Related/ideas

• database schema are usually split when putting it in https://en.wikipedia.org/wiki/Database_normalization#Normal_forms
• knowledge bases avoid that problem by putting data in its atomic form : triples

The knowledge base way is close to what we are doing (for example <1,name,stone>)
and it's even possible to convert all our data to a knowledge base forma (like n triples) by using json ld (http://json-ld.org/)

Also see http://wikiba.se/

[rom1504]

I think I might try to see if our json can be considered json-ld (by providing a context inside the files, or outside). It could make it possible/easy to put data in a knowledge base.

[rom1504]

11:48 <rom1504> where would it make sense to put data about minecraft, on a dedicated wikibase instance or on wikidata itself ? (data like what's on the infobox in there http://minecraft.gamepedia.com/Stone )
11:49 <rom1504> I think it probably doesn't belong on wikidata, but I'm not sure I really understand what data wikidata should contain
11:50 <rom1504> after all google knowledge graph do have that kind of data about minecraft https://www.google.com/search?q=minecraft+stone
12:22 <harej> rom1504: You can probably put it on Wikidata. If the properties you want already exist, that's a good sign; otherwise, propose the properties at https://www.wikidata.org/wiki/WD:Property_proposal and see what others say.
12:22 <harej> Wikidata does have an abundance of Pokémon-related info after all.
12:24 <harej> I don't work with fictional universe stuff so I don't know where the particular boundaries are, but I have found with my general experience with Wikidata that there is a lot you can do :D