Clean up preferences handling and make them more consistent

[matthijskooijman]

Currently, there are a lot of places where configuration values can come from. There is preferences.txt, boards.txt, platform.txt, programmers.txt and some runtime generated values. Each of these somehow contribute to the values used for building, uploading, showing the interface, etc., but I believe this process is not entirely clear. Some values can be set in one place, some can be overridden in other places (but not all), etc.

It would be good to:

• Have better documentation about the available options
• Make the way options are used more consistent
• Try to keep backward compatibility as much as possible (perhaps by adding an explicit version number to all these files?)

As part of this, overriding options should probably be more explicit, and less accidental or ad-hoc as it is now. Explicitely specifying overrides (using e.g. an override. prefix) could help here, and/or making the source of a preference (or perhaps more the intended use of it) using prefixes like board., build., platform., pref. etc. I'm not sure how this should work exactly yet, but some thought should go into this in any case.

@lmihalkovic voiced similar ideas here: #4102 (comment)

Another related issue that should probably be fixed as part of this, is allowing custom compiler flags, as discussed in #421.

[lmihalkovic]

I would offer that looking at this in the context of #4421 will give you some useful extra requirements. this is not to say that #4421 has to be done, but simply that looking at this with the thinking 'lets make sure that its implementation does not make doing something the like of #4421 a bigger effort' then some of what looks now like possible design will be eliminated.