Add general override functionality. Closes #3001. [WIP] #8753
Conversation
|
Some rethinking. Override file is now an option so it can be set from the command line allowing you to enable it to an existing build dir or switch between override files. Also added a new option type that point to files and validates that they exist. |
|
It would be extremely nice to be able to override dependency lookups too. In the style of autotools having |
|
That can be added too. It's mostly a question of syntax. |
|
Added functionality to override an option for a single subproject rather than all of them. |
|
This pull request introduces 2 alerts when merging 9838a03 into 1730778 - view on LGTM.com new alerts:
|
|
@jpakkane I had thought about dependency overrides as well. I think that probably should be in the machine file, since they're a per-machine option, my thought had bee something like: [dependencies.zlib]
include_directories = ['/usr/local/include']
c_link_args = ['-L/usr/local/lib', '-lz']
c_args = []
; Additional language args could be placed here if you had both a C and C++ ABI |
|
Also, while we're talking about it, I'd really like to have a single "machine" file, instead of separate cross and native files, an imaginary syntax could be: [build.machine]
system = 'linux'
[build.built-in options]
c_args = ['-foo']
[host.machine]
system = 'windows'
[host.dependencies.zlib]
...Obviously my sort-of-toml-in-ini isn't a very nice format, but we don't have a ton of options without external dependencies unless you want to use json (I don't advice json for humans to write by hand), or write something by hand. With our already available layering then it's easy to compose them together. Also, I have a branch that makes *_std a per-subproject option, I didn't quite get it finished, but it's probably useful since currently the standard is part of the ninja compile rule template. |
Actually, one of the outcomes of this PR is that we would no longer need per-subproject options. Instead they could be written in the override file. The proliferation of per-subproject options is not a nice thing and makes the number of options explode and complicate things. |
|
This pull request introduces 1 alert when merging f8529b3 into 1730778 - view on LGTM.com new alerts:
|
|
And, having two files sucks. Let's not do that. Let's have one file. just one, with one syntax, one argument, and not have a proliferation of files with slightly different formats. |
|
This is something I have thought about for a lot and unfortunately there does not seem to be a simple solution. First of all if you put this and cross and native files in a single file then it must be JSON or something else that has multiple hierarchical nested levels. A single flat ini-style thing becomes unwieldy if it has tens of entries. Writing JSON by hand is doable, but not that pleasant. Further, there are cases where you'd want to switch between different override sets but keep all other info the same. As an example you'd typically want to use the setup described in the doc where the top project is debug but all others are debugoptimized. Then you might want to switch to a different set with additional debugs or warnings. Or build some individual projects as shared rather than static for build speedups. If everything is in the same file, you need to copy all the other definitions as well. If you specify multiple conf files (like with cross files) then specifying If someone comes up with a nicer UI for this, then we could and should definitely do that. |
|
Realistically we have three choices:
All of them suck in some way. Ini stinks because it's flat, underdefined, and there's a ton of variants all called "ini". JSON is really annoying to write by hand, since it has obnoxious rules about commas. It was designed as a human inspectable language. It's great for machine -> machine situations. I've done a ton of work with JSON, I really don't recomend it for human use. There's a python builtin. YAML is really big and complicated, it has some odd whitespace interaction between tabs and spaces. It is, on the other hand, very flexible and thanks to github and gitlab quite popular and thus well known. There's several good implementaitons for python. It's also a strict superset of JSON. TOML is the current darling of the world, it's nice in that it's basically a speced ini format with heiarchy. It's got some odd syntax, it can be a little verbose, but it's generally not bad to write. There's one good implementaiton for python. The thing they all have in common is that they are structured. You can represent exactly the same data in all three formats: {
"a": 6,
"b": [
1,
2
],
"c": {
"x": "foo"
}
}a: 6
b:
- 1
- 2
c:
x: 'foo'a = 6
b = [1, 2]
[c]
x = "foo"And, pyyaml, json, and toml all will give you exactly the same data structure. So, here's a crazy idea. We could use all of them. JSON always, and one or both of the other two if you have the necessary dependencies. That gives us a format we can always accept, even if it's very annoying to write by hand and one or two formats that are actually designed to be written by humans that we can accept in most cases (really other than boostrapping). |
|
Toml is flat too. Nesting is defined by encoding the full hierarchy, dot-separated, into each section name. This is not hierarchical, this is flat but with some easy-for-humans-to-mess-up convention for inferring hierarchies. (You can arbitrarily decide the same rule for your project's ini files, leading to an identical experience and no better or worse wieldiness.) It quickly becomes unwieldy. yaml solves that problem, but does add different problems. I'm not aware of a single truly good file format. At least configparser is well defined as being configparser, and is in the stdlib! |
Whether it is unwieldy, it is speced that a The reality is that we need a hierarchical format if we're going to have "one file to rule them all". I'd gladly pick an imperfect format (since, as you say, there is no perfect config format) to have one file with one format than have two files with different formats. |
|
Currently machine files are ConfigParser but with values interpreted for basic types such as strings, lists, additions. Note that the "ini" format leaves values undefined, so our machine files are still valid ini, just with extra formatting. I worked a long time ago on bringing the same syntax to wrap files: xclaesse@ab1caf1. I think it's important to have the same syntax to all files formats exposed to users. I personally would like to keep using that format if possible because that's what our users know already. With a bit of structured section names, it shouldn't be that bad, we do that already for options in machine files. If we start using json or something else maybe we should switch wrap files too for consistency. |
|
I coded up a . separated syntax using our config parser, it's not bad. I need to write a few real tests and I'll push it for review. |
|
fully backwards compatible, but allows . separate section headers like toml. |
|
For reference, here is a copypaste of the syntax: [host.machine]
system = 'linux'
endian = 'little'
cpu_family = 'aarch64'
cpu = 'aarch64'
[host.built-in options]
foo = 'bar'
[host.built-in options.zlib]
default_library = 'static'I'm not a fan of the One major question here is what should be done if override and cross files are combined? I'd be inclined to say that that should be prohibited. If you specify an override file then:
The latter is a bit controversial, however I have found that when you have a project with something like 5 subprojects then the output of As an idea that came to mind: if the user does not specify an override file then we could write an empty one to the build dir and use that. It could have some comments at the top to tell people how to use it. |
|
I would prefer to have a single file that replaces the separate files. Combing cross/native files with the new file should be prohibited. I wonder if we could do something like: Where @ means "under the previous section"?, Ie that would be equvalent to [host.machine] |
|
I personally don't see the problem in having |
[host.built-in options.zlib]
default_library = 'static'That section name is starting to be ridiculous, the same can be achieved by: [host.options]
zlib:default_library = 'static' |
|
This looks great. The only thing I'm missing is the coherence between subprojects and differently obtained dependencies. Or there should be the possibility to declare the same for dependencies: Maybe it's better to leave them separated because the behavior can differ between pkgconfig and subprojects. |
|
Why not have the config file be in the meson dsl? You can already evalutate some meson expressions / assign and retrieve variables. # like json, but with expressions, variables, no weird comma rules
{
'a': 6,
'b': [1, 2],
'c': {
'x': "foo"
}
}
# or, define with functions
set_options(
for: 'host',
machine: machine_options(
system: 'linux'
endian: 'little'
cpu_family: 'aarch64'
cpu: 'aarch64'
),
built_in: {
'foo': 'bar',
'zlib:default_library': 'static',
},
)
# or anything in between!Pros:
Cons:
|
This is the very beginning of #3001. Currently only allows overriding an option value for all subprojects (not wired to the backend so does not actually work yet). Should permit at least:
c_std) per-subprojectI currently put the definitions inside the native file. This is probably bad. Maybe we should create its own file type and argument for this? Definitely editing the override file should cause a reconfiguration and rebuild as needed.
Ideas and design requirements welcome.