Switch and package configuration variables

[dbuenzli]

Switch and package configuration variables

Switch and package configuration variables are respectively per switch and per package key-values bindings that can be accessed in opam files to configure packages.

Configuration variables come in two flavors, input and state variables. The difference being that input variables can be set by the opam end-user (they express a configuration desire) while state variables are usually defined by the result of install processes (they express a configuration reality).

More precisely for package configuration variables:

• input variables can be set before package installation to configure the package installation process.
• state variables can be declared by the package install to provide information about its installation for other packages to use.

For switch configuration variables:

• input variables can be set to globally configure packages that may use the variable.
• state variables are variables determined by the switch creation procedure

Note that both input and state variables share the same namespace. It is not possible to declare both an input and state variable with the same name.

Also note that the current value of an input variable says nothing about the current install state of a switch or a package as they may be set by the end-user at any time without further side-effects. (N.B. if we had precise dependency tracking of variables in the opam files we could actually reinstall any package that depends on a variable whenever it is changed, but that may not be desirable either).

Input variable declaration

Switch input variables are defined in the switch's .comp file and package input variables are defined in the package's opam file via the config-input-vars field as follows:

config-input-vars: [ ([IDENT (BOOL | STRING | [ STRING+ ]) STRING])+ ]

That is each input variable needs to define a name, a default value (which can be a variable interpolated string, e.g. in a package it can refer to a switch input variable) and a documentation
string. Example:

config-input-vars: [
  [ "modernity" false "Use CPU extensions (SSE2, AES-NI)." ]
]

State variables declaration

State variables are defined in the switch's .comp file and the package's opam files via the config-state-vars variables as follows:

config-state-vars: [ ([IDENT STRING])+ ]

That is each state variable needs to define its name and a documentation string.

For packages, the package must then define the value of each of the declared variables by installing a opam.config file with the package in its lib directory, i.e. $opam/$SWITCH/lib/$PKG/opam.config.

It is an install error if the package declares a state variables and fails to define it in its opam.config file.

FIXME I know less about the switch install procedure but I guess something similar could be done.

Setting input variables

Input variables can be set by using the following invocations:

opam config var-set $PKG:$VAR $VALUE
opam config var-set $VAR $VALUE

For packages this commands writes the $VAR to $VALUE binding in $opam/$SWITCH/config/$PKG.config file. For switches it does so in the file $opam/$SWITCH/config/global-config.config. The invocations error if $VAR is not declared in the package's or switch's config-input-vars field.

Input variables can be reset to their default state using:

opam config var-reset $PKG[:$VAR]
opam config var-reset [$VAR]

Invoking this command resets the value of $VAR to the one that is declared in the opam or .comp file. If $VAR is unspecified all the input variables are reset. The invocations error if $VAR is not declared in the package or switch's config-input-vars field.

Getting configuration variables

Both input and state configuration variables can be accessed using the following invocation:

opam config var [$PKG:]$VAR

They can also be used in the fields of opam files that support string interpolation to get their value by using $PKG:$VAR and $VAR. The invocations error if $VAR is not in declared in one of the config-*-vars field.

[dbuenzli]

Related to #2245

[samoht]

Should it be .comp file for the switch? Otherwise, looks very sensible to me.

/cc @mato to check that the proposal could help to cross-compile to rumprun.

[dbuenzli]

Should it be .comp file for the switch? Otherwise, looks very sensible to me.

Yes, updated.

/cc @mato to check that the proposal could help to cross-compile to rumprun.

What do you mean exactly ? I don't think cross-compile is in reach before opam actually provide global variables that make the build/host machine distinction and we actually get the packages to use them (see here).

[samoht]

@dbuenzli see here. This could be solved by opam config var-set.

[dbuenzli]

@dbuenzli see here. This could be solved by opam config var-set.

Ok but not exactly that way since opam config var-set errors if the variable doesn't exist. Also IIUC this is a package trying to declare a switch output configuration variable, this proposal does not allow this. The package could however provide output variables that could be used by the other packages.

[AltGr]

Thanks!
A few remarks:

• I am curently thinking very seriously about removing the .comp files altogether¹. The compiler variables (in global-config.config) are actually a choking point here, and that would more or less result in merging package and switch configuration variables... We'd need a way to specify input vars to opam switch install but otherwise the proposal seems to fit well for that purpose.
• How does this relate to features ? Those also define variables, and in the same namespace, but dynamically evaluated; and they are used both for the package's own build and readable from others. It feels like too many options, could they be replaced by this ? Their limitation is that they can't be set by the user at the moment.
• Checking the existence of an input variable when setting can't be very strict: what if different versions of the package don't define the same variables, or if the package or variable gets removed from the repository ? This should still avoid spelling errors (but there may even be a --force).
• install error on undefined output var: it shouldn't be as harsh as removing the package and resetting the build though. As for input vars, we can have different ouptput variables for different versions, and can't completely avoid the case of a variable being undefined.
• referring to undefined variables is already supported -- we handle the undefined value and propagate it in formulas. There is no built-in way to check if a variable is defined or not though. History has taught us to be careful when adding warnings that could completely spam the users on opam update.

¹ it's natural after moving the actual compilers to packages. The main motivation is not to duplicate all signature and validation mechanisms for both package and compilers, which is a very different hierarchy and file naming conventions.

[dbuenzli]

I am curently thinking very seriously about removing the .comp files altogether¹. The compiler variables (in global-config.config) are actually a choking point here, and that would more or less result in merging package and switch configuration variables... We'd need a way to specify input vars to opam switch install but otherwise the proposal seems to fit well for that purpose.

I don't know exactly what your plans are but don't you think you'd still need some kind of switch description file ? If that's the case that seems to be the right place to describe the switch variables. If you think you can eschew the switch description file and that packages create switches (how ? via information provided by a field ?) then yes we can merge the two notions and have switch creating packages define the global variables through their own opam.config file (variables which should then be accessible withouth the $PKG: prefix. Trying to set user variables for a switch creating package would simply create the $opam/$SWITCH/config/$PKG.config file. But then maybe this should be sorted out after we get a concrete compiler as packages proposal.

How does this relate to features ? Those also define variables, and in the same namespace, but dynamically evaluated; and they are used both for the package's own build and readable from others. It feels like too many options, could they be replaced by this ? Their limitation is that they can't be set by the user at the moment.

This was also a question I had. Reading again about them in the dev manual, it seems to me that we could replace them by this.

Checking the existence of an input variable when setting can't be very strict: what if different versions of the package don't define the same variables,

Ah yes. Can't we simply union the variables found in the descriptions.

or if the package or variable gets removed from the repository ? This should still avoid spelling errors (but there may even be a --force).

Maybe having a --force is a good idea.

install error on undefined output var: it shouldn't be as harsh as removing the package and resetting the build though. As for input vars, we can have different ouptput variables for different versions, and can't completely avoid the case of a variable being undefined.

Why not ? I think that not checking this is going to bring more problems in by propagating wrong information in the system. Packages can have different output variables for different versions as described by their opam files, I don't see this as a problem. Besides I think that usage of an undefined output variable by another package should also make its install fail.

referring to undefined variables is already supported -- we handle the undefined value and propagate it in formulas. There is no built-in way to check if a variable is defined or not though. History has taught us to be careful when adding warnings that could completely spam the users on opam update.

I'm not talking about opam update here I'm talking about opam install which is quite different. Undefinedness can be checked at that point.

[AltGr]

features vs. config vars: features are not used at all at the moment and they might be a bit too complex ; if it turns out the functionality is indeed included in config vars, the simpler alternative is fine with me.

Besides I think that usage of an undefined output variable by another package should also make its install fail.

Rather, this could be detected statically. My concern is that it adds failure to a package's definition depending on what's in another package ; e.g., your package could get broken because a new version of a dependency that doesn't define some variable anymore. While this is obviously the case for code, it isn't the case for package definitions as of now.

[AltGr]

on the switch/package: I am still thinking about how it should be done. Indeed some specific fields (or flags) could allow opam to recognise switch options within the set of package definitions...

[dbuenzli]

My concern is that it adds failure to a package's definition depending on what's in another package ; e.g., your package could get broken because a new version of a dependency that doesn't define some variable anymore. While this is obviously the case for code, it isn't the case for package definitions as of now.

I don't understand the concern here. I fail to see how this is different from an API break: the other package changed the way it works by no longer exporting a package state variable in its definition which makes it incompatible with my package since I was actually using that package state variable to configure myself. E.g. suppose I'm using --conf-mypackage=$OTHERPKG:$VAR to configure my package and that a subesquent version of $OTHERPKG no longer exports $VAR, I certainly don't want opam to try to run my install instructions with an empty string instead of $OTHERPKG:$VAR for obvious (it seems to me) sanity reasons.

[mato]

On Tuesday, 07.07.2015 at 11:47, Daniel Bünzli wrote:

@dbuenzli see here. This could be solved by opam config var-set.

Ok but not exactly that way since opam config var-set errors if the variable doesn't exist. Also IIUC this is a package trying to declare a switch output configuration variable, this proposal does not allow this. The package could however provide output variables that could be used by the other packages.

ocaml-rumprun is a re-targetable cross toolchain, and it needs to know the GNU autoconf-style tuple of the underlying (C) cross compiler to use.

This is currently set by the user at opam install time by specifying RUMPRUN_PLATFORM as an environment variable, for example:

RUMPRUN_PLATFORM=i486-rumprun-netbsdelf opam install ocaml-rumprun

Note: I will be changing the RUMPRUN_PLATFORM name, probably to RUMPRUN_TOOLCHAIN_TUPLE, as we have now standardised on "platform" to mean "target board or hypervisor" in the context of rumprun.

This variable does not have a meaningful default, it depends on which toolchain(s) the user may have available. If I understand this proposal correctly, then this could be a per-package input variable?

Regarding the global ocaml-rumprun-* variables that the package sets, these can probably be reduced to a single ocaml-rumprun-toolchain-tuple output variable, which reflects the users toolchain choice at the time ocaml-rumprun was installed.

Provided this output variable is accessible to all packages during build, it can be a per-package variable instead of a global variable. The variable is required during the build process in order to install the cross-compiled packages in a separate directory namespace in the switch, so that the cross-built packages do not conflict with native packages in the same switch.

It is also required for some packages which build C code in order to know which toolchain to use.

On a related note, I would like to document the cross-compilation internals and start a discussion on how to improve OPAM support for this going forward. Is this best had on opam-devel or in a Github issue?

[dbuenzli]

If I understand this proposal correctly, then this could be a per-package input variable?

Yes.

Provided this output variable is accessible to all packages during build, it can be a per-package variable instead of a global variable.

Yes. It's just that you'll need to prefix it with the name of the package, i.e. ocaml-rump:$VAR.

On a related note, I would like to document the cross-compilation internals and start a discussion on how to improve OPAM support for this going forward. Is this best had on opam-devel or in a Github issue?

I think it's better if we start with a discussion on the mailing list. FWIW I unsucessefully started one last week here. You are also welcome to comment on my cross-compilation support ideas for assemblage here.

[mato]

On Thursday, 09.07.2015 at 04:40, Daniel Bünzli wrote:

If I understand this proposal correctly, then this could be a per-package input variable?

Yes.

Provided this output variable is accessible to all packages during build, it can be a per-package variable instead of a global variable.

Yes. It's just that you'll need to prefix it with the name of the package, i.e. ocaml-rump:$VAR.

Ok. Why the constraint on input and output variables not having the same
name? For ocaml-rumprun it makes sense for the input and output variable be
called ocaml-rumprun:toolchain-tuple.

[dbuenzli]

Ok. Why the constraint on input and output variables not having the same
name? For ocaml-rumprun it makes sense for the input and output variable be
called ocaml-rumprun:toolchain-tuple.

Because a reference to ocaml-rumprun:toolchain-tuple in an opam file would then be ambiguous.

[gasche]

I ended up here by chance and am wholly unqualified to comment on OPAM's design, but given that I read this interesting proposal I may as well make some (probably naive) comments.

For packages, the package must then define the value of each of the declared variables by installing a opam.config file with the package in its lib directory, i.e. $opam/$SWITCH/lib/opam.config.

This is a typo and you mean $opam/$SWITCH/lib/$PKG/opam.config, right?

N.B. if we had precise dependency tracking of variables in the opam files we could actually reinstall any package that depends on a variable whenever it is changed, but that may not be desirable either

Well at least it would be very useful if packages stored the ambient state at the time of their installation. It is rather common to have users report "if installed X but cannot do Y, why?" with the answer being "Z is not configured on your system so Y was opted out"; not being able to debug these situations by knowing for sure the state available at installation time would be a pain. The command opam config var could be extended with a parameter --at-install-of=$PKG so that the query is made against the stored state at $PKG's install time.

(As a user I thought it would be nice if, as a special case, opam config var $PKG:$VAR would check if the $VAR setting changed since $PKG's install, and also provide the old value in that case. But for scripting purposes it is better if the output of this command is simple with a regular format, so it's probably not such a great idea after all.)

(about opam config var-reset $VAR) If $VAR is unspecified all the input variables are reset.

This looks like a major shooting-yourself-in-the-foot opportunity. It would be much safer to have a opam config var-reset-all command, or an --all parameter to pass to var-reset before you trash the user's environment. (Is there a good way to dump and restore environments? Would it be reasonable for mutation commands to store by default the previous state in a temporary file, at least when used interactively?)

opam config var [$PKG:]$VAR

What is the command to list all variables set for a given package?

(Is there a command that shows the variable values, but also their documentation strings? It may be useful to have an --interactive argument to enable more user-useful behaviour, and a --non-interactive one for scripting-useful output, but that is a cross-cutting opam concern.)

what if different versions of the package don't define the same variables, or if the package or variable gets removed from the repository

Could we explicit set and get variables for some specific version of the package only?

[AltGr]

Parts of this are now here. Still missing:

• input variables
• switch variables are there, but can only be specified manually opam config set, after switch creation.

So the proposal still makes sense as it is. Input variables could be quite useful.