Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

default value calculation #2231

Closed
markus2330 opened this Issue Sep 9, 2018 · 20 comments

Comments

4 participants
@markus2330
Copy link
Contributor

markus2330 commented Sep 9, 2018

To continue the discussion of lcdproc/lcdproc#104

I think the scripting languages are quite clearly off the table (only /bin/sh might be present).

@Piankero can you take the task to design a language for default value calculation?

As first step we need to collect requirements.

@Piankero already started with:

  1. Size calculations from multiple other settings where users put in i.e. "4GB, 128MB" etc: There is no possibility to just extract a part of a string and make a calculation with it
  2. Prohibit values (by throwing an Err in the calculation process) to hinder the user from setting a value if another value is already present: Cassandra uses this for example (you may either have listen_address or listen_interface set). This is no "default value" calculation per se but this potentially bigger plugin has more advantages.
  3. Get the username/current IP/timezone/cpu threads/RAM/free space/ etc. from the system as default value
  4. Check if a desired port is open as default, if not try desired port+1/2/3/4/5 etc.
    much much much more

For me:

  1. is wrong use of configuration settings. A configuration setting should always contain exactly one value. No need of extracting values should be necessary.
  2. Yes, this is not default value and has little to do with the issue.
  3. This is also not provided by interpreters (you need modules that do this). Writing such plugins is quite trivial in Elektra (see uname plugin)
  4. I am confused. This is already discussed in #2169 and not something usual interpreters can do (again you would need modules for the interpreter)
@Piankero

This comment has been minimized.

Copy link
Contributor

Piankero commented Sep 9, 2018

@Piankero can you take the task to design a language for default value calculation?

Yes, but the question is what should be possible with this plugin. Designing it just for LCDproc is trivial.

As first step we need to collect requirements.

Yes, but where should I collect such requirements?

Ad 1. Well, in the JIRA/Confluence configuration I already came to a point where I could have needed that. You had multiple ways of setting RAM usage (eg. 1gb, 1024mb, etc.). One use case for example would be to gather all those RAM settings, add them up and calculate how much you can allocate to e.g. Bitbucket as default. This would again require modules (to see the overall available RAM).
Ad 3. It is not possible to provide some default modules like the os module?
Ad 4. #2169 deals with prohibiting settings of invalid ports. The default value calculation plugin would search for an open port by itself.

If it is impossible to provide additional modules in python we will just find another way and create a normal plugin.

@haraldg

This comment has been minimized.

Copy link

haraldg commented Sep 10, 2018

After sleeping over this I think (with my lcdproc hat on) that the root of this issue is a confusion: Namely you are confusing (default) configuration and auto-detection. Yes, clearly auto-detection can't be specified in a straight forward way, thus you are looking for a turing complete scripting language.

However I believe configuration specification shouldn't be turing complete. Quite to the contrary: If auto-detection (or whatever other calculation) is a valid configuration, then the specification should state this. Ie the type of a configuration item could be (int or autodetect), where the autodetect value might be represented by a special string or maybe just by the absence of the key or both. Perhaps haskells Maybe monad is a good analogon.

I'm not sure if there is merit in elektra supporting simple auto-detection features, but it is obvious that complex auto-detection can only be handled by the application itself, so your API will need to support that anyway. Therefore I recommend that you focus on the auto-detection on the application side case first and implement lcdproc in that way.

@markus2330

This comment has been minimized.

Copy link
Contributor Author

markus2330 commented Sep 10, 2018

Piankero writes:

Yes, but where should I collect such requirements?

From the applications you look at. Maybe also by looking at the KDE KConfig XT files? Of course I will also help.

Well, in the JIRA/Confluence configuration I already came to a point where I could have needed that.

Yes, but then it is more a problem of how the configuration system works. In Elektra, with a unit plugin #1398 later plugins would always see the full numbers (without SI-units or metric prefixes), so in the higher-level rules you do not have to worry if the user says 1GB or 1024MB.

If it is impossible to provide additional modules in python we will just find another way and create a normal plugin.

Only impossible for embedded systems like LCDproc. JIRA/Confluence, MongoDB, ... are completely different topics.

haraldg writes:

I'm not sure if there is merit in elektra supporting simple auto-detection features, but it is obvious that complex auto-detection can only be handled by the application itself, so your API will need to support that anyway.

A big advantage of having auto-detection within Elektra is that users/admins can query what an application would get with a kdb get. Obviously, more complex auto-detection in Elektra would require either duplicated communications at startup or some IPC. But if I understood correctly auto-detections features do not exist in LCDproc anyway, so this should be a non-issue?

@haraldg

This comment has been minimized.

Copy link

haraldg commented Sep 10, 2018

@markus2330

This comment has been minimized.

Copy link
Contributor Author

markus2330 commented Sep 11, 2018

I fully agree that returning values that might be outdated is very dangerous. Actually this is a well-known problem, it is called situation-awareness: admins often believe that the configuration is A (because everywhere they look it says A) but in reality it is B. Of course, simply not showing A nor B is only a slight improvement, the admin is still not aware of the situation then.

And I also agree that Elektra is not always the right tool. If values quickly and constantly change (like temperatures) I would not call it configuration and then Elektra is simply the wrong tool. What makes sense, however, is that some daemon translates such transient low-level values to higher-level values to be used in the configuration (specification), like "overheat" or "mobile phone is in the pocket".

Luckily, due to the excellent work of @waht we now have notification mechanisms that can avoid situations where such higher-level values get out of sync (between the daemons and the applications using them). Of course, higher-level APIs are essential so that the application developers do not need to implement the synchronization themselves.

Having a completely separate APIs would undermine the main purpose of Elektra: allowing to integrate and connect situation-aware configuration settings in order to improve default-behavior of applications and situation-awareness of admins.

@haraldg

This comment has been minimized.

Copy link

haraldg commented Sep 12, 2018

Of course, simply not showing A nor B is only a slight improvement, the admin is still not aware of the situation then.

The solution IMO is to show "auto-detected" thus prompting the admin to look up the actually uses value somewhere, like a log file.

What makes sense, however, is that some daemon translates such transient low-level values to higher-level values to be used in the configuration (specification), like "overheat" or "mobile phone is in the pocket".

Yes, that's basically what I did in libautomation too. However my plan is (once libautomation is elektrified) that the configuration contains a link to the run-time database. (Ie use elektra to retrieve the key to lookup the actual and current value in the other database)

Having a completely separate APIs would undermine the main purpose of Elektra: allowing to integrate and connect situation-aware configuration settings in order to improve default-behavior of applications and situation-awareness of admins.

Well, I don't care about that. I think the selling points of elektra are intelligent merging of configurations from different sources and maybe also keeping lineorder and comments during non-human updates. I don't care much about specification and applications integration, but I can see that there are mild benefits for lcdproc and it is clearly part of the same problem, so I think it is the right thing to do that too. (I'm already curious what you people come up with in regards to application integration between lcdproc's server and clients.) But it you start mixing "situation", as you call it, with configuration, I think you are murkying everything. Maybe I just don't know the technical details, but how do you even hope to do configuration merges properly, when the definition what configuration itself is becomes vague?

How many users do you have that request having auto-detection on the elektra side? Because my estimate is, that if only a small part of your users is actually requesting such a feature, then probably having it will overall be an additional obstacle to adoption of elektra.

@markus2330

This comment has been minimized.

Copy link
Contributor Author

markus2330 commented Sep 12, 2018

The solution IMO is to show "auto-detected" thus prompting the admin to look up the actually uses value somewhere, like a log file.

I have never seen a satisfying solution where everything is logged in a way so that it is easy to know which value is now actually used, especially if reload is supported.

However my plan is (once libautomation is elektrified) that the configuration contains a link to the run-time database.

Ok.

I think the selling points of elektra are intelligent merging of configurations from different sources and maybe also keeping lineorder and comments during non-human updates.

From my experience the selling points are very different for different people. For example, KDE people liked the idea that distributions could connect configuration items for a better user experience (like having shortcuts to be changed on one place for the whole system).

Maybe I just don't know the technical details, but how do you even hope to do configuration merges properly, when the definition what configuration itself is becomes vague?

Merging happens on concrete namespaces, i.e. only on persistent configuration files. So it is not concerned about calculated values and stuff.

How many users do you have that request having auto-detection on the elektra side?

In the questionnaire 29% said they prefer auto-detection to reduce the number of options. It did not, however, specifically ask where the auto-detection should take place.

From the point of complexity, however, it is quite clear that most applications will never write auto-detection features because it is difficult to achieve in nearly every case (except when some envs tell you what you detected). In Elektra auto-detection code would be reusable. This is not really important for lcdproc but very important for the many apps that need very similar auto-detection (like "which desktop I am running on", "which OS I am running on", "which proxy to use" and similar).

But you are right, it might be better to not sell this too much because it is a very advanced feature, which needs a lot of effort to work properly.

@haraldg

This comment has been minimized.

Copy link

haraldg commented Sep 14, 2018

@markus2330

This comment has been minimized.

Copy link
Contributor Author

markus2330 commented Sep 14, 2018

That's quite a bold statement. I think the Xserver does a good job of combining configuration, auto-detection and logging.

Of course the Xserver is the most developed piece of free software in that area (together with gpsd). But especially with Xserver I have a lot of mixed experience. For me it was hard to find out which changes in the Xorg.conf disable which auto-detection mechanisms using the log files. What rescues the situation is that with xrandr you can dynamically query what the actual settings are. And this is basically what I propose to do with Elektra.

The low-level API allows you to see what's actually in the configuration files and the high-level API offers type-safety and auto-detection?

Nearly: the configuration in the namespaces reflect what is in the config files (or command-line arguments, ...) and the cascading namespace reflects what overall wins and this could be calculated/default/... The high-level API lets you only see what is in the cascading namespace. The low-level API supports both the logical (=cascading) and the physical (=namespaces) view.

So for example, if you have a system/my/key you know it is from a config file in /etc. If you lookup /my/key you do not know from which config file, or if it is from a config file at all (it might be calculated or default) unless you look at the name of the key you got after the lookup (only in low-level API).

How easy is it to use both APIs at the same time?

Applications should exclusively use the cascading view, either via low-level or high-level API. Using the specification you can restrict the config in a way so that you are sure that every config comes from some specific config file, for example kdb setmeta namespace/#0 /my/key system ensures that the key comes from /etc (or is calculated or default). If you do not want calculated values or defaults, simply do not specify them 😄

I hope the similarity between systemd and elektra is easy to see.

It is a difficult topic because it is also very sensible and political. Elektra tries hard to be portable and that completely independent implementations are easily possible, afaik, systemd does not. Furthermore, Elektra tries hard that applications only have minimal dependencies to internals of Elektra, it is only a key/value API after all.

Where will this auto-detection logic live

It is in the configuration specification. The same where the default values and validations are.

and how is it maintained?

Specifications need to be maintained by either the applications or distributions. Elektra Initiative can help on request and of course provides reusable modules. But most of the specifications should be written by the application developers, they are the only ones who know what their configs should do.

How are you going to handle the case where auto-detection really only can be handled by the application itself, because doing auto-detection would require reimplementing vital parts of the application.

As said, it is application/distributions that need to handle this. They can refactor detection code to be in libraries or the detection data to be shared via IPC. It is not rocket science to do that.

@kodebach

This comment has been minimized.

Copy link
Contributor

kodebach commented Sep 14, 2018

How easy is it to use both APIs at the same time?

To expand on this question: The high-level API simply builds on top of the existing low-level API. But the current WIP implementation, considers the structures of the low-level API an implementation detail and therefore doesn't provide access. It would be possible to expose these structures, but can't think of any reason why that would be necessary.

Personally I think, if something is considered part of the configuration, the user should be able to manually set it and if the user did set a value, this exact value should be used. If the value should always be calculated it should not be part of the configuration. Therefore default value calculation and auto-detection would only kick in, if no value was present in the configuration. In turn it would not make sense to bypass those mechanisms, because you would either get the same value as before, or no value at all.

Regarding auto-detection in general:
I think one very simple but possible approach for auto-detection would be to specify an executable which would be called via exec with a given list arguments anytime the configuration value is accessed. The resulting value would simply be whatever the executable writes to stdout. This would be a very transparent but flexible approach. It would also allow for reuse of existing tools instead of writing a new plugin for every little thing like uname. For ease of use we could also allow piping the output through some other executables.

@markus2330

This comment has been minimized.

Copy link
Contributor Author

markus2330 commented Sep 14, 2018

Therefore default value calculation and auto-detection would only kick in, if no value was present in the configuration.

This is the situation with default values anyway. Default values have the lowest priority.

or no value at all

This is not possible with the high-level API. We directly return the types the user wants, there is no way to represent "there is nothing". Thus specified default values are essential so that we can provide our guarantees.

specify an executable

The shell plugin basically does what you describe. At the moment it has hard-coded placements and cannot work as storage plugin. But to change this shouldn't be too hard.

@kodebach

This comment has been minimized.

Copy link
Contributor

kodebach commented Sep 14, 2018

or no value at all

This is not possible with the high-level API. We directly return the types the user wants, there is no way to represent "there is nothing". Thus specified default values are essential so that we can provide our guarantees.

With the high-level API yes, but the question was about using the low-level API to bypass all auto-detection and default-value calculation, to access the actual stored value. Which is as I described not very sensible, as you would not really gain any information.

@haraldg

This comment has been minimized.

Copy link

haraldg commented Sep 16, 2018

@kodebach

This comment has been minimized.

Copy link
Contributor

kodebach commented Sep 16, 2018

The information to gain would be whether a value is default or explicit. I think applications might want to behave differently depending on that.

If that is all you need to know, we can simply provide a function for checking that within the high-level API.

For example think about a future driver of LCDd, where the display size can be queried from the hardware (at least for some variants) but you need pretty much a full implementation of some custom communication protocol to do so. But at the same time the display size should also be setable by the user in configuration.

For a case like this, I would recommend using the shell plugin or a modification thereof. So the specification would (in some form or another) say: If this key contains no explicit value, call this executable with the following arguments to calculate it. The shell plugin does still require /bin/sh, but as mentioned before, I think something similar could be achieved by using exec.

@markus2330

This comment has been minimized.

Copy link
Contributor Author

markus2330 commented Sep 21, 2018

My question was: What would be in the specification in this case? I still think it is unanswerd.

In the specification there is metadata which can be processed in plugins. doc/METADATA.ini describes which metadata currently is available in Elektra's specification language. Applications are free to extend this language with new metadata and implement the necessary behavior with plugins.

So for example, in LCDproc one might write a plugin "detectlcd" which processes the metadata "default/detect/lcd/size" and replaces such default values with the detected LCD size. Then one could write the string "default/detect/lcd/size" in the configuration specification. When mounting this specification, the plugin "detectlcd" would be automatically added. In configuration items which use this specification, the detected LCD size would be used if the user does not provide anything else (defaults have the least priority).

It's not important in this context, but JFTR: I think of the specification as part of the source code of an application (especially also in the sense that it will be version specific: new version -> new specification). While distributions are free to patch applications how ever they want, usually distribution maintainers and application maintainers both frown upon that. Thus I think distributions will usually set default values in (fall-back) configuration files, not in the specification itself.

Specifications in Elektra are configuration files, so distributions can patch this without diving into the source code. And because specifications live in a different namespace (spec) they do not preoccupy system configuration. So distributions can patch the default configurations and still leave /etc empty.

The information to gain would be whether a value is default or explicit. I think applications might want to behave differently depending on that.
If that is all you need to know, we can simply provide a function for checking that within the high-level API.

I think we should be very careful here. If applications implement different behavior based on this, it makes it more difficult for admins to understand what a specific configuration will do. Most likely a separate boolean is the better way.

@haraldg

This comment has been minimized.

Copy link

haraldg commented Sep 24, 2018

@kodebach

This comment has been minimized.

Copy link
Contributor

kodebach commented Sep 24, 2018

ie it would need to reimplement a fairly huge part of what is the driver in the first place. It was this case, that I asked about.

Lcdproc drivers are shared libraries, so the plugin for elektra could simply rely on a function provided by the shared library.

introducing new settings causes configfile incompatibility

This is one of the many applications for default values. One would simply set the default so that the program would behave the same way it did before the configuration value existed.

@haraldg

This comment has been minimized.

Copy link

haraldg commented Sep 24, 2018

@markus2330

This comment has been minimized.

Copy link
Contributor Author

markus2330 commented Sep 30, 2018

If it is treated as configuration file then I expect there will be problems with old versions lying around, multiple versions installed on the same system, etc.

Configuration in Elektra has versions (part of key name to be incremented if there are incompatible changes).

  • the new boolean would basically say "ignore the value of the other key (at least in some cases)" which is a bit messy and confusing

Elektra could validate that the other key is not set in that case. But you are right, it has other problems on its own. So if it creates compatibility problems it might be better if we leave it as is.

@Piankero

This comment has been minimized.

Copy link
Contributor

Piankero commented Sep 30, 2018

Yes, but where should I collect such requirements?

From the applications you look at. Maybe also by looking at the KDE KConfig XT files? Of course I will also help.

I added a PR in #2259 which summarizes my findings. It's just another basis for discussion

@markus2330 markus2330 closed this Jan 23, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.