LCDproc ABI changes #2748
Comments
I think that is the only change necessary. But the issue you linked concerns the client/server protocol not the driver API, so I'm not sure it is really relevant.
That is of course true, but I am not sure the Elektra implementation is ready for a full release. The biggest disadvantage (which I am not planning to address) is that with the Elektra implementation, all drivers have to be compiled together with the server. The reason for this is that the code generation API provides no mechanism for modular specifications (it requires exactly 1 specification per executable). It would be possible to have modular specifications (I think), but it would require significant extensions of the code generation mechanism. Of course you could just ignore the |
How was it before? Does the build system allow to compile a single driver?
What is your schedule?
I would also not use it for these drivers, it would be the most modular way, though. |
The drivers (like Elektra's plugins) are loaded via
There are no targets in the root directory for that, but
Ideally I would like to finish ASAP and only fix problems in the current PR (ElektraInitiative/lcdproc#7), but not add anything. An exception to that is some mechanism to convert old config files. I still need to add that.
I am not sure I understand... |
I think building the drivers separately is not really needed then.
Is our parser not able to do parse the old files? Actually it would be nicer if a plugin makes the migration: then people could simply continue to use their old files. Actually, @llukask wants to work exactly on this topic. |
It is not really about building the drivers in the official repository separately. The problem is more that there might be users out there that have implemented their own unofficial drivers that might not work anymore. I am not sure whether these third party drivers are an officially supported feature or just a side effect of the current implementation.
Our INI parser has some problems, but it would probably be able to read the files. However, the format of the configuration was changed to varying degrees to accommodate the code generation.
Of course that would be nicer, but since their is currently no Elektra feature for that the question is purely theoretical at this point. Also I think we should encourage people to get to know the new format and all of Elektra's nice features, otherwise the update is pointless. Transforming the new format into the old one is also somewhat problematic and would definitely require some hackery. |
Yes, it is a point we should mention in the PR.
I fully agree. As already said before, ideally we also migrate to YAML. (@sanssecours 😉)
The only difference is if the transformation code is not a standalone application but a plugin transforming from the old KeySet to the new one. The new work will be about a nice language to do such transformations. For LCDproc we could implement it in python or C. As @llukask work will most likely not ready soon but we can discuss it the next meeting. Maybe @llukask can even do a case study, comparing hand-written transformations with the transformation written in his language. |
|
Hi just stumbled across this discussion and thought I should give some input. I believe you are overthinking things a bit. There is no problem with giving users a bit of a rough migration every decade or so. There is no need to put serious effort into the migration path IMO. OTOH I should remind you, that a requirement for making elektra a dependency of LCDproc is, that elekra is included in major distributions. (Think of debian testing as a good benchmark, though others are important as well.) Regarding the questions raised above: There aren't many out of tree drivers because we are fairly liberal with what we merge, but there are some. We try to keep a defined driver API to support this even though there is no lcdproc-dev package providing the header files at the moment. That being said, I don't mind if oot drivers aren't fully integrated via elektra. No need to get distracted for this corner case. About scheduling: I fully expect to do a release before any elktra specific code is merged, though it would be nice, if we could start merging elektra bits soon after the release. However checking again now, it seems there has been little progress on the distribution support front since we last discussed this, so maybe not realistic. Also I'd like to see some closer cooperation and coordination between ElektraInitiative and LCDproc. I'd rather review and discuss a PR for some small isolated part of LCDproc like one of the clients before all the work is finalized, than be presented with one monster PR, that will take me two years to review. Also we should get people from the LCDproc community to help with porting some of the drivers - that way more people will get some exposure to elektra too. |
You can find the current PR here ElektraInitiative/lcdproc#7. The PR is big in terms of files changed, but the actual changes per file are quite small (with a few exceptions). |
Thank you for jumping in!
Then we can relax more about this. I was considering here the total amount of work time needed. One person invests X days to write a migration code, and Y people save Z minutes because of this. Of course the question is how big X, Y, and Z is if it makes any sense to do that.
I think we already have very good coverage.
The upgrade from 0.8.14 to 0.8.27 (or whatever will be needed for LCDproc) should be fairly easy: especially because we also maintain Debian packages for Elektra's master.
We do not yet have code which successfully starts. The current status is here.
This is a very good suggestion. @kodebach what do you think?
Yes, we will definitely need help from the community. There are too many drivers for us to port them all and we do not have the hardware to test if it works. |
Actually it should start and run properly (if your build of Elektra includes #2707). There just isn't a way to migrate old configurations yet, although I am working on this.
I could split the PR up into one for each client and one for the server+drivers (cannot be split because it is just one specification). But for now, I will leave it as one PR, because the development is much easier if I'm just working on a single branch instead of 4. |
Very nice, I merged #2707 right now, so you could make your first PR for LCDproc.
This seems to be a low-priority issue. Better to update the tutorial (ElektraInitiative/lcdproc#5) so that people who did not hear yet of Elektra will know how to configure LCDproc.
No, this is not needed. The idea was that you make one PR of one client, obviously the one you tested most. You also do not need to work on this branch, the branch is only for review and for people who want to try it out. The PR should contain the tutorial, though. |
|
Sadly, I am now fairly certain that this current version is not suitable for a release. It turns out that, because of the massive and non-modular specification for There are also major problems with |
That is why we suggest you to finish one client for now. There you should not have these problems.
Validation was also not considered very important by @haraldg. So it should also not hold us back to make the PR for one client now. |
On the LCDproc side of things the clients as well as the server core and the drivers I updated are already finished. The remaining problems are on the Elektra side.
The problem is not the advanced validation. The problem is actually that because of #2760 the If we just want something that works, but we don't care about the guarantees that the highlevel API and code-generation can provide (if properly supported by the rest of Elektra), then the work is basically done. As I stated above, running the clients and the server (with one of the updated drivers) should work fine already. I will provide a tutorial on setting up things tomorrow. The intended setup of LCDproc (will be automated through the install script) is this: # assume lcdproc source is in /lcdproc
cd /lcdproc
# build and install LCDproc
make
sudo make install
# mount specifications
sudo kdb mount -R noresolver lcdd_spec.eqd "spec/sw/lcdproc/lcdd/#0/current" specload "app=/usr/local/sbin/LCDd"
sudo kdb mount -R noresolver lcdproc_spec.eqd "spec/sw/lcdproc/lcdproc/#0/current" specload "app=/usr/local/bin/lcdproc"
sudo kdb mount -R noresolver lcdexec_spec.eqd "spec/sw/lcdproc/lcdexec/#0/current" specload "app=/usr/local/bin/lcdexec"
sudo kdb mount -R noresolver lcdvc_spec.eqd "spec/sw/lcdproc/lcdvc/#0/current" specload "app=/usr/local/bin/lcdvc"
# spec-mount specifications
sudo kdb spec-mount "/sw/lcdproc/lcdd/#0/current"
sudo kdb spec-mount "/sw/lcdproc/lcdproc/#0/current"
sudo kdb spec-mount "/sw/lcdproc/lcdexec/#0/current"
sudo kdb spec-mount "/sw/lcdproc/lcdvc/#0/current"All of this works as expected (apart from the fact that some specifications are limited by the too many plugins error). However after using The key kdb set /sw/lcdproc/lcdproc/#0/current/lcdproc/foreground foo
#> Using name user/sw/lcdproc/lcdproc/#0/current/lcdproc/foreground
#> Sorry, module type issued the error 52:
#> error in the type plugin: The value 'foo' of key user/sw/lcdproc/lcdproc/#0/current/lcdproc/foreground could not be normalized (type is 'boolean')(Note that the However, this does not produce an error: kdb set user/sw/lcdproc/lcdproc/#0/current/lcdproc/foreground foo
#> Create a new key user/sw/lcdproc/lcdproc/#0/current/lcdproc/foreground with string "foo"Because we successfully set the key to a non-boolean value lcdproc
#> ERROR: The value 'foo' of key 'lcdproc/foreground' could not be converted to type 'boolean'.The error message is understandable for the user and because |
Yes, but the performance problem is specific to LCDd. We could and should create a PR before we start profiling and optimizing.
#2760 has nothing to do with this. You can always tamper the config file in some ways. During elektraOpen we also need to check if we have correct config. See #2382, last sentence (Expected behavior: elektraOpen should fail, telling that the type of myint does not match with the build-in spec.). Only then we can provide the guarantees. Of course #2760 is a serious usability problem and of course it is better if the
If you are root (e.g. sudo) the correct "system" namespace is used. This is also a matter of the tutorial to describe this. |
|
Yes, validation etc are not important. As long as there are no (major) regressions I'm happy to switch to elektra. The new features are a bonus, not a requirement.
Then let's find out how big Z is by giving people something to play with ...
We also have lots of users on the BSDs ...
Let's hope this is true. It looks like a lot of work is necessary (even if it is easy work) until a version of elektra useful for LCDproc will enter testing. I just had a first look at the high level API and some samples of your work. As I feared you have done a lot of grunt work without putting up anything for discussion. It is clear that we are month away from being able to merge this. The divergence, that is going to happen in between, will cause lot's of extra work. OTOH it seems it helped you to find some problems on the elektra side, so maybe it was kind of necessary to do it this way. I will post some comments about the high level API and the samples of LCDproc code into a new issue. In any case it would be great if you could give people a branch, that is ready to test, and advertise it on the LCDproc mailing list. This should help with getting feedback from the more obscure systems like the BSDs and also help getting a clear picture about the priorities on the remaining issues. |
I would say otherwise. The problem is that the specification cannot be modular and therefore has to be quite big. This can only be solved by modifications to Elektra not to LCDd.
It has everything to do with this. The type checking during
This is done via
There is no correct namespace here. The same problem occurs with the sudo kdb set /sw/lcdproc/lcdproc/#0/current/lcdproc/foreground foo
#> Using name system/sw/lcdproc/lcdproc/#0/current/lcdproc/foreground
#> Sorry, module type issued the error 52:
#> error in the type plugin: The value 'foo' of key system/sw/lcdproc/lcdproc/#0/current/lcdproc/foreground could not be normalized (type is 'boolean')
sudo kdb set system/sw/lcdproc/lcdproc/#0/current/lcdproc/foreground
#> Create a new key system/sw/lcdproc/lcdproc/#0/current/lcdproc/foreground with string "foo"I only noted that the
To be honest my top priority is producing something I can use for my thesis. I fully expected that changes will be required to actually release this version. And just to be clear, I always intended to get my changes merged (possibly after finishing my thesis). If I had put everything up for discussion I would probably have been slowed down by minor details that are important for a release, but not important for my thesis. Also I tried to keep everything as similar to the old code as possible. The major changes in
I explicitly stated above that I don't think the current version is suitable for a release. I fully expected this.
I will create a PR in the LCDproc repo containing only some or all of the clients and a tutorial on how to use them. This won't showcase all of the features, but it will showcase most of them. |
|
@haraldg wrote:
We had several problems with BSDs some time ago but as we now have BSD build servers we are quite confident that BSD is quite well supported.
Yes, it was necessary: The high-level API is a completely different view point, with different behavior and most importantly: much more guarantees. @kodebach now thinks in terms of these guarantees and started many discussions how we can also improve the lower-level parts of Elektra (in particular the spec plugin). These discussions were very fruitful, although we unfortunately do not have the man-power to implement everything soon.
Thank you!
Yes, especially the priorities will be very important to make @kodebach's work more focused. @kodebach wrote:
I created #2765 for further discussion about this guarantee.
Thank you very much for this!
I know and highly appreciate this. The point of view from @haraldg, however, is that LCDproc people were mostly left out of the discussion.
Perfect! 💖 |
|
I forgot to comment on an important statement: @kodebach wrote:
Yes, this is exactly the point of everything we are doing here: configuration-related problems should be a problem of Elektra, not of individual applications. But this does not mean that a small PR is easier to review and might expose less problems. |
Fair enough. I guess I'm seeing things too much from the "let's not do duplicate work" POV.
It's not (only) about the changes required to the PR. It's also about the fact, that it will take quite some time before a suitable version of elektra will be in enough disributions to do the merge, even if your code is ready per se. Anyway, thanks for your work and looking forward to your announcment on the mailing list. |
|
Unfortunately, I didn't have time to create a PR and write a tutorial yet. However, I managed to find the minimal set of plugins required for LCDproc. Using this CMake option should work: -DPLUGINS="c;cache;dump;gopts;hexnumber;ini;list;mmapstorage;network;ni;noresolver;path;quickdump;range;reference;resolver;resolver_fm_hpu_b;spec;specload;type;validation"
|
|
Thanks for your instructions. I actually compiled elektra and lcdproc
this evening. Here are my results, I will answer the conversations tomorrow.
After compiling I compared binary sizes with and without your modifications
(commits f0cb7bb1 vs. 7973efc3) and noticed that all binaries increased
a lot. For example:
| harald@debian:~$ ls -l lcdproc*/clients/lcdexec/lcdexec
| 33852 Jun 11 17:39 lcdproc-0.5base/clients/lcdexec/lcdexec
| 73520 Jun 11 17:56 lcdproc/clients/lcdexec/lcdexec
In some ways a disappointing result. Especially considering that the
base version has a statically linked configuration parser while the
new version is dynamically linked against libelektra.
The main cause seems to be the code inside elektragen.c, but most other
object files have increased in size too.
I also tried to take some timings, but any client segfaults immediatly.
This is on buster armhf. ("kdb ls /" works fine.)
I haven't looked into it yet, but I guess the segfault will be fairly
easy to fix. The size issue OTOH bothers me a lot.
Here is what gdb says about the segfault:
Reading symbols from lcdproc/clients/lcdproc/lcdproc...(no debugging symbols found)...done.
(gdb) r
Starting program: /home/harald/lcdproc/clients/lcdproc/lcdproc
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Program received signal SIGSEGV, Segmentation fault.
0xb6745ed0 in getPluginPlacementList (plugin=0x0)
at /home/harald/libelektra/src/plugins/list/list.c:496
496 Key * pluginInfo = keyNew ("system/elektra/modules/", KEY_END);
|
Yes, this increase is heavy, did you find out which symbols are so big? elfdiff or comparision of output from readelf. Ideally, the wrappers in the code generation should mostly disappear, as they are only for type-safety. |
See #2772 (comment). I suspect a lot of the size increase comes from the strings needed to construct the specification KeySet. |
LCDproc upstream wants to make some ABI changes:
lcdproc/lcdproc#74
It makes sense, that we also propose which changes we want to do. When these changes happen together, users have an easier upgrade path (recompile only once).
Do we need more changes than passing the Elektra handle to the plugins?
The text was updated successfully, but these errors were encountered: