FR: BasicUI: Item sort order in groups

[stefanhaustein]

Hi, I am quite happy with the Basic UI in general... However, it would be great if group items would be ordered by label by default.

Supporting sorting by any property would be even better (I have seen a forum entry where somebody wanted to sort a group of 100 items by battery level), but label should be a very good starting point / default.

[bjoernbrings]

For me it was initially sorted in the order items were created. But that got destroyed at any point in time and now it seems to be quite random.
Anyways I'd go for an initial order (not knowing whether that is possible ...). Disadvantage may be items that are later added always go to the end.

[stefanhaustein]

Creation order seems dangerous to me: users will be tempted to delete and re-create items to get a desired order, causing frustration. Label seems to be a simple default that should be immediately obvious to users.

[kaikreuzer]

Sitemap sorting is done by order - specifically by the order of items in the items file.
@bjoernbrings If this is broken, it should be fixed, because many setups rely on it.
Anyone else that wants a different sorting is usually anyhow better off to detail out the sitemap and not use the dynamic groups.

[stefanhaustein]

How does one establish the items file order with the Paper UI?

[kaikreuzer]

The same way you define a sitemap with the Paper UI: Not at all ;-)

[stefanhaustein]

Not sure what the smiley means? If this is supposed to be some form of joke, I don't get it?

Is editing files somehow the preferred way? Is this a missing feature in Paper UI?

[kaikreuzer]

No, not meant to be funny - it is just how it is and I would indeed prefer if everything could be done through the UI, but we are not there (yet?)...

Is editing files somehow the preferred way? Is this a missing feature in Paper UI?

We have some comparison of the different options at https://www.openhab.org/docs/configuration/#versatility

While discovery and thing management is quite convenient through Paper UI, item management is highly recommended to be done textually, especially once setups become bigger and more complex.

[stefanhaustein]

If just adding label order is not an option, I think there are three basic alternatives:

• Add an "order by" property to Group items. In my opinion, this would be the best option. If one cares more about functionality than presentation, this would enable group hierarchies replace sitemaps
• Add an "order by" property to UI groups
• Enable Paper UI to define the item order (just establishing parity with text file capabilities)

[DigiH]

I am also sorely missing the broken sitemap group ordering. All my items are definded in an items file, and the current random ordering is very unfortunate.

While new options will be needed for items created through the Paper UI, it would be great to have the expected behaviour reinstated for items definded in items files.

Additionally I just realised that this also messes up any group charts. Especially anoying if two related charts are shown together - in my case temperature and humidity for various rooms, with the rooms in different order and hence different colours.

[lolodomo]

All my items are only defined in a file and Basic UI and Android App are showing members of groups in the order they are declared in my file.

[DigiH]

@lolodomo - All my items are also only declared in files (openHAB 2.4) and do show up members of groups in random order when putting groups in my sitemaps.

What happens when you make changes/additions to the files where your group members are definded?

While I have manually changed this for group member items in sitemaps, unfortunately there is no manual fix for group charts :(

[lolodomo]

I never change my config files without restarting OH.
Does your problem only occur when updating your item file (and not restarting openHAB) ?

[DigiH]

No, it also happes when restarting openHAB. @bjoernbrings @kaikreuzer - Should this have been fixed in 2.4, or are you still seeing the same issues as me? I'm unsure now, hearing the different experiences in this thread

@ lolodomo - are you running openHAB 2.4 Release Build as well?

[lolodomo]

No I am running a 2.5 snapshot but I never encountered this issue with all 2.x versions I run in the past.
Can you provide files to reproduce the problem ? Maybe it is linked to the way you define your groups or your sitemap.

[DigiH]

It must definitely be a display dependant loading issue into the json database files, which obviously does happen, even if all items are defined manually.

I just copied all the group member item definitions from the original items file into a new file, commented them out in the original, and added the new items file. Now they all show up in the newly created expected order.

Not an ideal hack, as I would prefer to have these items together with their related items in the original items file, but ok for a working hack until this issue is addressed in a future build.

Just testing to see if this also survives a restart of openHAB - will report back in 30 minutes.

[DigiH]

The above mentioned hack doesn't survive a restart, but reloading the newly created items file after a startup does show the group members in the order of the items file again.

Not really sure how a load at startup differs internally form just reloading the new items file again right after the startup, especially since the new file was actually loaded last during the startup - but it does make all the difference.

[lolodomo]

You mean you have items in config files + items defined in Paper UI ?

[DigiH]

ALL my items are defined in manually configured .times files.

They must get loaded into the same database with openHAB 2, which would also contain items created in the Paper UI, as manually defined items (and Things etc. for that matter) also do show up in Paper UI with all their properties showing up nicely.

This loading from manually defined items, as described above, does not guarantee the order of items in the items file to be the order when having groups in a sitemap, but seems to be hackable by manually reloading and items file with JUST the affected group items after startup.

I only have just above 50 items which are relevant to this issue, but knowing that others rely on groups for their whole sitemap setups, they might be sticking with openHAB 1.8 until this is addressed.

@lolodomo - do you have your group items sequentially in your items files, or are there other non-group items in between them?

[lolodomo]

I have non group items.

[stefanhaustein]

No, not meant to be funny - it is just how it is and I would indeed prefer if everything could be done through the UI, but we are not there (yet?)...

So hypothetically, how could it be done there in the future and how would this be represented in data?

[DigiH]

So hypothetically, how could it be done there in the future and how would this be represented in data?

And how can we get the regression bug fixed so that we can at least get the previous sort order of order of items in the items files back, until a solution is found for UI created items.

[stefanhaustein]

In my opinion, the most stable solution that would avoid such regressions in the future would be to add an explicit numeric order field that is populated at item load time with an increasing number. This would make sure the item order can't be lost anywhere in the system while data is passed around, as it's now explicit instead of depending on stable data structures....

[lolodomo]

Is there a scenario to get the bad order issue because the order has always and still is ok for me when using items delcared in an item file.

[DigiH]

With some groups it is looking ok for me as well, with some smaller groups which have been set up ages ago, and never been touched since.

My G_Lights group however, which has been modified, has members added and sometimes taken away many times over the years, is a total mess up. Having been initially set up with OH 1.5 - 1.8, the lights always showed up in the order they were defined in in a single items file, albeit with many other items mixed in between them (I have a total of 1350 items spread over 9 items files). Now with changing to openHAB 2.4/2.5M3 I can't quite figure out what the new random order adheres to.

Similarly my temperature group (only 7 items) is also in a strange random order. It has also been slightly modified since the initial setup. Since it has fewer members than the lights group I have tried to move these members to a separate items file. This also yields a random order result after the initial startup, but just touching and therefor reloading this single temperatures items file after a startup remedies the issue and shows the items in the expected order or their order in the file.

The above solution for the temperature group is the only make shift solution I have found so far for this issue. It's not ideal, it breaks up the temperature items from their related thing items in the original file, it requires a manual touching of the new separate file after each restart of openHAB, but it is the only way I can get these group items in the desired order again, as they are displayed in a chart and therefor any shift to a manual sitemap order is not possible.

What I take away from all this is that groups which have never or rarely been touch are fine, groups which get constant work over the years are unreliably messed up.

I'm not sure where to look to see why these temperature items are in correct order again after manually touching/reloading the items file, but if someone can direct me in the right direction I'd be happy to help to sort this issue out.

Recently I have also been chatting to people, which heavily rely on item order in the files to order their whole house sitemaps. For them this is still a blocking issue to upgrade from OH 1.8 to openHAB 2. With the 5th release now of openHAB 2 should this really be a blocking issue any longer?

@bjoernbrings - since you have seen the same issue for your groups, are there any similarities to what I have described above?

[lolodomo]

Is the order wrong only after updating the items file while OH is running? Or is it wrong even after a OH restart?

On my side, I have only one file for items and always restart OH after updating my config files. Maybe that is the reason why I get no such issue.

[DigiH]

The order is always wrong after restarting OH, even after clearing the cache. If I then update the separate item file, which I have created as a test to display my temperature chart, after restarting OH the order gets restored. But always only then, after manually updating after a restart.

Such a manual update of the original items file, where the group items are spread throughout other items, does not result in the order they appear in the items file. Only keeping them is a totally separate items and updating it manually does result in the correct order.

With 1300+ items I split up my items over several file, for organisational reason, but still making sure that all group related items are together in one relevant items file.

This has consistently worked for me for years in OH 1.5-1.8, as Kai mentioned above, as it should. Only now, that I was able to upgrade to OH2, once the Raspberry Pi 4 came out with more RAM, do I consistently see the issue with several of my groups.

[lolodomo]

Could you check if having only one file solves the problem?

[DigiH]

Yes, having all 1350 items in one single file immediately solves all my group display issues. The temperature and humidity group members are now again in order in the charts, and even the 35 lights are listed in order in the sitemap, after I have reinstated showing them by group, instead of the manual individual sitemap order I have cerated since encountering this issue. All this after a restart with the single one item file.

How OH differentiates between 35 lights being listed in order in a separate file, compared to the same 35 lights being listed in the very same order, but now among hundreds of other items still boggles my mind.

Will this be the new status quo then - if you want group items to be reliably shown in order in a sitemap, or be listed in order in charts, you will have to have ALL items in just one single file, instead of just making sure that all the items in a group are together in a single items file, but allowing separate items files, as previously?

Or will this help to fix the issue, by getting closer to a solution to have the previously applicable requirement reinstated, where individual items files get loaded sequentially? The prerequisite to have all items in one single file is quite awkward once a certain amount of items are present.

[lolodomo]

At least now we know the source of the problem (and why most of users included myself never encountered it).
And we even know how to avoid it.

Remains to analyze how the loading of several files is handled. But I can't imagine how the order could be always the same in such case.
I will have a quick look only, in case it is obvious to fix.

[DigiH]

Thanks for helping me get to the bottom of the problem, even if such a very long single items file is not my desired final solution in the long run.

I'm quite surprised to hear that obviously most users do keep their items all in one single file.

What I still don't understand is how the loading of several files can mess up the order of group items, all of which were always kept within single respective files. No other items of the same groups were ever in other files.

So just the fact that other items might get loaded concurrently definitely messes up the group member order.

Would limiting the loading of items files to only ever happen successively be a solution here?

[lolodomo]

Maybe.
As I don't know enough this part (xtend loading) , I can't yet be sure.

[DigiH]

Thanks. And just to confirm, after another restart of OH all the orders of the group items are consistently fine.

[lolodomo]

Ok, I tried to reproduce.
Here is the content of my additional items file:

Group GTest

String TestItem1 "Test item 1" (GTest)
Switch TestItem2 "Test item 2" (GTest)
Number TestItem3 "Test item 3" (GTest)
DateTime TestItem4 "Test item 4" (GTest)
Location TestItem5 "Test item 5" (GTest)

String TestItem6 "Test item 6" (GTest)
Switch TestItem7 "Test item 7" (GTest)
Number TestItem8 "Test item 8" (GTest)
DateTime TestItem9 "Test item 9" (GTest)
Location TestItem10 "Test item 10" (GTest)

I restarted openHAB with this new file and my updated sitemap to show the content of group GTest.
The order of items in the group is ok as you can see:

The order is correct in all UIs !
I check quickly all my other groups and the order of items looks as expected !

For those having problems, can you provide an example of items file that allows reproducing the wrong order of items ?

[DigiH]

I'm not that surprised that with just one separate additional short items file, the issue doesn't show up.

I really think it must be the way OH 2 loads the items files concurrently, instead of sequentially, as OH 1.8 probably used to. And this then only shows up when several separate long items files are loaded.

I used to have 9 separate items files. First one, for example astro.items, with 38 astro items. Then an admin.items file, with around 150 items, most of them for background tasks and states, an mqtt items file, with 112 items, some other similar smaller or larger items files, but with the most important, and the one causing the major group order issues for me is the lights.items file, which does contain all the light items, but also many other items related to the lights, by using the same bindings for example. This file contains around 800 items, among which are 38 items which are part of the G-Lights Group. The are not listed directly one after each other, but kept within their respective Thing groups, as in this example:

Number Milight_Wohnzimmer_Decke_Presets "Presets"
Number Milight_Wohnzimmer_Decke_MasterSwitch "Decke"
Switch Milight_Wohnzimmer_Decke_Switch "Wohnzimmer Decke" (G_Lights)
Dimmer Milight_Wohnzimmer_Decke_Dimmer "Dimmer [%d]"
Color Milight_Wohnzimmer_Decke_Colour "Colour"
Switch Milight_Wohnzimmer_Decke_WhiteMode "White Mode"
Switch Milight_Wohnzimmer_Decke_NightMode "Night Mode"
Number Milight_Wohnzimmer_Decke_ColourChangeSpeed "Speed [%d]"
...

With several of these MiLights spread throughout the lights items file, whereas some other lights are defined through other bindings, like the exec binding to control Bluetooth and radio wall switch lights. Each one of these having a Switch though, as in the example above, which shows and controls the basic state of the light being on or off. Showing this G_Lights group in my sitemap allows me to instantly see how many lights are on, and when the group is shown in order gives me a quick overview of where in the house.

If this light.items file is loaded separately, along with the 8 other items files, the order of the lights in the G_Lights Group is always randomly messed up, when shown in my sitemap, but as you suggested, bringing all the items of the 9 files together in one single items file by just copying each file's content as is into a unified items file, everything works perfectly.

In OH 1.5-1.8 there has never been an issue with group member order when these separate files were being loaded, but OH 2 seems to load separate items files in a differently.

While it's not ideal for me to just have one single items file, as spreading a large amount of items over several files is more manageable, at least it works for me now as expected. I even kept my number of items files below 10, which was sort of a prerequisite for OH 1.5-1.8, but now the OH 2 docs state "… and you can create as many .items files as you need/want …"

I know it's not easy to recreate my above mentioned setup for investigating the issue, but I'm happy to help in any way I can.

Thanks

[lolodomo]

Just provide your items files (without the channel links) and we could then reproduce the issue.

[DigiH]

This will take me a while to prepare all the items. I'm also wondering if the channel links do affect the loading of the items and will therefor do some testing once I have 'sanitised' all my items files.

[DigiH]

I could kick myself for not seeing this earlier, only giving my attention to the individual items in a group, their order within the items file(s) etc.

The issue was and is that the actual group item definition was in a different file to the group member items. Obviously this was remedied when trying out just having one single items file. But even now, with being back to my 9 separate items files, but making sure that the group item definition is in the SAME file as all the group member items, everything is working fine. Why would anyone have it in a different file anyway ;)

This obviously wasn't an issue for OH 1.5-1.8 for the last several year, but looks like an acceptable prerequisite for OH2 to me, now knowing what condition has to be met.

[lolodomo]

@ghys : I believe your tag "Basic UI" is not appropriate. The problem is relative to loading of config files by the core framework and this will impact any UI, not only Basic UI.
Maybe a "won't fix" tag will be more appropriate.

[ghys]

You are right, I skimmed through the discussion a little too fast when triaging the issues here, this one it's about sitemaps in general; if the issue is still relevant it should be opened again in openhab-core because that's where an eventual solution could come from, so I'll close this one here - a new issue can link to this discussion if needed. Thanks!