Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some openzwave devices may be missing because the instance are the same but not the index #803

Closed
sbouchex opened this issue Aug 19, 2016 · 27 comments

Comments

@sbouchex
Copy link
Contributor

I'm using a qubino ZMNHTD smart meter zwave device (with openzwave) and the CommandClass looks like this:

<CommandClass id="50" name="COMMAND_CLASS_METER" version="4" request_flags="2" innif="true">
                <Instance index="1" />
                <Value type="decimal" genre="user" instance="1" index="0" label="Energy" units="kWh" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="0.0" />
                <Value type="decimal" genre="user" instance="1" index="4" label="Energy" units="kVAh" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="0.0" />
                <Value type="decimal" genre="user" instance="1" index="8" label="Power" units="W" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="0.0" />
                <Value type="decimal" genre="user" instance="1" index="16" label="Voltage" units="V" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="0.0" />
                <Value type="decimal" genre="user" instance="1" index="20" label="Current" units="A" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="0.0" />
                <Value type="decimal" genre="user" instance="1" index="24" label="Power Factor" units="Power Factor" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="0.0" />
                <Value type="decimal" genre="user" instance="1" index="28" label="Unknown" units="" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="0.0" />
                <Value type="bool" genre="user" instance="1" index="32" label="Exporting" units="" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="False" />
                <Value type="button" genre="system" instance="1" index="33" label="Reset" units="" read_only="false" write_only="true" verify_changes="false" poll_intensity="0" min="0" max="0" />
            </CommandClass>

All entries have the same instance id but different indexes.

Once loaded into Domoticz, this device only reports a single sensor : the first one !!!

Analysis: In ZWaveBase::GenerateDeviceStringID, COpenZWave::UpdateValue and other functions, the DeviceStringID does not include the index, the unicity is not complete.
In ZWaveBase::SendDevice2Domoticz, same problem : ID is built by only using the instance, not by including the index as well.

Note : The issue may appear also with Razberry

@sbouchex sbouchex changed the title Some openzwave devices may be missing Some openzwave devices may be missing because the instance is the same but not the indexw Aug 19, 2016
@sbouchex sbouchex changed the title Some openzwave devices may be missing because the instance is the same but not the indexw Some openzwave devices may be missing because the instance is the same but not the index Aug 19, 2016
@sbouchex sbouchex changed the title Some openzwave devices may be missing because the instance is the same but not the index Some openzwave devices may be missing because the instance are the same but not the index Aug 19, 2016
@gizmocuz
Copy link
Contributor

gizmocuz commented Aug 20, 2016

Yep, struggling with this myself actually this morning... but not something easy that can be changed without all users having new sensors, and old sensors not working anymore, or other sensor data being put in another (old) sensor
(It is quite easy to solve... if the above wont matter)
There was also a line at the top of the file mentioning this

Basically we should use both instance and index values

Mayor problem, every manufacturers implement it 'their' way, for instance look at this:

                <Value type="decimal" genre="user" instance="1" index="3" label="Luminance" units="%" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="54" />
                <Value type="decimal" genre="user" instance="2" index="3" label="Luminance" units="%" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="54" />
                <Value type="decimal" genre="user" instance="3" index="1" label="Temperature" units="C" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="26" />

(There is actually only 1 Luminance sensor on the device)

or this (which looks better):

                <Value type="decimal" genre="user" instance="1" index="1" label="Temperature" units="C" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="21.5" />
                <Value type="decimal" genre="user" instance="1" index="3" label="Luminance" units="lux" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="49" />

@gizmocuz
Copy link
Contributor

Or this one:

            <CommandClass id="50" name="COMMAND_CLASS_METER" version="2" request_flags="3" innif="true">
                <Instance index="1" endpoint="1" />
                <Instance index="2" endpoint="2" />
                <Instance index="3" endpoint="3" />
                <Instance index="4" endpoint="4" />
                <Instance index="5" endpoint="5" />
                <Instance index="6" endpoint="6" />
                <Value type="decimal" genre="user" instance="1" index="0" label="Energy" units="kWh" read_only="true" write_only="false" verify_changes="false" poll_intensity="1" min="0" max="0" value="2.3600" />
                <Value type="decimal" genre="user" instance="1" index="1" label="Previous Reading" units="kWh" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="2.3600" />
                <Value type="int" genre="user" instance="1" index="2" label="Interval" units="seconds" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="-2147483648" max="2147483647" value="9" />
                <Value type="decimal" genre="user" instance="1" index="8" label="Power" units="W" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="2.3" />
                <Value type="decimal" genre="user" instance="1" index="9" label="Previous Reading" units="W" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="2.3" />
                <Value type="int" genre="user" instance="1" index="10" label="Interval" units="seconds" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="-2147483648" max="2147483647" value="9" />
                <Value type="bool" genre="user" instance="1" index="32" label="Exporting" units="" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="False" />
                <Value type="button" genre="system" instance="1" index="33" label="Reset" units="" read_only="false" write_only="true" verify_changes="false" poll_intensity="0" min="0" max="0" />
                <Value type="decimal" genre="user" instance="2" index="0" label="Energy" units="kWh" read_only="true" write_only="false" verify_changes="false" poll_intensity="1" min="0" max="0" value="1.0511" />
                <Value type="decimal" genre="user" instance="2" index="1" label="Previous Reading" units="kWh" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="1.0511" />
                <Value type="int" genre="user" instance="2" index="2" label="Interval" units="seconds" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="-2147483648" max="2147483647" value="9" />
                <Value type="decimal" genre="user" instance="2" index="8" label="Power" units="W" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="1.0" />
                <Value type="decimal" genre="user" instance="2" index="9" label="Previous Reading" units="W" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="1.0" />
                <Value type="int" genre="user" instance="2" index="10" label="Interval" units="seconds" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="-2147483648" max="2147483647" value="9" />
                <Value type="bool" genre="user" instance="2" index="32" label="Exporting" units="" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="False" />
                <Value type="button" genre="system" instance="2" index="33" label="Reset" units="" read_only="false" write_only="true" verify_changes="false" poll_intensity="0" min="0" max="0" />
                <Value type="decimal" genre="user" instance="3" index="0" label="Energy" units="kWh" read_only="true" write_only="false" verify_changes="false" poll_intensity="1" min="0" max="0" value="9.2412" />
                <Value type="decimal" genre="user" instance="3" index="1" label="Previous Reading" units="kWh" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="9.2411" />
                <Value type="int" genre="user" instance="3" index="2" label="Interval" units="seconds" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="-2147483648" max="2147483647" value="9" />
                <Value type="decimal" genre="user" instance="3" index="8" label="Power" units="W" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="43.0" />
                <Value type="decimal" genre="user" instance="3" index="9" label="Previous Reading" units="W" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="42.2" />
                <Value type="int" genre="user" instance="3" index="10" label="Interval" units="seconds" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="-2147483648" max="2147483647" value="9" />
                <Value type="bool" genre="user" instance="3" index="32" label="Exporting" units="" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="False" />
                <Value type="button" genre="system" instance="3" index="33" label="Reset" units="" read_only="false" write_only="true" verify_changes="false" poll_intensity="0" min="0" max="0" />
                <Value type="decimal" genre="user" instance="4" index="0" label="Energy" units="kWh" read_only="true" write_only="false" verify_changes="false" poll_intensity="1" min="0" max="0" value="1.8164" />
                <Value type="decimal" genre="user" instance="4" index="1" label="Previous Reading" units="kWh" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="1.8163" />
                <Value type="int" genre="user" instance="4" index="2" label="Interval" units="seconds" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="-2147483648" max="2147483647" value="9" />
                <Value type="decimal" genre="user" instance="4" index="8" label="Power" units="W" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="31.4" />
                <Value type="decimal" genre="user" instance="4" index="9" label="Previous Reading" units="W" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="32.6" />
                <Value type="int" genre="user" instance="4" index="10" label="Interval" units="seconds" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="-2147483648" max="2147483647" value="9" />
                <Value type="bool" genre="user" instance="4" index="32" label="Exporting" units="" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="False" />
                <Value type="button" genre="system" instance="4" index="33" label="Reset" units="" read_only="false" write_only="true" verify_changes="false" poll_intensity="0" min="0" max="0" />
                <Value type="decimal" genre="user" instance="5" index="0" label="Energy" units="kWh" read_only="true" write_only="false" verify_changes="false" poll_intensity="1" min="0" max="0" value="0.0064" />
                <Value type="decimal" genre="user" instance="5" index="1" label="Previous Reading" units="kWh" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="0.0064" />
                <Value type="int" genre="user" instance="5" index="2" label="Interval" units="seconds" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="-2147483648" max="2147483647" value="9" />
                <Value type="decimal" genre="user" instance="5" index="8" label="Power" units="W" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="0.0" />
                <Value type="decimal" genre="user" instance="5" index="9" label="Previous Reading" units="W" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="0.0" />
                <Value type="int" genre="user" instance="5" index="10" label="Interval" units="seconds" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="-2147483648" max="2147483647" value="9" />
                <Value type="bool" genre="user" instance="5" index="32" label="Exporting" units="" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="False" />
                <Value type="button" genre="system" instance="5" index="33" label="Reset" units="" read_only="false" write_only="true" verify_changes="false" poll_intensity="0" min="0" max="0" />
                <Value type="decimal" genre="user" instance="6" index="0" label="Energy" units="kWh" read_only="true" write_only="false" verify_changes="false" poll_intensity="1" min="0" max="0" value="0.0000" />
                <Value type="decimal" genre="user" instance="6" index="1" label="Previous Reading" units="kWh" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="0.0000" />
                <Value type="int" genre="user" instance="6" index="2" label="Interval" units="seconds" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="-2147483648" max="2147483647" value="9" />
                <Value type="decimal" genre="user" instance="6" index="8" label="Power" units="W" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="0.0" />
                <Value type="decimal" genre="user" instance="6" index="9" label="Previous Reading" units="W" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="0.0" />
                <Value type="int" genre="user" instance="6" index="10" label="Interval" units="seconds" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="-2147483648" max="2147483647" value="9" />
                <Value type="bool" genre="user" instance="6" index="32" label="Exporting" units="" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="False" />
                <Value type="button" genre="system" instance="6" index="33" label="Reset" units="" read_only="false" write_only="true" verify_changes="false" poll_intensity="0" min="0" max="0" />
            </CommandClass>

@sbouchex
Copy link
Contributor Author

Even if you find a way to move from the "old" ids to the "new" ids, the database contains data for device which is probably not the right one from the beginning.

You way migrate automatically ids when it's possible (because there is no conflict between devices) and for the others, you migrate by using the first found and the other will be discovered and added afterwards.

@gizmocuz
Copy link
Contributor

Not sure what you mean with this, but for instance i have years of logging that is correct.
In your case with the quibino it might not.
you could try adding both index/instance as device string for COMMAND_CLASS_METER, COMMAND_CLASS_SENSOR_MULTILEVEL
i.e. your voltage sensor i would have expected without issues ?

@sbouchex
Copy link
Contributor Author

Your data are correct because you don't have devices having the same instance but different indexes and that's fine.
In my case, the logged data are probably mixed up because several data are logged for the same device id.

Example:

  • Device "A" : instance 1, index 1
  • Device "B" : instance 1, index 2
  • Device "C" : instance 2, index 1
    In the current implementation, only Device "A" and "C" are created

If we migrate ids during the first ZWave devices discovery, Device "A" device ids will be based on instance 1 + index 1 and the device will not be touched and all data will be kept.
Device B will be discovered and created and new data will be logged.
Device C will be migrated as well to comply with the new device id format and data will be kept as well

Sounds feasable ?

@gizmocuz
Copy link
Contributor

With device, you mean sensor on a Node?
Simply adding index+instance goes wrong, that won't work, this will go wrong

<Value type="decimal" genre="user" instance="1" index="3" label="Luminance" units="%" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="54" />
                <Value type="decimal" genre="user" instance="2" index="3" label="Luminance" units="%" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="54" />
                <Value type="decimal" genre="user" instance="3" index="1" label="Temperature" units="C" read_only="true" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="0" value="26" />

First and last will return 4

@sbouchex
Copy link
Contributor Author

Sorry for the misunderstanding, I did not mean "adding" both value but composing the id with both: In ZWaveBase::SendDevice2Domoticz, the szID is composed of: [0][nodeID (8 upper bits][nodeID (8 lower bits)][instanceID], we can move to [nodeID (8 upper bits][nodeID (8 lower bits)][instanceID][indexID]

The same think must be done in ZWaveBase::GenerateDeviceStringID (and other places)

Most difficult part: At startup, Domoticz will migrate the id to the new one by adding the index and update the database and all internal structure.

@gizmocuz
Copy link
Contributor

Most sensors (if not all) also have a unit (next to the deviceID), so we could have as device id the (NodeID<<8) | Index, and as unit the instance
Migration is one of the issues, the other is the default send functions in the base hardware class

@sbouchex
Copy link
Contributor Author

Yes but this unit may not go to this unique identifier as long as the nodeId, instance and index identifies it. Once this device ID is defined, of course all code relying on these id must be modified.

@gizmocuz
Copy link
Contributor

Yep, so , first the internal device string should be extended with both index/instance, then all generated sensors should have nodeid+index+instance

@sbouchex
Copy link
Contributor Author

That's right !

@sbouchex
Copy link
Contributor Author

sbouchex commented Nov 6, 2016

What are we doing for this subject ? Some devices like the Qubino Smart Meter is not working, do you expect a patch from me ? Need to specify more ? You do it ?

@gizmocuz
Copy link
Contributor

gizmocuz commented Nov 7, 2016

If you would like to do this, perfect.
I might release a new 'stable' version before merging ;)

@sbouchex
Copy link
Contributor Author

sbouchex commented Nov 7, 2016

ok, I will try to work on a patch but tests will be required since most ids will be changed

@gizmocuz
Copy link
Contributor

gizmocuz commented Nov 7, 2016

What is also on the 'wish list', is that we extend the zwave node table
because...

For instance, you have a zwave stick with 30 nodes, you excluded some during setup, so they wont be incrementally numbered

Now, you stick broke down... or you switch to a zwave+ stick

you need to include all your nodes again.....

It would be great if we could 'map' the new nodes to the old nodes (or visa versa)

@sbouchex
Copy link
Contributor Author

sbouchex commented Nov 7, 2016

Yes, make sense but I wouldput it into a separate issue

@dannybloe
Copy link
Contributor

Just one point of advice: always separate ids with some extra character when you combine them to form a new one:
nodeid=11, index=1 => newid=11_1
nodeid=1, index=11 => newid=1_11

without the separator you would still end up with duplicate ids (111).

@gizmocuz
Copy link
Contributor

the ids must be a numeric value

@gizmocuz
Copy link
Contributor

AABBCCDD

aa = node_id
bb = instance
cc = index
dd = free to use

something like this should be fine

@gizmocuz
Copy link
Contributor

unfortunately internally a temp sensor has only a 16bit ID... yep....
So all sensors using this, should be rebuild with pTypeGeneral, sTypeXXX.... lot of work... but....

@gizmocuz
Copy link
Contributor

gizmocuz commented Apr 2, 2017

could be done when #1412 is done

@sbouchex
Copy link
Contributor Author

sbouchex commented Apr 4, 2017

@gizmocuz sure ! as long as the id is correctly built

@frankhommers
Copy link

I am running version: 3.8153 (on RPi with Razberry).
Will it still work if I copy the current Config directory from this github repo over the running one?

@davidiluvatar
Copy link

Hello!
First of all, my thanks for domoticz, which is perfect for my needs.
I would like to ask about this issue (I asume there isn't yet a release that solves this problem).
I have been testing a workaround, which in brief is:

  1. In domoticz I create virtual devices for variables I'm interested in
  2. I periodically run a script (nodejs) that polls https://127.0.0.1/ozwcp/poll.xml
  3. I parse the XML response to get the values I need
  4. I update the virtual devices calling https:/127.0.0.1/json.htm?type=command&param=udevice...

However, I don't get always responses from poll.xml. I have traced calls from the OZWCP pages in domoticz, and found that maybe a previous call to /json.htm?type=openzwavenodes could help, but I'm not sure if this is the way. So my question is: could you, please, confirm if this workaround is valid, or give me some directions to develop a good script? In particular I'm interested in correct URLs.

Thank you,
David.

@gizmocuz
Copy link
Contributor

Why don't try to update to the beta version?

@davidiluvatar
Copy link

Does the latest beta correct this issue? My main concern is the Qubino Smart Meter.

@davidiluvatar
Copy link

If the latest beta solves this, wil I have to exclude/include the device again so domoticz assigns correct new IDs?

@gizmocuz gizmocuz closed this as completed Jan 1, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants