Many entries in database.db

[ciotlosm]

I joined only 1 device and I see this in my database.db:

{
    "$$indexCreated": {
        "fieldName": "id",
        "unique": true,
        "sparse": false
    }
}
{
    "id": 1,
    "type": "Coordinator",
    "ieeeAddr": "0x00124b000e896dd0",
    "nwkAddr": 0,
    "manufId": 0,
    "epList": [
        1,
        2,
        3,
        4,
        5,
        6
    ],
    "status": "online",
    "joinTime": 1527937154,
    "endpoints": {
        "1": {
            "profId": 260,
            "epId": 1,
            "devId": 5,
            "inClusterList": [],
            "outClusterList": [],
            "clusters": {}
        },
        "2": {
            "profId": 257,
            "epId": 2,
            "devId": 5,
            "inClusterList": [],
            "outClusterList": [],
            "clusters": {}
        },
        "3": {
            "profId": 261,
            "epId": 3,
            "devId": 5,
            "inClusterList": [],
            "outClusterList": [],
            "clusters": {}
        },
        "4": {
            "profId": 263,
            "epId": 4,
            "devId": 5,
            "inClusterList": [],
            "outClusterList": [],
            "clusters": {}
        },
        "5": {
            "profId": 264,
            "epId": 5,
            "devId": 5,
            "inClusterList": [],
            "outClusterList": [],
            "clusters": {}
        },
        "6": {
            "profId": 265,
            "epId": 6,
            "devId": 5,
            "inClusterList": [],
            "outClusterList": [],
            "clusters": {}
        }
    },
    "_id": "6aOj4oHjIO6vtdz5"
}
{
    "id": 2,
    "type": "EndDevice",
    "ieeeAddr": "0x00158d0001a67780",
    "nwkAddr": 64522,
    "manufId": 4151,
    "epList": [
        1
    ],
    "status": "online",
    "joinTime": 1527937243,
    "endpoints": {
        "1": {
            "profId": 260,
            "epId": 1,
            "devId": 24321,
            "inClusterList": [
                0,
                6,
                65535
            ],
            "outClusterList": [
                0,
                4,
                65535
            ],
            "clusters": {}
        }
    },
    "_id": "EOq489ppjiB5fRkY"
}
{
    "id": 2,
    "type": "EndDevice",
    "ieeeAddr": "0x00158d0001a67780",
    "nwkAddr": 64522,
    "manufId": 4151,
    "manufName": "LUMI",
    "powerSource": "Battery",
    "modelId": "lumi.sensor_switch.aq2",
    "epList": [
        1
    ],
    "status": "online",
    "joinTime": 1527937243,
    "endpoints": {
        "1": {
            "profId": 260,
            "epId": 1,
            "devId": 24321,
            "inClusterList": [
                0,
                6,
                65535
            ],
            "outClusterList": [
                0,
                4,
                65535
            ],
            "clusters": {}
        }
    },
    "_id": "EOq489ppjiB5fRkY"
}
{
    "id": 2,
    "type": "EndDevice",
    "ieeeAddr": "0x00158d0001a67780",
    "nwkAddr": 64522,
    "manufId": 4151,
    "manufName": "LUMI",
    "powerSource": "Battery",
    "modelId": "lumi.sensor_switch.aq2",
    "epList": [
        1
    ],
    "status": "online",
    "joinTime": 1527937243,
    "endpoints": {
        "1": {
            "profId": 260,
            "epId": 1,
            "devId": 24321,
            "inClusterList": [
                0,
                6,
                65535
            ],
            "outClusterList": [
                0,
                4,
                65535
            ],
            "clusters": {
                "genBasic": {
                    "dir": {
                        "value": 3
                    },
                    "attrs": {}
                },
                "genGroups": {
                    "dir": {
                        "value": 2
                    },
                    "attrs": {}
                },
                "genOnOff": {
                    "dir": {
                        "value": 1
                    },
                    "attrs": {}
                },
                "manuSpecificCluster": {
                    "dir": {
                        "value": 3
                    },
                    "attrs": {}
                }
            }
        }
    },
    "_id": "EOq489ppjiB5fRkY"
}

[tb-killa]

I see that my entries are also duplicated in the database.db

[Koenkk]

Yes, note that the database.db is a file from zigbee-shepherd, not from zigbee2mqtt.

[ciotlosm]

@Koenkk Happy to close the issue if this is not an issue. I was just trying to make sure this is not some bug inside permit_join on the fly.

Glad to hear it's expected. I guess the removing a paired device (#35) is required if users end up needing to clear things from this db to fix issues .

[lolorc]

weird, so this is not an issue to have duplicates in the database.db

wc -l database.db => **37**
sort -u database.db | wc -l => **20**

17 duplicates, i guess I'll remove them next time I restart the daemon.

[lolorc]

@Koenkk @ciotlosm
looks like an issue to me.
a database should not have duplicate entries.
and in my case I've also have devices with duplicate entries...
for instance some device appears several time, sometimes with joinTime:null and also with joinTime:somevalue

[Koenkk]

No it's is not an issue, this is how the underlying zigbee library (zigbee-shepherd) handles it database. I agree that it is not a nice solution.

Note that this file is not from zigbee2mqtt.

[lolorc]

Ok, I guess I'll write a simple script to clean it up before starting the daemon. I haven't read the code, so blindly I would expect it to be sub-optimal for zigbee-shepherd to have so many duplicates

[lolorc]

relevant issue zigbeer/zigbee-shepherd#39
solution might as well come from ioBroker/ioBroker.zigbee#14