record of one hour of messages

[8ohamed]

I have added the folder conaining 1 hour of data for 2 topics, and a script to publish them. So we can run the experiments without changes, except for the mqtt config. I have changed the production.json.template accordingly.
This branch and the "pre-data" branch shouldn't be merged into main.

[prasadtalasila]

@8ohamed I first replayed the recorded values

....
[data1.jsonl] → cpsens/recorded/1/data (len=92)
[data2.jsonl] → cpsens/recorded/2/data (len=92)
[data1.jsonl] → cpsens/recorded/1/data (len=92)
[data2.jsonl] → cpsens/recorded/2/data (len=92)
.....

and then I used the following config and the experiments fail

{
    "MQTT": {
        "host": "mqtt.eclipseprojects.io",
        "port": 1883,
        "userId": "",
        "password": "",
        "ClientID": "test_client_id",
        "QoS": 1,
        "TopicsToSubscribe": ["cpsens/recorded/1/data", "cpsens/recorded/2/data"]
    }
}

The tracelog of the failure is:

(.venv) $ python .\src\examples\example.py --config .\config\replay.json accelerometers
JSON configuration loaded successfully.
on_connect: Connected with response code Success
Subscribing to topic: cpsens/recorded/1/data
on_subscribe: Subscription ID 1 with QoS levels [ReasonCode(Suback, 'Granted QoS 1')]
Data requested []

(.venv) $python .\src\examples\example.py --config .\config\replay.json align-readings
JSON configuration loaded successfully.
Traceback (most recent call last):
  File "C:\Users\foo\git\example-shm\src\examples\example.py", line 76, in <module>
    cli(obj={})
  File "C:\Users\foo\git\example-shm\.venv\Lib\site-packages\click\core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\foo\git\example-shm\.venv\Lib\site-packages\click\core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\Users\foo\git\example-shm\.venv\Lib\site-packages\click\core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\foo\git\example-shm\.venv\Lib\site-packages\click\core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\foo\git\example-shm\.venv\Lib\site-packages\click\core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\foo\git\example-shm\.venv\Lib\site-packages\click\decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\foo\git\example-shm\src\examples\example.py", line 35, in align_readings
    align_acceleration_readings(ctx.obj["CONFIG"])
  File "C:\Users\foo\git\example-shm\src\examples\aligning_readings.py", line 13, in align_acceleration_readings
    selected_topics = [all_topics[i] for i in topic_indexes]
                       ~~~~~~~~~~^^^
IndexError: list index out of range

[prasadtalasila]

@8ohamed The following error comes

$poetry run python src/examples/example.py model-update-remote-sysid
JSON configuration loaded successfully.
JSON configuration loaded successfully.
Waiting for OMA data...
Failed to connect to MQTT broker. Code: Not authorized
Failed to connect to MQTT broker. Code: Not authorized
Failed to connect to MQTT broker. Code: Not authorized

this comes only when MQTT authentication is required. For replay scenario, the test.mosquitto.org does not require authentication and it works well. The problem appears only when authentication is required.

[prasadtalasila]

There is a deprecated API in record.py

python .\record\record.py
C:\Users\foo\git\example-shm\record\record.py:54: DeprecationWarning: Callback API version 1 is deprecated, update to latest version
  client = mqtt.Client(client_id=MQTT_CONFIG["ClientID"], protocol=mqtt.MQTTv311)
Recording for 30 seconds...
Connected with result code 0
Subscribed to cpsens/d8-3a-dd-37-d2-7e/3160-A-042_sn_999998/1/acc/raw/data
Subscribed to cpsens/d8-3a-dd-37-d2-7e/3160-A-042_sn_999998/1/acc/raw/metadata
Subscribed to cpsens/d8-3a-dd-37-d2-7e/3160-A-042_sn_999998/2/acc/raw/data
C:\Users\foo\git\example-shm\record\record.py:42: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
  timestamp = datetime.utcnow().isoformat()
Recording complete.

[8ohamed]

Hi @prasadtalasila the API warning is resolved.
And for the MQTT authentication error, is this poetry run python src/examples/example.py oma-and-publish running (on a different terminal) and poetry run python src/examples/example.py model-update-remote-sysid on another, where both use the same config.
So in total you need to run 3 terminals/machines.

1. for replaying the MQTT data 2. to get those data and publish OMA results to another config 3. To get the OMA and run model update

[prasadtalasila]

Hi @prasadtalasila the API warning is resolved.
And for the MQTT authentication error, is this poetry run python src/examples/example.py oma-and-publish running (on a different terminal) and poetry run python src/examples/example.py model-update-remote-sysid on another, where both use the same config.
So in total you need to run 3 terminals/machines.

1. for replaying the MQTT data 2. to get those data and publish OMA results to another config 3. To get the OMA and run model update

@8ohamed the code is working well for the replay scenario. The replay scenario is using test MQTT server without any credentials.

The problem comes when I run the code with credentials. I tried running the distributed code with HBK stream and then the authentication failure happens.

[8ohamed]

@prasadtalasila Then you should only run poetry run python src/examples/example.py oma-and-publish
and poetry run python src/examples/example.py model-update-remote-sysid

You also need to make sure of this section in the config used
"sysID":{
"host": "test.mosquitto.org",
"port": 1883,
"userId": "",
"password": "",
"ClientID": "TOTOT3232_id",
"QoS": 1,
"TopicsToSubscribe": [
"cpsens/d8-3a-dd-f5-92-48/cpsns_Simulator/1_2/oma_results"
]