HIL: adjust OTA test timing #483
Conversation
Use semaphores to trigger tests and decrease loop delay from 1000ms to 100ms. Signed-off-by: Mike Szczys <mike@golioth.io>
When testing reason and state codes, propagation delay from the device to the server were causing erroneous test failures. This increases the delay before checking the updated server value, and also increases the device delay between sending reason/state updates. The marginal difference is 2s per reason code, or 20 seconds total increase in test time. Signed-off-by: Mike Szczys <mike@golioth.io>
|
Visit the preview URL for this PR (updated for commit 4f1e82f): https://golioth-firmware-sdk-doxygen-dev--pr483-szczys-ota-hil-7xfm26tj.web.app (expires Tue, 14 May 2024 16:46:10 GMT) 🔥 via Firebase Hosting GitHub Action 🌎 Sign: a9993e61697a3983f3479e468bcb0b616f9a0578 |
Code Coverage
|
|
Have you seen tests fail at the margins of these timeouts? Or is this more of a guess and check? |
|
More of a guess and check. I have not been able to observe a status update sent to the server fail to update when I check manually so I am guessing that the 1.5s timeout pytest uses from seeing the log message to running the API call is the culprit. I doubled that to 3 seconds and built in some space for that in the C code timing. I'd prefer not to bank on these timings, but the alternative that comes to mind is introducing an RPC to tell the device to go to the next status. That doesn't seem like a good idea. |
Adjust OTA test timing to reduce tests that fail due to propagation delay.
https://github.com/golioth/firmware-issue-tracker/issues/568