Some nodes are not waking up within 24 hours

[scottyeager]

Since the update to v0.2.2, farmers using the farmerbot have reported that some of their nodes are not waking up within 24 hours of going standby.

We'll update this issue with farmerbot logs as they come in.

[scottyeager]

I have reviewed relevant log files and have some more information about what's happening in the case of this issue. The general sequence of events is like this:

1. Farmerbot initiates a random wakeup for a node shortly before its next scheduled periodic wakeup (within one hour in the examples I see)
2. The randomly awoken node is put back to sleep very close to the periodic wakeup time
3. The node is not awoken again for periodic wakeup
4. On the next day, the node wakes up more than 24 hours after it went to sleep, because it previously went to sleep earlier than it would have during a normal periodic wakeup

Here are two examples with relevant lines extracted from the farmerbot log files.

Node 5248

Log file: Colossus06.log.gz

2023-09-19 05:26:55 [INFO ] [POWERMANAGER] Random wakeup for node 5248

2023-09-19 05:41:55 [INFO ] [DATAMANAGER] Node 5248 is ON.

2023-09-19 06:17:54 [INFO ] [POWERMANAGER] Resource usage too low: 0. Turning off unused node 5248

2023-09-19 06:21:55 [INFO ] [DATAMANAGER] Node 5248 shutdown was successful.

2023-09-20 06:17:06 [INFO ] [POWERMANAGER] Periodic wakeup for node 5248

2023-09-20 06:31:56 [INFO ] [DATAMANAGER] Node 5248 is ON.

(Resource usage too low log message missing)

2023-09-20 07:11:56 [INFO ] [DATAMANAGER] Node 5248 shutdown was successful.

Note how the shutdown after random wakeup happens almost an hour before the shutdown for regular periodic wakeup. Then when the nodes comes back online for its periodic wakeup, it happens about ten minutes too late for the 24 hour cutoff (6:31 vs 6:21).

Node 4486

Log file: Colossus02.log.gz

2023-09-13 08:26:45 [INFO ] [POWERMANAGER] Random wakeup for node 4486

2023-09-13 08:46:46 [INFO ] [DATAMANAGER] Node 4486 is ON.

2023-09-13 09:17:42 [INFO ] [POWERMANAGER] Resource usage too low: 0. Turning off unused node 4486

2023-09-13 09:21:45 [INFO ] [DATAMANAGER] Node 4486 shutdown was successful.

2023-09-14 09:21:54 [INFO ] [POWERMANAGER] Periodic wakeup for node 4486

2023-09-14 09:41:48 [INFO ] [DATAMANAGER] Node 4486 is ON.

2023-09-14 10:16:54 [INFO ] [POWERMANAGER] Resource usage too low: 0. Turning off unused node 4486

2023-09-14 10:21:47 [INFO ] [DATAMANAGER] Node 4486 shutdown was successful.

Timing here is very similar to before. In this case, the node wakes up 20 minutes too late.

Suggested Fix

The issue here seems to be that nodes which have a random wakeup just before periodic wakeup time are not triggered for a periodic wakeup.

To me, the most logical fix is to just not schedule random wakeups so close to the periodic wakeup time. Another approach would be to keep the nodes awake so that they go to sleep at the same time as they would at a periodic wakeup.

[MarioBassem]

Thanks @scottyeager for the detailed issue.
i figured that, as you said, the nodes don't go through the periodic wakeup because they had just been woken up by a random wakeup. That being said, i don't see the reason why we want to explicitly avoid this, especially if the wakeups are too close. is there any??

[scottyeager]

The problem is that nodes will miss out on minting if they don't wake up within 24 hours of going to sleep (currently there is a one hour grace period so nodes in these examples will mint normally, but we want to remove that soon).

Farmerbot behavior needs to change such that nodes will always stay awake for the set duration after periodic wakeup time, no matter what happens with random wakeups.

I see two ways to accomplish this:

1. Don't do random wakeups at times that cause periodic wakeups to be canceled
2. If the random wakeup comes just before the periodic wakeup, keep the node awake until the time it would usually go to sleep (put it to sleep with the rest of the nodes that are