Add watchdog for watchers #106
Conversation
ghost
assigned 
|
Please change the target branch to |
.env.example
Outdated
| @@ -28,6 +28,8 @@ REDIS_LOCK_TTL=1000 | |||
| HOME_START_BLOCK= | |||
| FOREIGN_START_BLOCK= | |||
|
|
|||
| MAX_PROCESSING_TIME=600000 | |||
There was a problem hiding this comment.
Is my understanding correct and 10 min is too much for the watchdog timer? Could you estimate max time expected to handle one iteration of watcher?
There was a problem hiding this comment.
I guess it depends on the number of events processed at the same time. I will try sending several transactions to the kovan-sokol bridge and see how much time is spent processing them.
There was a problem hiding this comment.
Sent 10 transactions to Sokol. 2 of them were processed in one block, 8 in another. The time difference between the first log, and the log of the sender receiving the last transactions to send was 3 seconds (that is Sokol's block time, isn't it?).
The time is going to depend on the chain, but I guess one minute should be enough in the vast majority of cases (and it can be configured, anyway).
There was a problem hiding this comment.
Hmm. I think the timout should be comparable with the polling time. I think 3 or 4 times more than the polling time of the corresponding network. What do you think?
There was a problem hiding this comment.
Yes, that sounds reasonable. You mean using that as the example, or that the timeout should be computed from the polling time?
There was a problem hiding this comment.
Yes, let's set the timeout in the example to be 4 times more that the longest polling time and provide these recommendations in README.
|
I believe we need to expanse these changes on the senders as well. What do you think? |
Sure. They didn't cause issues yet, but it wouldn't hurt to have it there too. |
src/sender.js
Outdated
| @@ -32,6 +33,8 @@ const nonceLock = `lock:${config.id}:nonce` | |||
| const nonceKey = `${config.id}:nonce` | |||
| let chainId = 0 | |||
|
|
|||
| const maxProcessingTime = process.env.MAX_PROCESSING_TIME | |||
There was a problem hiding this comment.
We need to define a default value which will be used if MAX_PROCESSING_TIME was missed by some reason in the configuration file.
There was a problem hiding this comment.
This was intentional, to allow not setting the watchdog. If you think it should always be enabled, we can add a default value here.
There was a problem hiding this comment.
What risks can you see if we enable the watchdog by default? In this case the user could disable the watchdog by zeroing the configuration parameter explicitly.
There was a problem hiding this comment.
That is a bit complicated. Javascript treats 0 as a falsy value, so doing 0 || 10 evaluates to 10.
On the other hand, the environment variables are strings, so '0' || 10 does evaluate to '0', but it's not good to rely on that; we could easily "break" it if we add environment variables validation and casting, like we did in poa-bridge-contracts/deploy.
A possible solution is to use a negative value to disable it. That will work both with strings and with numbers: '-1' < 0 is equivalent to -1 < 0.
There was a problem hiding this comment.
I do not understand this.
Could you use something like:
if (process.env.MAX_PROCESSING_TIME === '0') {
...
} else {
...
}
And later, when you add environment variables validation and casting you will change this.
There was a problem hiding this comment.
That can work. This is what I did (see f38688c):
- If value is explicitly set to 0, the watchdog is not used.
- If the value is not set, it is computed using 4 * maxPollingTime
- Otherwise, the set value is used
I didn't want to add this logic twice (in the watcher and the sender), so I moved it to the config. And since the senders weren't including the base config, I also added it to them.
What do you think?
src/watcher.js
Outdated
| @@ -30,6 +31,8 @@ const eventContract = new web3Instance.eth.Contract(config.eventAbi, config.even | |||
| const lastBlockRedisKey = `${config.id}:lastProcessedBlock` | |||
| let lastProcessedBlock = BN.max(config.startBlock.sub(ONE), ZERO) | |||
|
|
|||
| const maxProcessingTime = process.env.MAX_PROCESSING_TIME | |||
There was a problem hiding this comment.
We need to define a default value which will be used if MAX_PROCESSING_TIME was missed by some reason in the configuration file.
|
Merge the branch |
Closes #105.
The environment variable used for the max time is
MAX_PROCESSING_TIME. If it's not set, the watchdog is not used.I also moved the exit codes to the constants files, so that we know what each of them means.