chore: use graceful timeout mechanism #1841
Conversation
size-limit report 📦
|
|
|
||
| export function withGracefulTimeout( | ||
| asyncOperation: () => Promise<void>, | ||
| timeoutDuration: number, |
There was a problem hiding this comment.
let's make it same as MOCHA_HOOK_MAX_TIMEOUT global config and optional to pass
| import { Logger } from "@waku/utils"; | ||
| const log = new Logger("test:mocha-hook"); | ||
|
|
||
| export function withGracefulTimeout( |
There was a problem hiding this comment.
to decrease amount of lines changed - I want to propose following:
export const beforeEachCustom = function (self, cb: Promise<void>, setter: any) {
self.beforeEach((done) => {
this.timeout(MOCHA_HOOK_MAX_TIMEOUT);
withGracefulTimeout(async () => {
const [nwaku, waku] = await runMultipleNodes(
this,
[DefaultPubsubTopic],
strictCheckNodes
);
subscription = await waku.filter.createSubscription();
setter({ nwaku, waku/*, nwaku1, ...etc */});
}, 20000, done);
});
});do you think it would work, @fbarbu15 ?
There was a problem hiding this comment.
I did made it work but there are multiple functions that we need to wrap:
runMultipleNodes
runNodes (2 of them)
tearDownNodes
teardownNodesWithRedundancy
and direct usage of
nwaku.start + other commands like subscribe
So we would probably need to create multiple customFunctions (we will complicate stuff even further) or unify the mentioned methods to have only one general runNodes function and only one general tearDownNodes function (this one looks like a big refactoring but it may be worth it)
Wdyt?
There was a problem hiding this comment.
I think this is solvable the same way:
in utils file:
export const beforeEachCustom = function (self, cb: Promise<void>) {
self.beforeEach((done) => {
this.timeout(MOCHA_HOOK_MAX_TIMEOUT);
withGracefulTimeout(cb, 20000, done);
});
});
export const afterEachCustom = function (self, cb: Promise<void>, setter: any) {
self.afterEach((done) => {
this.timeout(MOCHA_HOOK_MAX_TIMEOUT);
withGracefulTimeout(cb, 20000, done);
});
});in test file:
describe("test", () => {
let nwaku, waku;
beforeEachCustom(async () => {
[nwaku, waku] = await runMultipleNodes(
this,
[DefaultPubsubTopic],
strictCheckNodes
);
subscription = await waku.filter.createSubscription();
});
afterEachCustom(async () => {
await teardownNodesWithRedundancy(serviceNodes, waku);
});
test("test case", () => {});
});There was a problem hiding this comment.
worked with some change regarding global timeout restoring
…ocha-hooks-timeouts
…ocha-hooks-timeouts
Problem
Looking at the flaky tests from the last months it seems most are failing at beforeEach and afterEach hooks, usually with a Timeout error when staring or stopping the nwaku container.
After investigating it seems that mocha retry mechanism only retries the test if the failure was in the test itself (inside it ) and it fails immediately, without any retry, if the failure is in those beforeEach/afterEach hooks.
Solution
To address those failures I've increased the timeouts for all beforeEach and afterEach hooks
Also added a function to gracefully handle timeouts in the staring/stopping of the nwaku containers, and making the hook pass in such cases, so the failure gets detected in the test itself and hence triggering the retry mechanism.