Random crashes due to possible bug in WaitTracker.ts #6624

nmf2 · 2023-07-07T20:40:23Z

Describe the bug
Our self-hosted main n8n instance randomly crashes with the following error trace:

2023-07-07T18:24:33.059Z | debug    | Wait tracker found 2 executions. Setting timer for IDs: 4692368, 4692374 "{ file: 'WaitTracker.js', function: 'getWaitingExecutions' }"

/usr/local/lib/node_modules/n8n/dist/WaitTracker.js:82
                const triggerTime = execution.waitTill.getTime() - new Date().getTime();
                                                       ^
TypeError: execution.waitTill.getTime is not a function
    at WaitTracker.getWaitingExecutions (/usr/local/lib/node_modules/n8n/dist/WaitTracker.js:82:56)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

To Reproduce
Steps to reproduce the behavior:
No idea yet... it crashes randomly with that error above.

Environment (please complete the following information):

OS: Using official n8n docker image n8nio/n8n:0.233.1
n8n Version Using official n8n docker image n8nio/n8n:0.233.1
Node.js Version: Using official n8n docker image n8nio/n8n:0.233.1
Database system Postgres
Operation mode queue, 3 workers, 1 main instance.

Additional context
I tracked the error to this line of n8n's code:

n8n/packages/cli/src/WaitTracker.ts

Line 89 in bf35124

const triggerTime = execution.waitTill!.getTime() - new Date().getTime();

I'm not sure what else to provide you with to help solve the issue so feel free to ask!

The text was updated successfully, but these errors were encountered:

netroy · 2023-07-07T20:46:03Z

@nmf2 Do you have (or could you please create) a small workflow to reproduce this?
That'd help a lot with fixing this quickly.

nmf2 · 2023-07-07T20:51:52Z

@netroy, thanks for the quick response!

Sincerely I wish I could provide a workflow but this is not really something that I even know how to reproduce. It just happens randomly and it doesn't affect the executions of a specific workflow. I do think it has to do with the wait node but only because of where the error happens in the code...

[Edit]:
Given how the following code is written, the waitTill property should always have the getTime() function but that's not happening in reality:

n8n/packages/cli/src/WaitTracker.ts

Line 89 in bf35124

const triggerTime = execution.waitTill!.getTime() - new Date().getTime();

netroy · 2023-07-08T21:30:09Z

@nmf2 are you using the webhook option in the wait node ?

nmf2 · 2023-07-09T13:17:14Z

@netroy we don't use that in any of our workflows, but we do use the Wait node with a long delay (days), so the execution goes into the waiting status.

Hope this helps.

How can I be of further assistance?

DRIMOL · 2023-07-26T20:33:40Z

this happens to me, also only if the wait has more than 65 seconds, I'm currently in version 1.0.5 but, I believe that in version 0 it was also happening, I'll leave a simple flow here so you can test it.

{
  "meta": {
    "instanceId": "33738330930e3881dd5571eca013f36ddf8aab20e4ea5c1f2ebaf4a2b4668ac6"
  },
  "nodes": [
    {
      "parameters": {
        "values": {
          "string": [
            {}
          ]
        },
        "options": {}
      },
      "id": "9f467564-55b1-44c3-93d2-a7aec275a713",
      "name": "Set",
      "type": "n8n-nodes-base.set",
      "typeVersion": 2,
      "position": [
        340,
        400
      ]
    },
    {
      "parameters": {
        "amount": 10,
        "unit": "seconds"
      },
      "id": "ea2b1005-2a2c-4866-abd4-2cafa6893ee5",
      "name": "Wait",
      "type": "n8n-nodes-base.wait",
      "typeVersion": 1,
      "position": [
        560,
        400
      ],
      "webhookId": "afe8deba-8749-4d3c-ae8f-c0887ba05770"
    },
    {
      "parameters": {
        "values": {
          "string": [
            {}
          ]
        },
        "options": {}
      },
      "id": "5c89b28b-523d-409f-8720-92a3ed8379ed",
      "name": "Set1",
      "type": "n8n-nodes-base.set",
      "typeVersion": 2,
      "position": [
        740,
        400
      ]
    },
    {
      "parameters": {
        "amount": 70,
        "unit": "seconds"
      },
      "id": "4fc941ff-4069-4aec-8549-f3c4d449c96f",
      "name": "Wait1",
      "type": "n8n-nodes-base.wait",
      "typeVersion": 1,
      "position": [
        920,
        400
      ],
      "webhookId": "a0f3ebfb-8d6d-4711-958a-0c90438ce3c6"
    },
    {
      "parameters": {
        "httpMethod": "POST",
        "path": "d7541d94-e26f-4495-9299-13f3055c9688",
        "options": {}
      },
      "id": "3736585e-2f82-4630-870c-a87efd30cf7c",
      "name": "Webhook",
      "type": "n8n-nodes-base.webhook",
      "typeVersion": 1,
      "position": [
        160,
        400
      ],
      "webhookId": "d7541d94-e26f-4495-9299-13f3055c9688"
    },
    {
      "parameters": {},
      "id": "4941ce6a-a3f5-4498-b50c-8eb7beb40c6c",
      "name": "No Operation, do nothing",
      "type": "n8n-nodes-base.noOp",
      "typeVersion": 1,
      "position": [
        1140,
        400
      ]
    }
  ],
  "connections": {
    "Set": {
      "main": [
        [
          {
            "node": "Wait",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Wait": {
      "main": [
        [
          {
            "node": "Set1",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Set1": {
      "main": [
        [
          {
            "node": "Wait1",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Wait1": {
      "main": [
        [
          {
            "node": "No Operation, do nothing",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Webhook": {
      "main": [
        [
          {
            "node": "Set",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  }
}

also here is a screenshot

netroy · 2023-07-28T08:03:45Z

so far I'm unable to reproduce this on regular mode and queue mode, with the workflow linked above, and with a bunch of different other workflows with wait nodes with wait times longer than 65 seconds.

stwonary · 2023-08-10T14:59:01Z

Hi. we are having the same problem. our waiting time is 3 minutes.

possible log trace relate to this issue:

(node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 global:completed listeners added to [Queue]. Use emitter.setMaxListeners() to increase limit
(Use `node --trace-warnings ...` to show where the warning was created)
(node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 global:failed listeners added to [Queue]. Use emitter.setMaxListeners() to increase limit
(node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 global:completed listeners added to [Queue]. Use emitter.setMaxListeners() to increase limit
(Use `node --trace-warnings ...` to show where the warning was created)
(node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 global:failed listeners added to [Queue]. Use emitter.setMaxListeners() to increase limit
This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
TypeError: execution.waitTill.getTime is not a function
    at WaitTracker.getWaitingExecutions (/usr/local/lib/node_modules/n8n/dist/WaitTracker.js:83:56)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
TypeError: execution.waitTill.getTime is not a function
    at WaitTracker.getWaitingExecutions (/usr/local/lib/node_modules/n8n/dist/WaitTracker.js:83:56)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
TypeError: execution.waitTill.getTime is not a function
    at WaitTracker.getWaitingExecutions (/usr/local/lib/node_modules/n8n/dist/WaitTracker.js:83:56)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
TypeError: execution.waitTill.getTime is not a function
    at WaitTracker.getWaitingExecutions (/usr/local/lib/node_modules/n8n/dist/WaitTracker.js:83:56)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
TypeError: execution.waitTill.getTime is not a function
    at WaitTracker.getWaitingExecutions (/usr/local/lib/node_modules/n8n/dist/WaitTracker.js:83:56)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
TypeError: execution.waitTill.getTime is not a function
    at WaitTracker.getWaitingExecutions (/usr/local/lib/node_modules/n8n/dist/WaitTracker.js:83:56)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
TypeError: execution.waitTill.getTime is not a function
    at WaitTracker.getWaitingExecutions (/usr/local/lib/node_modules/n8n/dist/WaitTracker.js:83:56)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
TypeError: execution.waitTill.getTime is not a function
    at WaitTracker.getWaitingExecutions (/usr/local/lib/node_modules/n8n/dist/WaitTracker.js:83:56)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
TypeError: execution.waitTill.getTime is not a function

stwonary · 2023-08-10T18:34:49Z

found something interesting

n8n/packages/cli/src/WaitTracker.ts

Lines 53 to 61 in bf35124

    
           const findQuery: FindManyOptions<ExecutionEntity> = { 
        
           	select: ['id', 'waitTill'], 
        
           	where: { 
        
           		waitTill: LessThanOrEqual(new Date(Date.now() + 70000)), 
        
           		status: Not('crashed'), 
        
           	}, 
        
           	order: { 
        
           		waitTill: 'ASC', 
        
           	},

Here, you can see some instances of waitTill with null values. These are causing the crashes.

@netroy could you please release an hotfix for 0.236.3 ? We are still not ready to migrate to 1.x version

stwonary · 2023-08-10T19:09:20Z

maybe update using this code for the meantime?

for (const execution of executions) {
  const executionId = execution.id;
  if (this.waitingExecutions[executionId] === undefined) {
    if (execution.waitTill instanceof Date) {
      const triggerTime = execution.waitTill.getTime() - new Date().getTime();
      this.waitingExecutions[executionId] = {
        executionId,
        timer: setTimeout(() => {
          this.startExecution(executionId);
        }, triggerTime),
      };
    } else {
      Logger.error(`execution.waitTill is not a Date object for execution ID: ${executionId}. Value: ${execution.waitTill}`);
    }
  }
}

netroy · 2023-10-17T15:28:57Z

I think this was probably being caused by time-zone issues between n8n and postgres. If that's the case, This issue should be resolved in n8n versions 1.9.0 or later.
Can any of you upgrade n8n and check if this issues is fixed for you?

netroy · 2023-10-17T15:56:02Z

If you can't upgrade to 1.x, I've created a custom docker image with all the postgres related fixes: n8nio/n8n:fix-0.237.

netroy · 2023-10-18T16:06:34Z

One of the customers facing this issue has confirmed that the custom image, as well as upgrading fixes this issue.
If anyone is still seeing this issue on versions after 1.9, please let us know, and we can re-open this ticket to investigate further.

caiquezanetoni · 2023-10-27T16:15:18Z

@netroy
I have this issue in version 1.11.2,
this only happens when using a Postgres node on the latest version

caiquezanetoni · 2023-10-27T16:19:44Z

@netroy

netroy · 2023-10-30T11:28:05Z

@caiquezanetoni Can you please share with us:

what version of postgres is n8n database on (not the version of postgres the postgres node is talking to)?
what timezone you are in?
have you customized the timezone for postgres or n8n, or are they picking up the system defaults?

janober · 2024-01-17T14:00:53Z

Fix got released with n8n@1.25.0

DRIMOL · 2024-02-06T14:12:16Z

This was only resolved for regular n8n, for those who have n8n queue with postgres it is not possible to update to version 1.25.1

https://community.n8n.io/t/n8n-worker-get-error-n8n-encryption-key-var-after-update-new-version/35878/3
https://community.n8n.io/t/error-when-upgrading-to-1-25-1/39631

netroy · 2024-02-16T09:30:27Z

@DRIMOL The links you posted are related to the encryption-key mismatch, and not related to the WaitTracker (which this issue is for).

netroy · 2024-02-16T09:30:41Z

For anyone still seeing this error, please upgrade to 1.29.1, which includes a fix for a bug in the postgresql driver where the timestamps were not converted to a valid Date object sometimes.

Joffcom mentioned this issue Jul 13, 2023

BUG: execution.waitTill.getTime is not a function #6655

Closed

This was referenced Aug 10, 2023

Fix date check in waittracker #6912

Closed

Fix date check in waittracker #6913

Closed

netroy closed this as completed Oct 18, 2023

arthurrferroni mentioned this issue Dec 22, 2023

Waiting Executions #8136

Closed

This was referenced Jan 12, 2024

fix: Report when waitTill is invalid and handle it #8314

Closed

fix(core): Report when waitTill is invalid and handle it #8356

Merged

janober added the Released label Jan 17, 2024

This issue was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Random crashes due to possible bug in WaitTracker.ts #6624

Random crashes due to possible bug in WaitTracker.ts #6624

nmf2 commented Jul 7, 2023

netroy commented Jul 7, 2023

nmf2 commented Jul 7, 2023 •

edited

Loading

netroy commented Jul 8, 2023

nmf2 commented Jul 9, 2023

DRIMOL commented Jul 26, 2023 •

edited by netroy

Loading

netroy commented Jul 28, 2023

stwonary commented Aug 10, 2023

stwonary commented Aug 10, 2023

stwonary commented Aug 10, 2023

netroy commented Oct 17, 2023

netroy commented Oct 17, 2023

netroy commented Oct 18, 2023

caiquezanetoni commented Oct 27, 2023

caiquezanetoni commented Oct 27, 2023

netroy commented Oct 30, 2023

janober commented Jan 17, 2024

DRIMOL commented Feb 6, 2024

netroy commented Feb 16, 2024

netroy commented Feb 16, 2024

Random crashes due to possible bug in WaitTracker.ts #6624

Random crashes due to possible bug in WaitTracker.ts #6624

Comments

nmf2 commented Jul 7, 2023

netroy commented Jul 7, 2023

nmf2 commented Jul 7, 2023 • edited Loading

netroy commented Jul 8, 2023

nmf2 commented Jul 9, 2023

DRIMOL commented Jul 26, 2023 • edited by netroy Loading

netroy commented Jul 28, 2023

stwonary commented Aug 10, 2023

stwonary commented Aug 10, 2023

stwonary commented Aug 10, 2023

netroy commented Oct 17, 2023

netroy commented Oct 17, 2023

netroy commented Oct 18, 2023

caiquezanetoni commented Oct 27, 2023

caiquezanetoni commented Oct 27, 2023

netroy commented Oct 30, 2023

janober commented Jan 17, 2024

DRIMOL commented Feb 6, 2024

netroy commented Feb 16, 2024

netroy commented Feb 16, 2024

nmf2 commented Jul 7, 2023 •

edited

Loading

DRIMOL commented Jul 26, 2023 •

edited by netroy

Loading