Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

n8n workers wont connect to main node. #8767

Closed
shankarmn94 opened this issue Feb 29, 2024 · 18 comments
Closed

n8n workers wont connect to main node. #8767

shankarmn94 opened this issue Feb 29, 2024 · 18 comments

Comments

@shankarmn94
Copy link

shankarmn94 commented Feb 29, 2024 •
edited
Loading

Bug Description

server 1

WEBHOOK_URL=http://10.102.0.5:5678
DB_TYPE=postgresdb
DB_POSTGRESDB_DATABASE=workflow
DB_POSTGRESDB_HOST=10.102.0.5
DB_POSTGRESDB_PORT=5432
DB_POSTGRESDB_USER=workflow
DB_POSTGRESDB_PASSWORD=workflow
N8N_PAYLOAD_SIZE_MAX=56MB
EXECUTIONS_DATA_PRUNE=true
EXECUTIONS_DATA_PRUNE_MAX_COUNT=50000
EXECUTIONS_DATA_SAVE_ON_ERROR=all
EXECUTIONS_DATA_SAVE_ON_SUCCESS=none
NODE_OPTIONS="--max-old-space-size=16048 --require /root/.n8n/set-max-listeners.js"
N8N_ENCRYPTION_KEY=DAWJBDAWIGBCAIUCNALA
#EXECUTIONS_MODE=queue
QUEUE_BULL_REDIS_HOST=10.102.0.3

server 2 - same env

server 1 i started n8n using of npm as in

cd ~/.n8n
here the .env will be there.

/usr/bin/n8n start

User settings loaded from: /root/.n8n/config
Initializing n8n process
n8n ready on 0.0.0.0, port 5678
Version: 1.27.3

Start Active Workflows:

  • "Azure AD" (ID: hBRHsi9IThId9Nof)
    => Started
  • "Notification Schedule" (ID: FACAS)CAC
    => Started
  • "Notification" (ID: QWDWDA)
    => Started
  • "Notification Ticket for Manual Ticket" (ID: AFAWFWFWA)
    => Started

Editor is now accessible via:
http://localhost:5678/

in server 2 with same folder and env
cd ~/.n8n
/usr/bin/n8n worker.

it will start as

n8n worker
User settings loaded from: /root/.n8n/config

n8n worker is now ready

  • Version: 1.27.3
  • Concurrency: 10

when i start a workflow it will will run in only server (server 1) and wont come to other server.

if i enable quemode it will give error like

ReplyError: ERR Error running script (call to 8f55ae4a3be429c6d38c5d5db3e80edf89197b64): @user_script:56: script tried accessing undeclared key

how can i handle and resolve this

To Reproduce

same settings as above

Expected behavior

workers should get running or how can i start multiple servers as n8n start for same db..

Operating System

ubuntu 22.04

n8n Version

1.27.3

Node.js Version

v20.10.0

Database

postgresql

Execution mode

main (default) and queue mode tried
@Joffcom
Copy link
Member

Joffcom commented Feb 29, 2024

@shankarmn94 this doesn't appear to be a bug and looks to probably be around your configuration.

For general support I would recommend posting on the community support forum.

To start with this make sure you have followed our documentation on setting up queue mode (both all instances need to be set to run in queue) it would also be handy to know how you are triggering your workflow, if you are doing it from the UI then I would expect the main n8n instance to process the job but if it is a webhook or a schedule the worker should take it.

@shankarmn94
Copy link
Author

shankarmn94 commented Feb 29, 2024 •
edited
Loading

could you just share that please i got confused after trying these que mode with different configs.

https://docs.n8n.io/hosting/scaling/queue-mode/#set-encryption-key

this is what i tried.

@shankarmn94
Copy link
Author

@Joffcom

@shankarmn94
Copy link
Author

sometimes we are getting context deadline exceeded.

@Joffcom
Copy link
Member

Joffcom commented Mar 1, 2024

@shankarmn94 Can you share the full output from the worker nodes when you have queue mode configured as documented?

@shankarmn94
Copy link
Author

shankarmn94 commented Mar 5, 2024 •
edited
Loading

WEBHOOK_URL=http://10.51.112.8:5678/
N8N_BASIC_AUTH_USER=cloudquen8n
N8N_BASIC_AUTH_PASSWORD=n8ncloudque
DB_TYPE=postgresdb
DB_POSTGRESDB_DATABASE=workflow
DB_POSTGRESDB_HOST=10.51.112.8
DB_POSTGRESDB_PORT=5432
DB_POSTGRESDB_USER=workflow
DB_POSTGRESDB_PASSWORD=workflow
#N8N_PAYLOAD_SIZE_MAX=56MB
EXECUTIONS_DATA_PRUNE=true
EXECUTIONS_DATA_PRUNE_MAX_COUNT=5000000
EXECUTIONS_DATA_SAVE_ON_ERROR=all
EXECUTIONS_DATA_SAVE_ON_SUCCESS=all
NODE_OPTIONS="--max-old-space-size=16048 --require /root/.n8n/set-max-listeners.js"
EXECUTIONS_MODE=queue
N8N_ENCRYPTION_KEY=Zwyi2FIVHTJEYb5+KY74esTPlKn4Vjru
QUEUE_BULL_REDIS_HOST=10.51.112.9
QUEUE_BULL_PREFIX=workflows
QUEUE_BULL_REDIS_DB=0
QUEUE_BULL_REDIS_PORT=6379

i kept this in two node.. one i started as main like n8n start and in other server n8n worker..

services will start..

worker wont get any request or take workflows...

if i trigger workflow in main nodes, below errror is mentioned

ReplyError: ERR Error running script (call to 8f55ae4a3be429c6d38c5d5db3e80edf89197b64): @user_script:56: script tried accessing undeclared key
ERR Error running script (call to 8f55ae4a3be429c6d38c5d5db3e80edf89197b64): @user_script:56: script tried accessing undeclared key

@shankarmn94
Copy link
Author

what is wrong with this.

@Joffcom
Copy link
Member

Joffcom commented Mar 5, 2024

Hey @shankarmn94,

Can you share the full output from the worker node when it is starting.

@shankarmn94
Copy link
Author

root@cs-dev-5:~/.n8n# /usr/bin/n8n start
User settings loaded from: /root/.n8n/config
Initializing n8n process
n8n ready on 0.0.0.0, port 5678
Version: 1.27.3

Start Active Workflows:

  • "Azure AD" (ID: tkMWYzIlDPXaEMsf)
    => Started
  • "Notification" (ID: UM4I7ViMzs0len3Q)
    => Started
  • "Notification Schedule" (ID: UtMRYFmttChIW7aO)
    => Started
  • "Notification for ticket template" (ID: bjvdGelpVJjzfzGC)
    => Started
  • "Notification manual ticket current flow" (ID: tKjOmB4MKwypv6CO)
    => Started
  • "Notification test schedule" (ID: rZNzKxFYsAGgsf9d)
    => Started

Editor is now accessible via:
http://localhost:5678/

Press "o" to open in Browser.
ReplyError: ERR Error running script (call to 8f55ae4a3be429c6d38c5d5db3e80edf89197b64): @user_script:56: script tried accessing undeclared key
ERR Error running script (call to 8f55ae4a3be429c6d38c5d5db3e80edf89197b64): @user_script:56: script tried accessing undeclared key

ReplyError: ERR Error running script (call to 8f55ae4a3be429c6d38c5d5db3e80edf89197b64): @user_script:56: script tried accessing undeclared key
ERR Error running script (call to 8f55ae4a3be429c6d38c5d5db3e80edf89197b64): @user_script:56: script tried accessing undeclared key
ReplyError: ERR Error running script (call to 8f55ae4a3be429c6d38c5d5db3e80edf89197b64): @user_script:56: script tried accessing undeclared key
ERR Error running script (call to 8f55ae4a3be429c6d38c5d5db3e80edf89197b64): @user_script:56: script tried accessing undeclared key

/usr/bin/n8n worker
User settings loaded from: /root/.n8n/config

n8n worker is now ready

  • Version: 1.27.3
  • Concurrency: 10

@Joffcom
Copy link
Member

Joffcom commented Mar 5, 2024

Hey @shankarmn94,

I don't see any issue from the worker not connecting to the redis queue to pick up jobs, When you are testing to show it isn't working are you running the workflow on a schedule or using the test button in the n8n interface?

@shankarmn94
Copy link
Author

@Joffcom schedule / trigger

@shankarmn94
Copy link
Author

now i made some changes;

  1. "n8n:cache:variables"
  2. "bull:jobs:2"
  3. "bull:jobs:id"
  4. "bull:jobs:4"
  5. "n8n:cache:webhook:POST-2960855a-ff6d-48cb-b2d9-77faf5c78533"
  6. "workflows:jobs:stalled-check"
  7. "bull:jobs:priority"
  8. "bull:jobs:wait"
  9. "n8n:cache:webhook:POST-2960855a-ff6d-48cb-b2d9-77faf5c78526"
  10. "n8n:cache:webhook:POST-573da17f-bd25-4604-a542-1d1cc455b6a4"
  11. "bull:jobs:5"
  12. "n8n:cache:webhook:POST-2960855a-ff6d-48cb-b2d9-77faf5c78523"
  13. "bull:jobs:3"
  14. "bull:jobs:1"

its creating the keys in redis but the worker is not picking this nor main server not runing this

@Joffcom
Copy link
Member

Joffcom commented Mar 5, 2024

Hey @shankarmn94

I am not sure what those settings are that you have there but if they are in Redis I would recommend not playing with them.

Can you share the current configuration for your main instance and your worker as they currently are so I can check them again I suspect you still don't have it configured correctly.

@shankarmn94
Copy link
Author

as i mentioned earlier:

WEBHOOK_URL=http://10.51.112.8:5678/
N8N_BASIC_AUTH_USER=cloudquen8n
N8N_BASIC_AUTH_PASSWORD=n8ncloudque
DB_TYPE=postgresdb
DB_POSTGRESDB_DATABASE=workflow
DB_POSTGRESDB_HOST=10.51.112.8
DB_POSTGRESDB_PORT=5432
DB_POSTGRESDB_USER=workflow
DB_POSTGRESDB_PASSWORD=workflow
#N8N_PAYLOAD_SIZE_MAX=56MB
EXECUTIONS_DATA_PRUNE=true
EXECUTIONS_DATA_PRUNE_MAX_COUNT=5000000
EXECUTIONS_DATA_SAVE_ON_ERROR=all
EXECUTIONS_DATA_SAVE_ON_SUCCESS=all
NODE_OPTIONS="--max-old-space-size=16048 --require /root/.n8n/set-max-listeners.js"
EXECUTIONS_MODE=queue
N8N_ENCRYPTION_KEY=Zwyi2FIVHTJEYb5+KY74esTPlKn4Vjru
QUEUE_BULL_REDIS_HOST=10.51.112.9
QUEUE_BULL_PREFIX=workflows
QUEUE_BULL_REDIS_DB=0
QUEUE_BULL_REDIS_PORT=6379

@Joffcom
Copy link
Member

Joffcom commented Mar 6, 2024

Hey @shankarmn94,

I just wanted to check again as the first post you had queue mode disabled which would have been an issue.

So assuming you have those settings on workers and the main instance it will work although there are some tweaks you can make as 2 of those keys don't exist anymore so in theory the below would work.

WEBHOOK_URL=http://10.51.112.8:5678/
DB_TYPE=postgresdb
DB_POSTGRESDB_DATABASE=workflow
DB_POSTGRESDB_HOST=10.51.112.8
DB_POSTGRESDB_PORT=5432
DB_POSTGRESDB_USER=workflow
DB_POSTGRESDB_PASSWORD=workflow
EXECUTIONS_DATA_PRUNE=true
EXECUTIONS_DATA_PRUNE_MAX_COUNT=5000000
EXECUTIONS_DATA_SAVE_ON_ERROR=all
EXECUTIONS_DATA_SAVE_ON_SUCCESS=all
NODE_OPTIONS="--max-old-space-size=16048 --require /root/.n8n/set-max-listeners.js"
EXECUTIONS_MODE=queue
N8N_ENCRYPTION_KEY=Zwyi2FIVHTJEYb5+KY74esTPlKn4Vjru
QUEUE_BULL_REDIS_HOST=10.51.112.9
QUEUE_BULL_PREFIX=workflows

How are you actually setting the env options?

@shankarmn94
Copy link
Author

am able to figure out the issue i had:

We were using dragonflydb which is alternative of redis.. while workers fails to get keys from dragonflydb later on i checked with redis where all my workers started take the triggered workflows and finish those..

i donno is that issue with n8n worker or dragonflydb.

@Joffcom
Copy link
Member

Joffcom commented Mar 8, 2024

Hey @shankarmn94,

We only officially support Redis so I would recommend sticking with that and not something that is Redis compatible, For now I am going to mark this as closed as it doesn't look like we have an issue here.

Thanks for the update 👍🏻

@Joffcom Joffcom closed this as not planned Won't fix, can't repro, duplicate, stale Mar 8, 2024
@shankarmn94
Copy link
Author

shankarmn94 commented Mar 8, 2024 •
edited
Loading

But Dragonlfydb is very big platform i donno why you dont support that. Anyway thanks for response. Have a great day...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Joffcom @shankarmn94