refactor(core): Streamline flows in multi-main mode (no-changelog)

[ivov]

Summary

This PR brings multi-main mode in line with our current understanding of the best approach to handle all n8n flows relevant to multi-main mode.

Details

See diagram

On instance startup (single- or multi-main mode)

• In single-main mode, the main instance retrieves all workflows marked as active in the database and adds triggers and pollers for active workflows, but it does not add webhooks for active workflows, because we no longer deregister webhooks on shutdown. The main instance also starts pruning.
• In multi-main mode, the leader behaves exactly like the single main instance in single-main mode, whereas the follower does nothing - webhooks need not be added because they were never removed on shutdown, and managing triggers and pollers is solely the responsibility of the leader.

On leader stepdown (only multi-main mode)

• The leader stepdown event is fired if either a former leader found the leader is another main, or if a main found there is no leader at all. In reaction to this event, the former leader stops running triggers and pollers and stops pruning, reaching the same state as the follower on instance startup. Additionally, if this event was fired because the former leader found there is no leader at all, then the former leader attempts to become leader again, where it will be competing with all other followers.

On leader takeover (only multi-main mode)

• The leader takeover event is fired if a main instance successfully set the leader key and marked itself as leader in memory via config.set('multiMainSetup.instanceType', 'leader'). In reaction to this event, the new leader behaves exactly like the leader on instance startup.

On workflow update (single- or multi-main mode)

• In single-main mode, we handle a workflow update differently depending on what changed.
  • If the workflow was updated and activated, the single main instance updates the workflow in the database, and adds webhooks, triggers and pollers for the workflow.
  • If the workflow was updated and deactivated, the single main instance updates the workflow in the database, and removes webhooks, triggers and pollers for the workflow. Removing workflows means deleting them from the webhook_entity table, and deregistering them from external services.
  • If the workflow was updated with no change to its active state, the single main instance updates the workflow in the database, removes webhooks, triggers and pollers for the workflow, and adds them all back again after updating. This allows any changes to the nodes to take effect, e.g. Schedule Trigger frequency.
• In multi-main mode, we handle the three kinds of workflow updates as above, but distributing additions and removals among leader and followers:
  • If the workflow was updated and activated, the instance that received the request, whether leader or follower, updates the workflow in the database, and sends a message to all other main instances with a payload containing the workflow ID to activate. Of the recipient main instances, only the leader reacts to this message by adding webhooks, triggers and pollers for the workflow.
  • If the workflow was updated and deactivated, the instance that received the request, whether leader or follower, updates the workflow in the database, removes webhooks for the workflow, and sends a message to all other main instances with a payload containing the workflow ID for which triggers and pollers need to be removed. Of the recipient main instances, only the leader reacts to this message by adding triggers and pollers for the workflow.
  • If the workflow was updated with no change to its active state, we first remove (same as the above deactivation flow), and then we re-add (same as the above activation flow).

Notes on workflow update flow and its subflows:

• All three subflows apply to both internal and public APIs.
• The deactivation subflow applies also to source control import and user deletion. When importing an active workflow, we deactivate the pre-import workflow. When there is no transferee on user deletion, we deactivate the user's workflows before deleting them.
• The activation subflow applies also to source control import. When importing an active workflow, after deactivating the pre-import workflow, we activate the imported workflow.
• In the activation subflow, on activation failure e.g. due to expired credentials in a trigger, the leader sets the workflow's active state in the database back to false, informs all its connected FEs of the activation failure, and sends a message to all other main instances, which react by also informing all their connected FEs of the activation failure. This update in reaction to an activation failure does not rely on versionId because there is no user intervention. Activation failure state resides in Redis and therefore is available to all connected FEs.
• Why handle webhooks on the recipient but triggers and pollers on the leader? Deactivation in multi-main mode needs to keep the webhook removal at the instance that received the workflow update request, because the update may have removed a webhook node. If webhook removal were to be handled by the leader, when an update removes e.g. a GitHub Trigger node and deactivates the workflow, the instance that received the workflow update request would update the workflow in the database and then send a message for the leader to remove webhooks, triggers and pollers for the workflow - and the leader would do so, but it would never deregister the webhook from GitHub, because by the time the leader takes action, it would be unaware of the GitHub Trigger node that used to be in the workflow and that still requires deregistration from the external service. Hence we keep webhook removal (deletion from database and deregistration from external service) at the instance that received the workflow update request.

On test webhook (single- or multi-main mode)

• In single-main mode, the single main instance registers the test webhook on the cache (to prepare for any request from the external service to confirm the creation of a test webhook), sends a request to register a webhook at the external service, and stores static data from the registration in the cache. The class to manage test webhooks relies on the test webhook registration in the cache to set up and tear down test webhooks, and uses the Workflow.testStaticData property to make test static data available to nodes. Then, when the single main instance receives a webhook from the external service, it executes the workflow, pushes execution lifecycle events to the FE, deregisters the test webhook from the external service, removes the test webhook from the cache, and pushes the final execution lifecycle event to the FE.
• In multi-main mode, a main instance, whether leader or follower, follows the above steps until waiting for a webhook from the external service. When the webhook arrives, the load balancer may forward it to any main instance. If the webhook is forwarded to the instance where the test webhook was created, the rest of the above steps apply. If the webhook is any main instance other than the main instance where the webhook was created, the recipient main instance executes the workflow and relays execution lifecycle events to all other main instances. Of the recipient main instances, the one having the session ID where the webhook was created pushes the execution lifecycle events to the FE. When the execution finishes, the main instance handling the execution deregisters the test webhook from the external service (using Workflow.testStaticData if needed), removes the test webhook from the cache, and relays the final execution lifecycle event to all other main instances. Of the recipient main instances, the one having the session ID where the webhook was created pushes the final execution lifecycle event to the FE.

Manual testing

Single-main regular mode

• Activating a webhook-based workflow should cause the instance to start listening for webhook calls.
• Running a test webhook should cause the instance to start listening for a test webhook call for two minutes.
• Activating a trigger-based workflow should cause the instance to start firing triggers.
• Deactivating a webhook-based workflow should deregister the webhook.
• Cancelling a test webhook should deregister the test webhook.
• Deactivating a trigger-based workflow should cause the instance to stop firing triggers.
• Live editing and saving an active webhook-based workflow should cause the change to take effect.
• Live editing and saving an active trigger-based workflow should cause the change to take effecct.
• Initializing the instance should cause the instance to activate all active workflows and start pruning.
• Activating a workflow leading to an activation failure that should cause the frontend to display the activation failure. (To test this, create a workflow with a webhook-based trigger, add an invalid credential, change active to true in the database, and start instance.)

Multi-main queue mode

• Activating a webhook-based workflow on the leader should cause the instance to start listening for webhook calls.
• Activating a webhook-based workflow on a follower should cause the instance to start listening for webhook calls.
• Activating a trigger-based workflow on the leader should cause the leader to start firing triggers.
• Activating a trigger-based workflow on a follower should cause the leader to start firing triggers.
• Deactivating a webhook-based workflow on the leader should cause the instance to stop listening for webhook calls.
• Deactivating a webhook-based workflow on a follower should cause the instance to stop listening for webhook calls.
• Deactivating a trigger-based workflow on the leader should cause the leader to stop firing triggers.
• Deactivating a trigger-based workflow on a follower should cause the leader to stop firing triggers.
• Live editing and saving an active webhook-based workflow on the leader should cause the change to take effect.
• Live editing and saving an active webhook-based workflow on a follower should cause the change to take effect.
• Live editing and saving an active trigger-based workflow on the leader should cause the change to take effect.
• Live editing and saving an active trigger-based workflow on a follower should cause the change to take effect.
• Live replacing a webhook for a trigger in an active workflow in the leader should cause the webhook to stop listening and the trigger to start firing.
• Live replacing a trigger for a webhook in an active workflow in the leader should cause the webhook to start listening and the trigger to stop firing.
• Live replacing a webhook for a trigger in a follower should cause the webhook to stop listening and the trigger to start firing.
• Live replacing a trigger for a webhook in a follower should cause the webhook to start listening and the trigger to stop firing.
• Initializing the leader should cause the instance to add triggers and pollers for all active workflows and start pruning.
• Initializing a follower should cause the instance to neither add triggers and pollers for all active workflows nor start pruning.
• On the former leader, losing leadership should cause the instance to remove all triggers and pollers and stop pruning.
• On the new leader, gaining leadership should cause the instance to add all triggers and pollers and start pruning.
• Regardless of leader or follower, a main that created a test webhook should cause the instance to start listening for a test webhook call for two minutes.
• Regardless of leader or follower, a main that did not create a test webhook, on receiving a call for it, should relay lifecycle events to the main that created the test webhook.
• Regardless of leader or follower, a main that created a test webhook, on cancelling it, should cause the instance to deregister the test webhook.

[cypress]

2 flaky tests on run #3979 ↗︎

0

339

5

0

2

Details:

🌳 🖥️ browsers:node18.12.0-chrome107 🤖 ivov 🗃️ e2e/*
Project: n8n	Commit: 1a352969ae
Status: Passed	Duration: 03:32 💡
Started: Jan 31, 2024 3:16 PM	Ended: Jan 31, 2024 3:19 PM

  5-ndv.cy.ts • 1 flaky test

View Output Video

Test		Artifacts
NDV > should not retrieve remote options when required params throw errors		Test Replay Screenshots Video

  19-execution.cy.ts • 1 flaky test

View Output Video

Test		Artifacts
Execution > should test manual workflow stop		Test Replay Screenshots Video

Review all test suite changes for PR #8446 ↗︎

[github-actions]

✅ All Cypress E2E specs passed

[krynble]

I'm loving to see the progress.

Left a comment about console messages and noticed something a bit odd - if I click the toggle to deactivate a workflow on a follower, the main won't reflect the deactivation in the UI, when I have the same workflow open in both leader and follower.

[krynble]

This should check be shouldDisplayActivationMessages and be based on whether activationMode === 'init' instead.

The result we're getting now is that whenever i disable and enable a workflow I get the messages on the current leader and I'm almost sure we used to display those messages only on init.

[krynble]

Just to be sure, the UI is ready to only display this if the currently open workflow ID matches to the one sent in the payload right?

[ivov]

The FE at pushConnection.ts receives this workflow ID and calls store methods:

if (receivedData.type === 'workflowActivated') {
	this.workflowsStore.setWorkflowActive(receivedData.data.workflowId);
	return true;
}

if (receivedData.type === 'workflowDeactivated') {
	this.workflowsStore.setWorkflowInactive(receivedData.data.workflowId);
	return true;
}

And the store methods compare the pushed workflow ID against this.workflow.id, which as far as I understand is the workflow on canvas.

setWorkflowActive(workflowId: string): void {
	const uiStore = useUIStore();
	uiStore.stateIsDirty = false;
	const index = this.activeWorkflows.indexOf(workflowId);
	if (index === -1) {
		this.activeWorkflows.push(workflowId);
	}
	if (this.workflowsById[workflowId]) {
		this.workflowsById[workflowId].active = true;
	}
	if (workflowId === this.workflow.id) {
		this.setActive(true);
	}
},

setWorkflowInactive(workflowId: string): void {
	const index = this.activeWorkflows.indexOf(workflowId);
	if (index !== -1) {
		this.activeWorkflows.splice(index, 1);
	}
	if (this.workflowsById[workflowId]) {
		this.workflowsById[workflowId].active = false;
	}
	if (workflowId === this.workflow.id) {
		this.setActive(false);
	}
},

[janober]

Got released with n8n@1.28.0