Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Enable parallel processing on multiple queue nodes #6295

Merged
merged 14 commits into from
Aug 16, 2023

Conversation

agobrech
Copy link
Contributor

@agobrech agobrech commented May 22, 2023

NODE-109
Github issue / Community forum post (link here to close automatically):

@github-actions
Copy link
Contributor

Great PR! Please pay attention to the following items before merging:

Files matching packages/**:

  • If fixing bug, added test to cover scenario.
  • If addressing forum or Github issue, added link to description.

Files matching packages/**/*.ts:

  • Added unit tests to cover new or updated functionality.

Files matching packages/nodes-base/nodes/**:

  • Added workflow tests for nodes if possible.

Make sure to check off this list before asking for review.

@n8n-assistant n8n-assistant bot added n8n team Authored by the n8n team node/improvement New feature or request labels May 22, 2023
@codecov
Copy link

codecov bot commented Aug 14, 2023

Codecov Report

Patch coverage has no change and project coverage change: +0.01% 🎉

Comparison is base (4e4a3cf) 24.84% compared to head (ea5c8f1) 24.85%.
Report is 9 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6295      +/-   ##
==========================================
+ Coverage   24.84%   24.85%   +0.01%     
==========================================
  Files        3144     3144              
  Lines      191232   191255      +23     
  Branches    21087    21101      +14     
==========================================
+ Hits        47519    47545      +26     
+ Misses     142735   142729       -6     
- Partials      978      981       +3     
Files Changed Coverage Δ
...ackages/nodes-base/credentials/Amqp.credentials.ts 0.00% <ø> (ø)
packages/nodes-base/nodes/Amqp/Amqp.node.ts 0.00% <0.00%> (ø)
packages/nodes-base/nodes/Amqp/AmqpTrigger.node.ts 0.00% <0.00%> (ø)
packages/nodes-base/nodes/MQTT/MqttTrigger.node.ts 0.00% <0.00%> (ø)

... and 9 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@maspio maspio marked this pull request as ready for review August 15, 2023 15:29
@github-actions
Copy link
Contributor

✅ All Cypress E2E specs passed

@cypress
Copy link

cypress bot commented Aug 16, 2023

Passing run #1881 ↗︎

0 221 0 0 Flakiness 0

Details:

🌳 node-109-refactor-nodes-to-handle-reconnection-on 🖥️ browsers:node18.12.0-ch...
Project: n8n Commit: ea5c8f1385
Status: Passed Duration: 08:18 💡
Started: Aug 16, 2023 10:07 AM Ended: Aug 16, 2023 10:16 AM

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings.

@maspio maspio merged commit 44afcff into master Aug 16, 2023
18 checks passed
@maspio maspio deleted the node-109-refactor-nodes-to-handle-reconnection-on branch August 16, 2023 11:06
@github-actions github-actions bot mentioned this pull request Aug 16, 2023
netroy added a commit that referenced this pull request Aug 16, 2023
# [1.3.0](https://github.com/n8n-io/n8n/compare/n8n@1.2.0...n8n@1.3.0)
(2023-08-16)


### Bug Fixes

* **core:** Don't let bull override the default redis config
([#6897](#6897))
([cfeb322](cfeb322))
* **core:** Fix fetching of EE executions
([#6901](#6901))
([f3fce48](f3fce48))
* **core:** Update frontend urls when using the `--tunnel` option
([#6898](#6898))
([718e613](718e613))
* **editor:** Disable telemetry in dev mode and in E2E tests
([#6869](#6869))
([808a928](808a928))
* **editor:** Fix code node’s content property to be reactive
([#6931](#6931))
([3b75bc6](3b75bc6))
* **editor:** Fix event emit on credential sharing
([#6922](#6922))
([297c3c9](297c3c9))
* **editor:** Fix multiOptions parameters resetting on initial load
([#6903](#6903))
([49867c2](49867c2))
* **editor:** Update execution view layout
([#6882](#6882))
([0339732](0339732))
* **Email Trigger (IMAP) Node:** Fix connection issue with unexpected
spaces in host ([#6886](#6886))
([f3248e4](f3248e4))
* Fix issue with key formatting if null or undefined
([#6924](#6924))
([4e4a3cf](4e4a3cf))
* Fix issue with key formatting introduced in 1.2.0
([#6896](#6896))
([0e075c9](0e075c9))
* Fix lag when node parameters are updated
([#6941](#6941))
([3eb65e0](3eb65e0))
* **HTTP Request Node:** Improve error handling for TCP socket errors
when `Continue On Fail` is enabled
([#6925](#6925))
([96ff1f8](96ff1f8))
* Prevent workflow breaking when credential type is unknown
([#6923](#6923))
([e83b93f](e83b93f))
* **Respond to Webhook Node:** Return headers in response
([#6921](#6921))
([a82107f](a82107f))


### Features

* **core:** Add support for not requiring SMTP auth with user management
([#3742](#3742))
([eead6d4](eead6d4))
* **core:** Descriptive message for common nodeJS errors
([#6841](#6841))
([3adb0b6](3adb0b6))
* **editor:** Ask AI in Code node
([#6672](#6672))
([fde6ad1](fde6ad1))
* Enable parallel processing on multiple queue nodes
([#6295](#6295))
([44afcff](44afcff))

Co-authored-by: netroy <netroy@users.noreply.github.com>
@janober
Copy link
Member

janober commented Aug 16, 2023

Got released with n8n@1.3.0

netroy pushed a commit that referenced this pull request Aug 17, 2023
* Add non-parallel execution

* Add parallel processing for MQTT

* Fix logic expression for trigger

* fixes

* remove unused import

* fix MQTT parallel processing

* fix AMQPTrigger node parallelProcessing

* MQTTTrigger node default parallelProcessing to true

* add AMQP credential test

* improve error handling

---------

Co-authored-by: Marcus <marcus@n8n.io>
@github-actions github-actions bot mentioned this pull request Aug 17, 2023
netroy pushed a commit that referenced this pull request Aug 17, 2023
* Add non-parallel execution

* Add parallel processing for MQTT

* Fix logic expression for trigger

* fixes

* remove unused import

* fix MQTT parallel processing

* fix AMQPTrigger node parallelProcessing

* MQTTTrigger node default parallelProcessing to true

* add AMQP credential test

* improve error handling

---------

Co-authored-by: Marcus <marcus@n8n.io>
@github-actions github-actions bot mentioned this pull request Aug 17, 2023
netroy added a commit that referenced this pull request Aug 17, 2023
#
[0.237.0](https://github.com/n8n-io/n8n/compare/n8n@0.236.3...n8n@0.237.0)
(2023-08-17)


### Bug Fixes

* **API:** Fix issue with workflow setting not supporting newer nanoids
([#6699](#6699))
([2e64c9d](2e64c9d))
* **AwsS3 Node:** Fix issue if bucket name contains a '.'
([#6542](#6542))
([219f897](219f897))
* **core:** Add missing primary key on the `execution_data` table on
postgres ([#6797](#6797))
([1c5f98d](1c5f98d))
* **core:** Fix `continueOnFail` for expression error in Set
([#6939](#6939))
([2a3937f](2a3937f))
* **core:** Fix property existence checks on AugmentObject
([#6842](#6842))
([3e4483a](3e4483a))
* **core:** Log crash causes to console when sentry is disabled
([#6890](#6890))
([f19c9c1](f19c9c1))
* **core:** Restrict read/write file paths access
([#6582](#6582))
([6e6b90e](6e6b90e))
* **core:** Serialize BigInts
([#6805](#6805))
([17ce49b](17ce49b))
* **core:** Update frontend urls when using the `--tunnel` option
([#6898](#6898))
([ec3c066](ec3c066))
* **core:** Update packages to address CVE-2023-2142 and CVE-2020-28469
([#6844](#6844))
([6dd0850](6dd0850))
* Correct typos in Taiga and ServiceNow nodes
([#6814](#6814))
([b9f52ec](b9f52ec))
* **editor:** Fix code node’s content property to be reactive
([#6931](#6931))
([f3b380d](f3b380d))
* **editor:** Prevent Code node linter from erroring on `null` parse
([#6934](#6934))
([baada92](baada92))
* **Email Trigger (IMAP) Node:** UTF-8 attachments are not correctly
named ([#6856](#6856))
([f3f1c14](f3f1c14))
* **Email Trigger (IMAP) Node:** Fix connection issue with unexpected
spaces in host ([#6886](#6886))
([35c29ac](35c29ac))
* Fix issue with key based credentials not being read correctly
([#6824](#6824))
([2459b0a](2459b0a))
* Fix issue with key formatting if null or undefined
([#6924](#6924))
([59d33db](59d33db))
* Fix issue with key formatting introduced in 1.2.0
([#6896](#6896))
([8423153](8423153))
* **FTP Node:** List recursive ignore . and .. to prevent infinite loops
([#6707](#6707))
([6a787dd](6a787dd))
* **GitLab Trigger Node:** Fix trigger activation 404 error
([#6711](#6711))
([8c5145b](8c5145b))
* **Gmail Trigger Node:** Early returns in case of no data
([#6727](#6727))
([df2f1d6](df2f1d6))
* **Google BigQuery Node:** Error description improvement
([#6715](#6715))
([261d73e](261d73e))
* **GoToWebinar Node:** Fix issue with timezone incorrectly being
required ([#6865](#6865))
([2f61e34](2f61e34))
* **HTTP Request Node:** Improve error handling for TCP socket errors
when `Continue On Fail` is enabled
([#6925](#6925))
([e8e4c9e](e8e4c9e))
* **Lemlist Node:** Fix pagination issues with campaigns and activities
([#6734](#6734))
([73eddc6](73eddc6))
* **Salesforce Node:** Fix Account update owner operation
([#6958](#6958))
([e71ff76](e71ff76))
* **Telegram Trigger Node:** Add guard to 'include' call on null or
undefined ([#6730](#6730))
([c3c845c](c3c845c))
* **Todoist Node:** Fix issue with section id being ignored
([#6799](#6799))
([e53f4a0](e53f4a0))


### Features

* Add support for not requiring SMTP auth with user management
([#3742](#3742))
([fc97d18](fc97d18))
* **core:** Add support for ready hooks, and credentials overwrite
endpoint in workers ([#6954](#6954))
([e5986c5](e5986c5))
* **editor:** Add "Download" button if JSON data is to large
([#6850](#6850))
([3d815cf](3d815cf))
* Enable parallel processing on multiple queue nodes
([#6295](#6295))
([e45461a](e45461a))

Co-authored-by: netroy <netroy@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
n8n team Authored by the n8n team node/improvement New feature or request Released
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants