feat(core): Integrate object store as binary data manager #7253

ivov · 2023-09-25T16:14:40Z

Depends on: #7225 | Story: PAY-848

This PR integrates the object store service as a new binary data manager for Enterprise.

krynble

Tested also in queue mode, all looking good. Just left a few minor comments, but it's overall great.

krynble · 2023-09-29T13:06:34Z

packages/cli/src/config/schema.ts

+				accountId: {
+					format: String,
+					default: '',
+					env: 'N8N_EXTERNAL_STORAGE_S3_ACCOUNT_ID',


This turned out a bit misleading - I tried using the AWS account ID but what's supposed to be here is the access key. I think we should rename this to access key and access secret WDYT?

krynble · 2023-09-29T13:46:06Z

packages/core/src/ObjectStore/AWS-S3-SETUP.md

@@ -0,0 +1,89 @@
+# AWS S3 Setup


If these instructions are part of the docs, I'd suggest removing them from here to prevent having them outdated - maybe we could keep the file and link to the docs page.

cypress · 2023-09-29T14:37:13Z

1 flaky test on run #2338 ↗︎

0	242	3	0	1

Details:

🌳 pay-848-integrate-object-store-service-as-binary-data-manager 🖥️ browsers:no...
Project: n8n	Commit: `7a0039c74c`
Status: Passed	Duration: 09:17 💡
Started: Oct 2, 2023 12:00 PM	Ended: Oct 2, 2023 12:10 PM

cypress/e2e/28-debug.cy.ts • 1 flaky test

View Output Video

Test		Artifacts
Debug > should be able to debug executions		`Output` `Screenshots` `Video`

Review all test suite changes for PR #7253 ↗︎

github-actions · 2023-09-29T14:37:17Z

✅ All Cypress E2E specs passed

flipswitchingmonkey

see note

flipswitchingmonkey · 2023-10-02T10:07:36Z

packages/cli/src/License.ts

@@ -103,6 +104,18 @@ export class License {
 				command: 'reloadLicense',
 			});
 		}
+
+		const isSelected = config.getEnv('binaryDataManager.mode') === 's3';


not a fan of the naming here since it does not show relation to the binary data mode and onfeaturechange() is a general function

github-actions · 2023-10-02T11:54:42Z

✅ All Cypress E2E specs passed

github-actions · 2023-10-02T12:10:14Z

✅ All Cypress E2E specs passed

…7307) Depends on #7253 | Story: [PAY-863](https://linear.app/n8n/issue/PAY-863/switch-binary-filesystem-mode-to-nested-path-structure) This PR introduces `filesystem-v2` to store binary data in the filesystem in the same format as `s3`.

@Version

# [1.11.0](https://github.com/n8n-io/n8n/compare/n8n@1.10.0...n8n@1.11.0) (2023-10-11) ### Bug Fixes * **core:** Add an option to enable postgres ssl with default certs ([#6889](#6889)) ([789e1e7](789e1e7)) * **core:** Fix error on missing paired item data ([#7399](#7399)) ([47e8953](47e8953)) * **core:** Missing pairing info ([#7326](#7326)) ([e2c3c7a](e2c3c7a)) * **core:** Prevent object deletion request on no prefix match ([#7366](#7366)) ([63e11e4](63e11e4)) * **editor:** Fix completions for `.json` on quoted node name in Code node ([#7382](#7382)) ([86e7ec7](86e7ec7)) * **editor:** Implement canvas zoom UX improvements ([#7376](#7376)) ([7e06b31](7e06b31)) * **editor:** Make workflow history button available only for dev builds ([#7392](#7392)) ([7ed466d](7ed466d)) * **editor:** Remove excess margin below run data editor ([#7372](#7372)) ([3fa2764](3fa2764)) * **editor:** Sanitize HTML binary-data before rendering in the UI ([#7400](#7400)) ([2b075bf](2b075bf)) * **editor:** Use display option's @Version specifier ([#7351](#7351)) ([afbf0c3](afbf0c3)) * **Google BigQuery Node:** Location default to jobReference ([#7354](#7354)) ([97bb703](97bb703)) * **Google Drive Trigger Node:** Add Shared Drives support ([#7369](#7369)) ([3e7a4d3](3e7a4d3)) * **Google Sheets Node:** Fix "Maximum call stack size exceeded" error on too many rows ([#7384](#7384)) ([732b15a](732b15a)) * **HTML Node:** Update property fields to not use expressions on drag ([#7379](#7379)) ([77643e5](77643e5)) * **Notion Node:** Handle empty values correctly for Notion selects + multi selects ([#7383](#7383)) ([fbcd1d4](fbcd1d4)) * **Set Node:** Increase search priority ([#7358](#7358)) ([e5ad1e7](e5ad1e7)) * **Webhook Node:** Backward compatible form-data parsing for non-array files ([#7385](#7385)) ([6479eb1](6479eb1)) ### Features * **core:** Add Job Summary to Worker response ([#7360](#7360)) ([b8608ce](b8608ce)) * **core:** Integrate object store as binary data manager ([#7253](#7253)) ([1a661e6](1a661e6)) * **core:** Switch binary filesystem mode to nested path structure ([#7307](#7307)) ([0847623](0847623)) * **editor:** Make PDF and Audio binary-data viewable in the UI ([#7367](#7367)) ([8187be1](8187be1)) * **editor:** Support autologin for upgrade path ([#7316](#7316)) ([1dfa052](1dfa052)) * **Execute Workflow Node:** Run once for each item mode ([#7289](#7289)) ([c8c14ca](c8c14ca)) * **Item Lists Node:** Split merge binary data ([#7297](#7297)) ([965db8f](965db8f)) * **Loop Over Items (Split in Batches) Node:** Automatically add a loop + rename ([#7228](#7228)) ([7b773cc](7b773cc)) * **Notion Node:** Fetch child blocks recursively ([#7304](#7304)) ([193181a](193181a)) Co-authored-by: netroy <netroy@users.noreply.github.com>

janober · 2023-10-11T13:56:48Z

Got released with n8n@1.11.0

@Version

# [1.11.0](https://github.com/n8n-io/n8n/compare/n8n@1.10.0...n8n@1.11.0) (2023-10-11) ### Bug Fixes * **core:** Add an option to enable postgres ssl with default certs ([#6889](#6889)) ([789e1e7](789e1e7)) * **core:** Fix error on missing paired item data ([#7399](#7399)) ([47e8953](47e8953)) * **core:** Missing pairing info ([#7326](#7326)) ([e2c3c7a](e2c3c7a)) * **core:** Prevent object deletion request on no prefix match ([#7366](#7366)) ([63e11e4](63e11e4)) * **editor:** Fix completions for `.json` on quoted node name in Code node ([#7382](#7382)) ([86e7ec7](86e7ec7)) * **editor:** Implement canvas zoom UX improvements ([#7376](#7376)) ([7e06b31](7e06b31)) * **editor:** Make workflow history button available only for dev builds ([#7392](#7392)) ([7ed466d](7ed466d)) * **editor:** Remove excess margin below run data editor ([#7372](#7372)) ([3fa2764](3fa2764)) * **editor:** Sanitize HTML binary-data before rendering in the UI ([#7400](#7400)) ([2b075bf](2b075bf)) * **editor:** Use display option's @Version specifier ([#7351](#7351)) ([afbf0c3](afbf0c3)) * **Google BigQuery Node:** Location default to jobReference ([#7354](#7354)) ([97bb703](97bb703)) * **Google Drive Trigger Node:** Add Shared Drives support ([#7369](#7369)) ([3e7a4d3](3e7a4d3)) * **Google Sheets Node:** Fix "Maximum call stack size exceeded" error on too many rows ([#7384](#7384)) ([732b15a](732b15a)) * **HTML Node:** Update property fields to not use expressions on drag ([#7379](#7379)) ([77643e5](77643e5)) * **Notion Node:** Handle empty values correctly for Notion selects + multi selects ([#7383](#7383)) ([fbcd1d4](fbcd1d4)) * **Set Node:** Increase search priority ([#7358](#7358)) ([e5ad1e7](e5ad1e7)) * **Webhook Node:** Backward compatible form-data parsing for non-array files ([#7385](#7385)) ([6479eb1](6479eb1)) ### Features * **core:** Add Job Summary to Worker response ([#7360](#7360)) ([b8608ce](b8608ce)) * **core:** Integrate object store as binary data manager ([#7253](#7253)) ([1a661e6](1a661e6)) * **core:** Switch binary filesystem mode to nested path structure ([#7307](#7307)) ([0847623](0847623)) * **editor:** Make PDF and Audio binary-data viewable in the UI ([#7367](#7367)) ([8187be1](8187be1)) * **editor:** Support autologin for upgrade path ([#7316](#7316)) ([1dfa052](1dfa052)) * **Execute Workflow Node:** Run once for each item mode ([#7289](#7289)) ([c8c14ca](c8c14ca)) * **Item Lists Node:** Split merge binary data ([#7297](#7297)) ([965db8f](965db8f)) * **Loop Over Items (Split in Batches) Node:** Automatically add a loop + rename ([#7228](#7228)) ([7b773cc](7b773cc)) * **Notion Node:** Fetch child blocks recursively ([#7304](#7304)) ([193181a](193181a)) Co-authored-by: netroy <netroy@users.noreply.github.com>

ivov added 30 commits September 14, 2023 11:02

Merge branch 'master' into pay-771-implement-soft-deletion

e8ccf05

Remove call from hook

2071c7f

Merge parent branch

be44721

Refix conflict

1d50b61

Add formats to schema

e042e91

Add ensureStringArray util

e948f8f

Adjust types based on parsing

0feb926

Add type guard

c73f18c

Clean up types in init()

f8fec92

Fix test

93ea57c

Remove unneeded modifier

c8676be

Cleanup

25006e8

Fix node tests

a7156f8

Remove unneeded line

e0e90b6

Rename constant

219addb

Remove comment

d7c7f5f

Fix tests

1ea380d

Cleanup

a5b388b

Rename error

9204754

Improve error message

9597508

Better error

8fde8be

Better message

ada012e

Make execution ID non-nullable

67e163f

Merge master

9665da1

Readability improvements

12636dd

Adjust types, followup to 67e163f

e5c8c72

Merge parent branch

8849007

Fix lint

b7062e5

Fix lint

79ecaa9

Cleanup

02b6d86

ivov added 5 commits September 28, 2023 10:33

Refactor test

a398719

Expand tests

e584be1

Improve error message

39b3188

Inject dependency

fe56d4e

Comment to clarify

05f2467

ivov marked this pull request as ready for review September 28, 2023 08:49

ivov added 2 commits September 28, 2023 11:18

Merge master

d4bb172

Refactor startup logic

02a54da

krynble previously approved these changes Sep 29, 2023

View reviewed changes

ivov mentioned this pull request Sep 29, 2023

feat(core): Switch binary filesystem mode to nested path structure #7307

Merged

ivov added 2 commits October 2, 2023 10:46

Remove setup doc

a8c39ed

Rename envs

4e99a90

ivov dismissed krynble’s stale review via 4e99a90 October 2, 2023 09:03

Remove Axios error from log

deb6076

flipswitchingmonkey previously approved these changes Oct 2, 2023

View reviewed changes

Better var naming

7a0039c

ivov dismissed flipswitchingmonkey’s stale review via 7a0039c October 2, 2023 11:52

flipswitchingmonkey approved these changes Oct 2, 2023

View reviewed changes

ivov merged commit 1a661e6 into master Oct 5, 2023
54 of 55 checks passed

ivov deleted the pay-848-integrate-object-store-service-as-binary-data-manager branch October 5, 2023 13:25

github-actions bot mentioned this pull request Oct 11, 2023

🚀 Release 1.11.0 #7402

Merged

janober added the Released label Oct 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(core): Integrate object store as binary data manager #7253

feat(core): Integrate object store as binary data manager #7253

ivov commented Sep 25, 2023 •

edited

krynble left a comment

krynble Sep 29, 2023

krynble Sep 29, 2023

cypress bot commented Sep 29, 2023 •

edited

github-actions bot commented Sep 29, 2023

flipswitchingmonkey left a comment

flipswitchingmonkey Oct 2, 2023

github-actions bot commented Oct 2, 2023

github-actions bot commented Oct 2, 2023

janober commented Oct 11, 2023

feat(core): Integrate object store as binary data manager #7253

feat(core): Integrate object store as binary data manager #7253

Conversation

ivov commented Sep 25, 2023 • edited

krynble left a comment

Choose a reason for hiding this comment

krynble Sep 29, 2023

Choose a reason for hiding this comment

krynble Sep 29, 2023

Choose a reason for hiding this comment

cypress bot commented Sep 29, 2023 • edited

1 flaky test on run #2338 ↗︎

Review all test suite changes for PR #7253 ↗︎

github-actions bot commented Sep 29, 2023

flipswitchingmonkey left a comment

Choose a reason for hiding this comment

flipswitchingmonkey Oct 2, 2023

Choose a reason for hiding this comment

github-actions bot commented Oct 2, 2023

github-actions bot commented Oct 2, 2023

janober commented Oct 11, 2023

ivov commented Sep 25, 2023 •

edited

cypress bot commented Sep 29, 2023 •

edited