feat(editor): Implement Resource Mapper component (#6207)

* ⚡ scaffolding * ⚡ finished scaffolding * ⚡ renamed types * ⚡ updated subtitle * ⚡ renamed functions file, UI updates * ⚡ query parameters fixes, ui updates, refactoring * ⚡ fixes for credentials test, setup for error parsing * ⚡ rlc for schema and table, error handling tweaks * ⚡ delete operation, new options * ⚡ columns loader * ⚡ linter fixes * ⚡ where clauses setup * ⚡ logic for processing where clauses * ⚡ select operation * ⚡ refactoring * ⚡ data mode for insert and update, wip * ⚡ data mapping, insert update, skip on conflict option * ⚡ select columns with spaces fix * ⚡ update operation update, wip * ⚡ finished update operation * ⚡ upsert operation * ⚡ ui fixes * Copy updates. * Copy updates. * ⚡ option to convert empty strings to nulls, schema checks * ⚡ UI requested updates * ⚡ ssh setup WIP * ⚡ fixes, ssh WIP * ⚡ ssh fixes, credentials * ⚡ credentials testing update * ⚡ uncaught error fix * ⚡ clean up * ⚡ address in use fix * ⚡ improved error message * ⚡ tests setup * ⚡ unit tests wip * ⚡ config files clean up * ⚡ utils unit tests * ⚡ refactoring * ⚡ setup for testing operations, tests for deleteTable operation * ⚡ executeQuery and insert operations tests * ⚡ select, update, upsert operations tests * ⚡ runQueries tests setup * ⚡ hint to query * Copy updates. * ⚡ ui fixes * ⚡ clean up * ⚡ error message update * ⚡ ui update * Minor tweaks to query params decription. * feat(Google Sheets Node): Implement Resource mapper in Google Sheets node (#5752) * ✨ Added initial resource mapping support in google sheets node * ✨ Wired mapping API endpoint with node-specific logic for fetching mapping fields * ✨ Implementing mapping fields logic for google sheets * ✨ Updating Google Sheets execute methods to support resource mapper fields * 🚧 Added initial version of `ResourceLocator` component * 👌 Added `update` mode to resource mapper modes * 👌 Addressing PR feedback * 👌 Removing leftover const reference * 👕 Fixing lint errors * ⚡ singlton for conections * ⚡ credentials test fix, clean up * feat(Postgres Node): Add resource mapper to new version of Postgres node (#5814) * ⚡ scaffolding * ⚡ finished scaffolding * ⚡ renamed types * ⚡ updated subtitle * ⚡ renamed functions file, UI updates * ⚡ query parameters fixes, ui updates, refactoring * ⚡ fixes for credentials test, setup for error parsing * ⚡ rlc for schema and table, error handling tweaks * ⚡ delete operation, new options * ⚡ columns loader * ⚡ linter fixes * ⚡ where clauses setup * ⚡ logic for processing where clauses * ⚡ select operation * ⚡ refactoring * ⚡ data mode for insert and update, wip * ⚡ data mapping, insert update, skip on conflict option * ⚡ select columns with spaces fix * ⚡ update operation update, wip * ⚡ finished update operation * ⚡ upsert operation * ⚡ ui fixes * Copy updates. * Copy updates. * ⚡ option to convert empty strings to nulls, schema checks * ⚡ UI requested updates * ⚡ ssh setup WIP * ⚡ fixes, ssh WIP * ⚡ ssh fixes, credentials * ⚡ credentials testing update * ⚡ uncaught error fix * ⚡ clean up * ⚡ address in use fix * ⚡ improved error message * ⚡ tests setup * ⚡ unit tests wip * ⚡ config files clean up * ⚡ utils unit tests * ⚡ refactoring * ⚡ setup for testing operations, tests for deleteTable operation * ⚡ executeQuery and insert operations tests * ⚡ select, update, upsert operations tests * ⚡ runQueries tests setup * ⚡ hint to query * Copy updates. * ⚡ ui fixes * ⚡ clean up * ⚡ error message update * ⚡ ui update * Minor tweaks to query params decription. * ✨ Updated Postgres node to use resource mapper component * ✨ Implemented postgres <-> resource mapper type mapping * ✨ Updated Postgres node execution to use resource mapper fields in v3 * 🔥 Removing unused import --------- Co-authored-by: Michael Kret <michael.k@radency.com> Co-authored-by: Giulio Andreini <g.andreini@gmail.com> * feat(core): Resource editor componend P0 (#5970) * ✨ Added inital value of mapping mode dropdown * ✨ Finished mapping mode selector * ✨ Finished implementing mapping mode selector * ✨ Implemented 'Columns to match on' dropdown * ✨ Implemented `loadOptionsDependOn` support in resource mapper * ✨ Implemented initial version of mapping fields * ✨ Implementing dependant fields watcher in new component setup * ✨ Generating correct resource mapper field types. Added `supportAutoMap` to node specification and UI. Not showing fields with `display=false`. Pre-selecting matching columns if it's the only one * ✨ Handling matching columns correctly in UI * ✨ Saving and loading resourceMapper values in component * ✨ Implemented proper data saving and loading * ✨ ResourceMapper component refactor, fixing value save/load * ✨ Refactoring MatchingColumnSelect component. Updating Sheets node to use single key match and Postgres to use multi key * ✨ Updated Google Sheets node to work with the new UI * ✨ Updating Postgres Node to work with new UI * ✨ Additional loading indicator that shown if there is no mapping mode selector * ✨ Removing hard-coded values, fixing matching columns ordering, refactoring * ✨ Updating field names in nodes * ✨ Fixing minor UI issues * ✨ Implemented matching fields filter logic * ✨ Moving loading label outside of fields list * ✅ Added initial unit tests for resource mapper * ✅ Finished default rendering test * ✅ Test refactoring * ✅ Finished unit tests * 🔨 Updating the way i18n is used in resource mapper components * ✔️ Fixing value to match on logic for postgres node * ✨ Hiding mapping fields when auto-map mode is selected * ✨ Syncing selected mapping mode between components * ✨ Fixing dateTime input rendering and adding update check to Postgres node * ✨ Properly handling database connections. Sending null for empty string values. * 💄 Updated wording in the error message for non-existing rows * ✨ Fixing issues with selected matching values * ✔️ Updating unit tests after matching logic update * ✨ Updating matching columns when new fields are loaded * ✨ Defaulting to null for empty parameter values * ✨ Allowing zero as valid value for number imputs * ✨ Updated list of types that use datepicker as widger * ✨ Using text inputs for time types * ✨ Initial mapping field rework * ✨ Added new component for mapping fields, moved bit of logic from root component to matching selector, fixing some lint errors * ✨ Added tooltip for columns that cannot be deleted * ✨ Saving deleted values in parameter value * ✨ Implemented control to add/remove mapping fields * ✨ Syncing field list with add field dropdown when changing dependent values * ✨ Not showing removed fields in matching columns selector. Updating wording in matching columns selector description * ✨ Implementing disabled states for add/remove all fields options * ✨ Saving removed columns separately, updating copy * ✨ Implemented resource mapper values validation * ✨ Updated validation logic and error input styling * ✨ Validating resource mapper fields when new nodes are added * ✨ Using node field words in validation, refactoring resource mapper component * ✨ Implemented schema syncing and add/remove all fields * ✨ Implemented custom parameter actions * ✨ Implemented loading indicator in parameter options * 🔨 Removing unnecessary constants and vue props * ✨ Handling default values properly * ✨ Fixing validation logic * 👕 Fixing lint errors * ⚡ Fixing type issues * ⚡ Not showing fields by default if `addAllFields` is set to `false` * ✨ Implemented field type validation in resource mapper * ✨ Updated casing in copy, removed all/remove all option from bottom menu * ✨ Added auto mapping mode notice * ✨ Added support for more types in validation * ✨ Added support for enumerated values * ✨ Fixing imports after merging * ✨ Not showing removed fields in matching columns selector. Refactoring validation logic. * 👕 Fixing imports * ✔️ Updating unit tests * ✅ Added resource mapper schema tests * ⚡ Removing `match` from resource mapper field definition, fixing matching columns loading * ⚡ Fixed schema merging * ⚡ update operation return data fix * ⚡ review * 🐛 Added missing import * 💄 Updating parameter actions icon based on the ui review * 💄 Updating word capitalisation in tooltips * 💄 Added empty state to mapping fields list * 💄 Removing asterisk from fields, updating tooltips for matching fields * ⚡ Preventing matching fields from being removed by 'Remove All option' * ⚡ Not showing hidden fields in the `Add field` dropdown * ⚡ Added support for custom matching columns labels * ⚡ query optimization * ⚡ fix * ⚡ Optimizing Postgres node enumeration logic * ⚡ Added empty state for matching columns * ⚡ Only fully loading fields if there is no schema fetched * ⚡ Hiding mapping fields if there is no matching columns available in the schema * ✔️ Fixing minor issues * ✨ Implemented runtime type validation * 🔨 Refactoring validation logic * ✨ Implemented required check, added more custom messages * ✨ Skipping boolean type in required check * Type check improvements * ✨ Only reloading fields if dependent values actually change * ✨ Adding item index to validation error title * ✨ Updating Postgres fetching logic, using resource mapper mode to determine if a field can be deleted * ✨ Resetting field values when adding them via the addAll option * ⚡ Using minor version (2.2) for new Postgres node * ⚡ Implemented proper date validation and type casting * 👕 Consolidating typing * ✅ Added unit tests for type validations * 👌 Addressing front-end review comments * ⚡ More refactoring to address review changes * ⚡ Updating leftover props * ⚡ Added fallback for ISO dates with invalid timezones * Added timestamp to datetime test cases * ⚡ Reseting matching columns if operation changes * ⚡ Not forcing auto-increment fields to be filled in in Postgres node. Handling null values * 💄 Added a custom message for invalid dates * ⚡ Better handling of JSON values * ⚡ Updating codemirror readonly stauts based on component property, handling objects in json validation * Deleting leftover console.log * ⚡ Better time validation * ⚡ Fixing build error after merging * 👕 Fixing lint error * ⚡ Updating node configuration values * ⚡ Handling postgres arrays better * ⚡ Handling SQL array syntax * ⚡ Updating time validation rules to include timezone * ⚡ Sending expressions that resolve to `null` or `undefined` by the resource mapper to delete cell content in Google Sheets * ⚡ Allowing removed fields to be selected for match * ⚡ Updated the query for fetching unique columns and primary keys * ⚡ Optimizing the unique query * ⚡ Setting timezone to all parsed dates * ⚡ Addressing PR review feedback * ⚡ Configuring Sheets node for production, minor vue component update * New cases added to the TypeValidation test. * ✅ Tweaking validation rules for arrays/objects and updating test cases --------- Co-authored-by: Michael Kret <michael.k@radency.com> Co-authored-by: Giulio Andreini <g.andreini@gmail.com>
n8n-io · May 31, 2023 · 04cfa54 · 04cfa54
1 parent 5ae1124
commit 04cfa54
Show file tree

Hide file tree

Showing 57 changed files with 3,431 additions and 178 deletions.
diff --git a/packages/cli/src/Server.ts b/packages/cli/src/Server.ts
@@ -31,6 +31,7 @@ import clientOAuth1 from 'oauth-1.0a';
 import {
 	BinaryDataManager,
 	Credentials,
+	LoadMappingOptions,
 	LoadNodeParameterOptions,
 	LoadNodeListSearch,
 	UserSettings,
@@ -49,6 +50,7 @@ import type {
 	ICredentialTypes,
 	ExecutionStatus,
 	IExecutionsSummary,
+	ResourceMapperFields,
 	IN8nUISettings,
 } from 'n8n-workflow';
 import { LoggerProxy, jsonParse } from 'n8n-workflow';
@@ -79,6 +81,7 @@ import type {
 	NodeListSearchRequest,
 	NodeParameterOptionsRequest,
 	OAuthRequest,
+	ResourceMapperRequest,
 	WorkflowRequest,
 } from '@/requests';
 import { registerController } from '@/decorators';
@@ -756,6 +759,58 @@ export class Server extends AbstractServer {
 			),
 		);
 
+		this.app.get(
+			`/${this.restEndpoint}/get-mapping-fields`,
+			ResponseHelper.send(
+				async (
+					req: ResourceMapperRequest,
+					res: express.Response,
+				): Promise<ResourceMapperFields | undefined> => {
+					const nodeTypeAndVersion = jsonParse(
+						req.query.nodeTypeAndVersion,
+					) as INodeTypeNameVersion;
+
+					const { path, methodName } = req.query;
+
+					if (!req.query.currentNodeParameters) {
+						throw new ResponseHelper.BadRequestError(
+							'Parameter currentNodeParameters is required.',
+						);
+					}
+
+					const currentNodeParameters = jsonParse(
+						req.query.currentNodeParameters,
+					) as INodeParameters;
+
+					let credentials: INodeCredentials | undefined;
+
+					if (req.query.credentials) {
+						credentials = jsonParse(req.query.credentials);
+					}
+
+					const loadMappingOptionsInstance = new LoadMappingOptions(
+						nodeTypeAndVersion,
+						this.nodeTypes,
+						path,
+						currentNodeParameters,
+						credentials,
+					);
+
+					const additionalData = await WorkflowExecuteAdditionalData.getBase(
+						req.user.id,
+						currentNodeParameters,
+					);
+
+					const fields = await loadMappingOptionsInstance.getOptionsViaMethodName(
+						methodName,
+						additionalData,
+					);
+
+					return fields;
+				},
+			),
+		);
+
 		// ----------------------------------------
 		// Active Workflows
 		// ----------------------------------------

diff --git a/packages/cli/src/requests.ts b/packages/cli/src/requests.ts
@@ -348,6 +348,23 @@ export type NodeListSearchRequest = AuthenticatedRequest<
 	}
 >;
 
+// ----------------------------------
+//        /get-mapping-fields
+// ----------------------------------
+
+export type ResourceMapperRequest = AuthenticatedRequest<
+	{},
+	{},
+	{},
+	{
+		nodeTypeAndVersion: string;
+		methodName: string;
+		path: string;
+		currentNodeParameters: string;
+		credentials: string;
+	}
+>;
+
 // ----------------------------------
 //             /tags
 // ----------------------------------

diff --git a/packages/core/src/Interfaces.ts b/packages/core/src/Interfaces.ts
@@ -11,6 +11,7 @@ import type {
 	ITriggerFunctions as ITriggerFunctionsBase,
 	IWebhookFunctions as IWebhookFunctionsBase,
 	BinaryMetadata,
+	ValidationResult,
 } from 'n8n-workflow';
 
 // TODO: remove these after removing `n8n-core` dependency from `nodes-bases`
@@ -89,3 +90,5 @@ export namespace n8n {
 		};
 	}
 }
+
+export type ExtendedValidationResult = Partial<ValidationResult> & { fieldName?: string };
diff --git a/packages/core/src/LoadMappingOptions.ts b/packages/core/src/LoadMappingOptions.ts
@@ -0,0 +1,34 @@
+import type { IWorkflowExecuteAdditionalData, ResourceMapperFields } from 'n8n-workflow';
+
+import * as NodeExecuteFunctions from './NodeExecuteFunctions';
+import { LoadNodeDetails } from './LoadNodeDetails';
+
+export class LoadMappingOptions extends LoadNodeDetails {
+	/**
+	 * Returns the available mapping fields for the ResourceMapper component
+	 */
+	async getOptionsViaMethodName(
+		methodName: string,
+		additionalData: IWorkflowExecuteAdditionalData,
+	): Promise<ResourceMapperFields> {
+		const node = this.getTempNode();
+
+		const nodeType = this.workflow.nodeTypes.getByNameAndVersion(node.type, node.typeVersion);
+		const method = nodeType?.methods?.resourceMapping?.[methodName];
+
+		if (typeof method !== 'function') {
+			throw new Error(
+				`The node-type "${node.type}" does not have the method "${methodName}" defined!`,
+			);
+		}
+
+		const thisArgs = NodeExecuteFunctions.getLoadOptionsFunctions(
+			this.workflow,
+			node,
+			this.path,
+			additionalData,
+		);
+
+		return method.call(thisArgs);
+	}
+}
diff --git a/packages/core/src/NodeExecuteFunctions.ts b/packages/core/src/NodeExecuteFunctions.ts
@@ -61,10 +61,12 @@ import type {
 	IWebhookFunctions,
 	BinaryMetadata,
 	FileSystemHelperFunctions,
+	INodeType,
 } from 'n8n-workflow';
 import {
 	createDeferredPromise,
 	isObjectEmpty,
+	isResourceMapperValue,
 	NodeApiError,
 	NodeHelpers,
 	NodeOperationError,
@@ -74,6 +76,7 @@ import {
 	deepCopy,
 	fileTypeFromMimeType,
 	ExpressionError,
+	validateFieldType,
 } from 'n8n-workflow';
 
 import pick from 'lodash.pick';
@@ -114,7 +117,7 @@ import { access as fsAccess } from 'fs/promises';
 import { createReadStream } from 'fs';
 
 import { BinaryDataManager } from './BinaryDataManager';
-import type { IResponseError, IWorkflowSettings } from './Interfaces';
+import type { ExtendedValidationResult, IResponseError, IWorkflowSettings } from './Interfaces';
 import { extractValue } from './ExtractValue';
 import { getClientCredentialsToken } from './OAuth2Helper';
 import { PLACEHOLDER_EMPTY_EXECUTION_ID } from './Constants';
@@ -1867,7 +1870,7 @@ function cleanupParameterData(inputData: NodeParameterValueType): void {
 	}
 
 	if (Array.isArray(inputData)) {
-		inputData.forEach((value) => cleanupParameterData(value));
+		inputData.forEach((value) => cleanupParameterData(value as NodeParameterValueType));
 		return;
 	}
 
@@ -1886,6 +1889,103 @@ function cleanupParameterData(inputData: NodeParameterValueType): void {
 	}
 }
 
+const validateResourceMapperValue = (
+	parameterName: string,
+	paramValues: { [key: string]: unknown },
+	node: INode,
+	skipRequiredCheck = false,
+): ExtendedValidationResult => {
+	const result: ExtendedValidationResult = { valid: true, newValue: paramValues };
+	const paramNameParts = parameterName.split('.');
+	if (paramNameParts.length !== 2) {
+		return result;
+	}
+	const resourceMapperParamName = paramNameParts[0];
+	const resourceMapperField = node.parameters[resourceMapperParamName];
+	if (!resourceMapperField || !isResourceMapperValue(resourceMapperField)) {
+		return result;
+	}
+	const schema = resourceMapperField.schema;
+	const paramValueNames = Object.keys(paramValues);
+	for (let i = 0; i < paramValueNames.length; i++) {
+		const key = paramValueNames[i];
+		const resolvedValue = paramValues[key];
+		const schemaEntry = schema.find((s) => s.id === key);
+
+		if (
+			!skipRequiredCheck &&
+			schemaEntry?.required === true &&
+			schemaEntry.type !== 'boolean' &&
+			!resolvedValue
+		) {
+			return {
+				valid: false,
+				errorMessage: `The value "${String(key)}" is required but not set`,
+				fieldName: key,
+			};
+		}
+
+		if (schemaEntry?.type) {
+			const validationResult = validateFieldType(
+				key,
+				resolvedValue,
+				schemaEntry.type,
+				schemaEntry.options,
+			);
+			if (!validationResult.valid) {
+				return { ...validationResult, fieldName: key };
+			} else {
+				// If it's valid, set the casted value
+				paramValues[key] = validationResult.newValue;
+			}
+		}
+	}
+	return result;
+};
+
+const validateValueAgainstSchema = (
+	node: INode,
+	nodeType: INodeType,
+	inputValues: string | number | boolean | object | null | undefined,
+	parameterName: string,
+	runIndex: number,
+	itemIndex: number,
+) => {
+	let validationResult: ExtendedValidationResult = { valid: true, newValue: inputValues };
+	// Currently only validate resource mapper values
+	const resourceMapperField = nodeType.description.properties.find(
+		(prop) =>
+			NodeHelpers.displayParameter(node.parameters, prop, node) &&
+			prop.type === 'resourceMapper' &&
+			parameterName === `${prop.name}.value`,
+	);
+
+	if (resourceMapperField && typeof inputValues === 'object') {
+		validationResult = validateResourceMapperValue(
+			parameterName,
+			inputValues as { [key: string]: unknown },
+			node,
+			resourceMapperField.typeOptions?.resourceMapper?.mode !== 'add',
+		);
+	}
+
+	if (!validationResult.valid) {
+		throw new ExpressionError(
+			`Invalid input for '${
+				String(validationResult.fieldName) || parameterName
+			}' [item ${itemIndex}]`,
+			{
+				description: validationResult.errorMessage,
+				failExecution: true,
+				runIndex,
+				itemIndex,
+				nodeCause: node.name,
+			},
+		);
+	}
+	return validationResult.newValue;
+};
+
 /**
  * Returns the requested resolved (all expressions replaced) node parameters.
  *
@@ -1947,6 +2047,16 @@ export function getNodeParameter(
 		returnData = extractValue(returnData, parameterName, node, nodeType);
 	}
 
+	// Validate parameter value if it has a schema defined
+	returnData = validateValueAgainstSchema(
+		node,
+		nodeType,
+		returnData,
+		parameterName,
+		runIndex,
+		itemIndex,
+	);
+
 	return returnData;
 }
 

diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts
@@ -9,6 +9,7 @@ export * from './Constants';
 export * from './Credentials';
 export * from './DirectoryLoader';
 export * from './Interfaces';
+export * from './LoadMappingOptions';
 export * from './LoadNodeParameterOptions';
 export * from './LoadNodeListSearch';
 export * from './NodeExecuteFunctions';

diff --git a/packages/design-system/src/components/N8nActionToggle/ActionToggle.stories.ts b/packages/design-system/src/components/N8nActionToggle/ActionToggle.stories.ts
@@ -8,7 +8,7 @@ export default {
 	argTypes: {
 		placement: {
 			type: 'select',
-			options: ['top', 'top-start', 'top-end', 'bottom', 'bottom-end'],
+			options: ['top', 'top-end', 'top-start', 'bottom', 'bottom-end', 'bottom-start'],
 		},
 		size: {
 			type: 'select',

diff --git a/packages/design-system/src/components/N8nActionToggle/ActionToggle.vue b/packages/design-system/src/components/N8nActionToggle/ActionToggle.vue
@@ -9,7 +9,10 @@
 			@visible-change="onVisibleChange"
 		>
 			<span :class="{ [$style.button]: true, [$style[theme]]: !!theme }">
-				<n8n-icon icon="ellipsis-v" :size="iconSize" />
+				<n8n-icon
+					:icon="iconOrientation === 'horizontal' ? 'ellipsis-h' : 'ellipsis-v'"
+					:size="iconSize"
+				/>
 			</span>
 
 			<template #dropdown>
@@ -79,6 +82,11 @@ export default defineComponent({
 			default: 'default',
 			validator: (value: string): boolean => ['default', 'dark'].includes(value),
 		},
+		iconOrientation: {
+			type: String,
+			default: 'vertical',
+			validator: (value: string): boolean => ['horizontal', 'vertical'].includes(value),
+		},
 	},
 	methods: {
 		onCommand(value: string) {

diff --git a/packages/design-system/src/components/N8nInputLabel/InputLabel.vue b/packages/design-system/src/components/N8nInputLabel/InputLabel.vue
@@ -33,7 +33,11 @@
 				v-if="$slots.options && label"
 				:class="{ [$style.overlay]: true, [$style.visible]: showOptions }"
 			/>
-			<div v-if="$slots.options" :class="{ [$style.options]: true, [$style.visible]: showOptions }">
+			<div
+				v-if="$slots.options"
+				:class="{ [$style.options]: true, [$style.visible]: showOptions }"
+				data-test-id="parameter-input-options-container"
+			>
 				<slot name="options" />
 			</div>
 		</label>

diff --git a/packages/editor-ui/src/Interface.ts b/packages/editor-ui/src/Interface.ts
@@ -1361,6 +1361,13 @@ export type NodeAuthenticationOption = {
 	displayOptions?: IDisplayOptions;
 };
 
+export interface ResourceMapperReqParams {
+	nodeTypeAndVersion: INodeTypeNameVersion;
+	path: string;
+	methodName?: string;
+	currentNodeParameters: INodeParameters;
+	credentials?: INodeCredentials;
+}
 export interface EnvironmentVariable {
 	id: number;
 	key: string;