Components: Fix FormTokenField validation preventing default behavior

[rushikeshmore]

What?

Fixes #69252

Why?

FormTokenField with __experimentalValidateInput and tokenizeOnSpace has two bugs:

1. Space key always prevented: When typing a value that fails validation and pressing space, addCurrentToken() sets preventDefault = true before knowing whether addNewToken() succeeded. This prevents the space character from being typed into the input, making it impossible to type multi-word values that don't yet pass validation.

2. Paste bypasses validation: When pasting comma/space-separated values, addNewTokens() (called via onInputChangeHandler) never runs tokens through __experimentalValidateInput, allowing invalid tokens to be added.

How?

• addNewToken() now returns boolean (true on success, false on validation failure)
• addCurrentToken() uses the return value to decide preventDefault for typed input (selected suggestions still always prevent default)
• addNewTokens() filters tokens through __experimentalValidateInput

Testing Instructions

1. Render a FormTokenField with tokenizeOnSpace and __experimentalValidateInput={ (v) => /^[A-Z]/.test(v) }
2. Type hello (lowercase, space) → the space should appear in the input (not tokenized)
3. Type Hello (uppercase, space) → should create a "Hello" token
4. Type Apple,banana,Cherry, → only "Apple" and "Cherry" should be added as tokens

Testing Instructions for Keyboard

• Verify space key types normally when validation fails with tokenizeOnSpace
• Verify Enter key still works for valid tokens
• Verify comma key still prevents default (comma is always a separator)

Use of AI Tools

Claude Code was used to assist with this bug fix. All changes were reviewed, tested, and verified manually.

[github-actions]

👋 Thanks for your first Pull Request and for helping build the future of Gutenberg and WordPress, @rushikeshmore! In case you missed it, we'd love to have you join us in our Slack community.

If you want to learn more about WordPress development in general, check out the Core Handbook full of helpful information.

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: rushikeshmore <rushikeshmore@git.wordpress.org>
Co-authored-by: ciampo <mciampini@git.wordpress.org>
Co-authored-by: Mamaduka <mamaduka@git.wordpress.org>
Co-authored-by: alshakero <alshakero@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[ciampo]

Some tests are still failing, indicating that the Space key fix is insufficient: onInputChangeHandler still tokenizes on space, clearing the input even when validation fails (see packages/components/src/form-token-field/index.tsx:287–298)

More in general, I appreciate the willingness to help the project. At the same time, please make sure to check the proposed changes and their correctness (such as having related tests passing) before asking for review. Especially now with AI tools assisting, review time is precious.

[rushikeshmore]

Fixed in 27aad02. Two issues addressed:

1. onInputChangeHandler now preserves tokens that fail validation in the input instead of clearing them. When separator-delimited text is processed, only valid tokens are removed from the input -- invalid ones stay visible.

2. Paste test was using user.type which sends comma keystrokes through handleCommaKey (in onKeyPress), not through onInputChangeHandler. Switched to user.paste which correctly exercises the separator-splitting logic in onInputChangeHandler.

All 70 tests pass locally. Sorry about the earlier incomplete fix -- I should have traced both code paths (onKeyDown + onInputChangeHandler) and verified tests before pushing.

[ciampo]

Things are mostly looking good. Apart for the last comment, we'll also need to rebase on top of latest trunk and fix CHANGELOG conflicts.

[ciampo]

We will need to move this entry to the Unreleased > Bug Fixes section

[ciampo]

Let's reword the entry for added clarity

Suggested change

	- `FormTokenField`: Fix `__experimentalValidateInput` not preventing default behavior correctly and not filtering pasted tokens ([#77181](https://github.com/WordPress/gutenberg/pull/77181)).
	- `FormTokenField`: Correct `preventDefault` handling for `__experimentalValidateInput` and validate pasted delimiter-separated tokens ([#77181](https://github.com/WordPress/gutenberg/pull/77181)).

[ciampo]

While reviewing this approach, I noticed one more edge case worth tightening up. If we treat “anything not in addedTokens” as something the user still needs to edit, we accidentally lump in tokens that were skipped because they’re already selected — same string as an existing chip, but not the same thing as “failed validation.” On paste (e.g. Apple,Cherry, when Apple is already there), that showed up as stray text like Apple, left in the combobox even though the only real job was to add Cherry.

I added two unit tests that reproduce that scenario (with and without __experimentalValidateInput) and assert the input clears once the valid new token lands. The fix is to only keep segments in the input when they actually fail __experimentalValidateInput, and to ignore duplicates that were filtered out via valueContainsToken / the batch addedTokens set — same spirit as your suggestion to derive acceptance from addNewTokens, just with duplicates carved out explicitly.

Diff vs PR branch HEAD (git diff HEAD)

diff --git c/packages/components/src/form-token-field/index.tsx w/packages/components/src/form-token-field/index.tsx
index 3c88390e431..bda9cf9b39a 100644
--- c/packages/components/src/form-token-field/index.tsx
+++ w/packages/components/src/form-token-field/index.tsx
@@ -296,12 +296,23 @@ export function FormTokenField( props: FormTokenFieldProps ) {
 			const tokensToProcess = items.slice( 0, -1 );
 			const addedTokens = addNewTokens( tokensToProcess );
 
-			// Keep tokens that were not accepted (invalid,
-			// duplicate, or empty after transform) in the input
-			// so the user can see what was rejected and fix it.
+			// Keep segments that failed validation in the input so the user can
+			// fix them. Skip empty-after-transform tokens, segments already merged
+			// in this batch (`addedTokens`), and duplicates of the current
+			// selection — those are omitted from `tokensToAdd` intentionally, not as
+			// validation failures.
 			const failedTokens = tokensToProcess.filter( ( token ) => {
 				const transformed = saveTransform( token );
-				return transformed && ! addedTokens.has( transformed );
+				if ( ! transformed ) {
+					return false;
+				}
+				if ( addedTokens.has( transformed ) ) {
+					return false;
+				}
+				if ( valueContainsToken( transformed ) ) {
+					return false;
+				}
+				return ! __experimentalValidateInput( transformed );
 			} );
 
 			if ( failedTokens.length > 0 ) {
diff --git c/packages/components/src/form-token-field/test/index.tsx w/packages/components/src/form-token-field/test/index.tsx
index 205b2e7e3b3..25b1d1c28d8 100644
--- c/packages/components/src/form-token-field/test/index.tsx
+++ w/packages/components/src/form-token-field/test/index.tsx
@@ -1957,6 +1957,53 @@ describe( 'FormTokenField', () => {
 			expectTokensToBeInTheDocument( [ 'Apple', 'Cherry' ] );
 			expectTokensNotToBeInTheDocument( [ 'banana' ] );
 		} );
+
+		it( 'should not leave a duplicate of an existing token in the input when pasting comma-separated values', async () => {
+			const user = userEvent.setup();
+
+			const onChangeSpy = jest.fn();
+
+			render(
+				<FormTokenFieldWithState
+					onChange={ onChangeSpy }
+					initialValue={ [ 'Apple' ] }
+				/>
+			);
+
+			const input = screen.getByRole( 'combobox' );
+
+			await user.click( input );
+			await user.paste( 'Apple,Cherry,' );
+
+			expect( onChangeSpy ).toHaveBeenCalledWith( [ 'Apple', 'Cherry' ] );
+			expectTokensToBeInTheDocument( [ 'Apple', 'Cherry' ] );
+			expect( input ).toHaveValue( '' );
+		} );
+
+		it( 'should not leave a duplicate of an existing token in the input when pasting comma-separated values with `__experimentalValidateInput`', async () => {
+			const user = userEvent.setup();
+
+			const onChangeSpy = jest.fn();
+			const startsWithCapitalLetter = ( tokenText: string ) =>
+				/^[A-Z]/.test( tokenText );
+
+			render(
+				<FormTokenFieldWithState
+					onChange={ onChangeSpy }
+					initialValue={ [ 'Apple' ] }
+					__experimentalValidateInput={ startsWithCapitalLetter }
+				/>
+			);
+
+			const input = screen.getByRole( 'combobox' );
+
+			await user.click( input );
+			await user.paste( 'Apple,Cherry,' );
+
+			expect( onChangeSpy ).toHaveBeenCalledWith( [ 'Apple', 'Cherry' ] );
+			expectTokensToBeInTheDocument( [ 'Apple', 'Cherry' ] );
+			expect( input ).toHaveValue( '' );
+		} );
 	} );
 
 	describe( 'maxLength', () => {

[rushikeshmore]

Thanks for the careful review @ciampo. Force-pushed:

Duplicate edge case: Carved duplicates out of the failed-segments filter as you suggested. A segment now only stays in the input when it actually fails __experimentalValidateInput. Empty transforms, segments accepted earlier in the same batch (addedTokens), and segments already present in value (via valueContainsToken) are intentional skips, not failures.

Added the two regression tests from your diff (with and without a custom validator). All 74 tests in form-token-field pass locally.

CHANGELOG: Moved the entry to Unreleased > Bug Fixes and reworded to your suggested phrasing.

Rebase: Rebased on top of latest trunk and resolved the CHANGELOG conflict.