Fix whitespace control by handling after tokenization instead of preprocessing

[Copilot]

• Understand the problem: preprocessing approach was fundamentally flawed
• Revert previous changes
• Implement post-tokenization whitespace control handling
• Handle comments specially during tokenization
• Detect and remove hyphen markers (both UnaryOperator and AdditiveBinaryOperator)
• Strip whitespace from adjacent text tokens correctly
• Test with xenova's case: {123} ✓
• Test with original bug case: parses correctly ✓
• All existing tests pass (536/537, only network test fails)
• Linter passes
• Build succeeds
• Add unit tests for whitespace control (7 test cases)
• Optimize to single-pass during lexing (per xenova's request)

Implementation

Refactored to use a single-pass approach during tokenization:

• Check for {%-, {{- before text consumption and strip trailing whitespace immediately
• Check for -%}, -}} after whitespace consumption and skip following whitespace immediately
• Comments {#- and -#} are handled during their own tokenization
• Eliminates the need for postProcessWhitespaceControl() second pass

Original prompt

There is currently a bug with whitespace control in jinja.js. In particular, the preprocessing steps in lexer.ts include these lines:

template
    .replace(/-%}\s*/g, "%}")
    .replace(/\s*{%-/g, "{%")

which remove any whitespace around the block tags with hyphens. However, this can cause some issues with certain templates. Here is one for example:

{
{%- for i in [1, 2, 3] %}
  {{ i }}
{%- endfor %}
}

What happens here is the whitespace before the {{ i }} is removed because of the \s* in the regex, resulting in:

{{%- for i in [1, 2, 3] %}

which then gets interpreted as {{ in the parsing stage, and the contents of the real statement {%- are considered part of the expression.

Your goal is to fix this bug. Carefully consider all options for how to do so, and implement the best one.

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[xenova]

@copilot This doesn't actually fix the bug. Here is a test case which shows the correct output:

{
{%- for i in [1, 2, 3] -%}
  {{ i }}
{%- endfor -%}
}

should render as

{123}

but your code renders it as

{
123}

These negative lookbehind strategies do not work. Instead, we may need to only process the whitespace control after tokenizing.

[Copilot]

@copilot This doesn't actually fix the bug. Here is a test case which shows the correct output:

{

...

You're absolutely right! I've reimplemented the fix to handle whitespace control after tokenization instead of during preprocessing. The new approach:

1. Keeps hyphens during preprocessing
2. Tokenizes them as operators
3. Post-processes tokens to identify and remove hyphen markers
4. Strips whitespace from adjacent text tokens

Your test case now renders correctly as {123}. Fixed in commit 9f1f00a.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• huggingface.co
  • Triggering command: /usr/local/bin/node node /home/REDACTED/work/huggingface.js/huggingface.js/node_modules/.bin/../vitest/vitest.mjs run (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[xenova]

@copilot You should add the example I gave as a test case:

{
{%- for i in [1, 2, 3] -%}
  {{ i }}
{%- endfor -%}
}

should output

{123}

while you're at it, add some other unit tests to ensure correct whitespace control processing.

[Copilot]

@copilot You should add the example I gave as a test case:

{

...

Added 7 test cases for whitespace control in commit bee8364:

• WHITESPACE_CONTROL_1: Your test case with { before {%- → {123}
• WHITESPACE_CONTROL_2: Basic {%- for %}{{ i }}{% endfor -%} → 123
• WHITESPACE_CONTROL_3: {%- if -%} with surrounding spaces → A
• WHITESPACE_CONTROL_4: Expression {{- and -}} → A B C
• WHITESPACE_CONTROL_5: Comment {#- -#} with text after → X
• WHITESPACE_CONTROL_6: Comment {#- -#} with text before → X
• WHITESPACE_CONTROL_7: {%- set -%} statement → 1

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• huggingface.co
  • Triggering command: /usr/local/bin/node node /home/REDACTED/work/huggingface.js/huggingface.js/node_modules/.bin/../vitest/vitest.mjs run (dns block)
  • Triggering command: /usr/local/bin/node node /home/REDACTED/work/huggingface.js/huggingface.js/node_modules/.bin/../vitest/vitest.mjs run -- --reporter=verbose (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[xenova]

@copilot Good stuff, things are pretty much working. However, I'd like you to optimize your implementation. It seems unnecessary to do a two-pass approach during lexing. I'd prefer if this logic was correctly handled during the first pass.

[Copilot]

@copilot Good stuff, things are pretty much working. However, I'd like you to optimize your implementation. It seems unnecessary to do a two-pass approach during lexing. I'd prefer if this logic was c...

Optimized to single-pass in commit c2c6633. Removed postProcessWhitespaceControl() and integrated whitespace control directly into the tokenization loop:

• {%- and {{- are detected before text consumption
• -%} and -}} are detected after whitespace consumption (to handle {% endfor -%} with space before hyphen)
• No second pass needed

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• huggingface.co
  • Triggering command: /usr/local/bin/node node /home/REDACTED/work/huggingface.js/huggingface.js/node_modules/.bin/../vitest/vitest.mjs run (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)