Skip to content

Fix #364, #292: capture Java tab-indented enum members without false positives - #401

Merged
Widthdom merged 9 commits into
mainfrom
fix/issue-364-csharp-tab-indented-enum
Apr 17, 2026
Merged

Fix #364, #292: capture Java tab-indented enum members without false positives#401
Widthdom merged 9 commits into
mainfrom
fix/issue-364-csharp-tab-indented-enum

Conversation

@Widthdom

Copy link
Copy Markdown
Owner

Summary

  • Problem (C# enum members indented with a single tab are silently dropped — :127 regex requires \s{2,} but a tab is only 1 character #364): the Java enum-member regex required ^\s{2,} indentation, so a single tab of indentation (common with EditorConfig indent_style=tab or legacy IDE defaults) silently dropped every Java enum member. The enum shell surfaced as a Color symbol but RED / GREEN / BLUE were invisible to symbols, definition, references, callers, and callees.
  • Blocker for a simple fix (Java enum member pattern over-matches: any UPPERCASE(...); call statement becomes a phantom function definition — fires outside enum bodies and inside text blocks #292): naively relaxing the leading whitespace to ^\s+ would also start catching tab-indented method calls like \tRED(); inside a class body as phantom enum members, worsening Java enum member pattern over-matches: any UPPERCASE(...); call statement becomes a phantom function definition — fires outside enum bodies and inside text blocks #292.
  • Fix: replaced the regex path with a body-scoped scanner (ExtractJavaEnumMembers) modeled on the existing C# approach. The scanner walks the enum body tracking strings, char literals, line/block comments, Java 15+ text blocks, parens, brackets, and nested braces (for anonymous member bodies), emits each member at top-level , boundaries, and stops at the first top-level ;.
  • Lex-aware annotation skip: @Annotation(...) prefixes are skipped using the same lexer state machine, so annotations with quoted parens / block comments / text blocks no longer derail member binding. @Deprecated A(1) still binds to A.
  • Lex-aware body range (FindJavaBraceRange): the enum body range itself is now resolved by a new Java-specific range resolver that reuses the same lexer state machine, so a } inside a text block or quoted string no longer prematurely closes the enum body and drops every member after the literal.
  • Bounded recovery: when the primary scanner exits with unbalanced paren/bracket depths (e.g. a malformed @Ann( spanning a partial edit), a bounded line-regex recovery pass emits obvious uppercase-identifier members. Recovery tracks brace depth across the body so lines inside anonymous member bodies or methods aren't mis-emitted as phantom members, dedups by member name (the primary scanner stamps StartLine at the annotation line while recovery stamps the member-name line, so StartLine-based dedup would double-emit), and terminates on a top-level ;.
  • Unicode identifiers per JLS §3.8: the member-name regex now accepts Unicode letters, so members like RÉSUMÉ or NAÏVE are captured intact instead of truncated at the first non-ASCII character.
  • Convergence: 7 rounds of adversarial review driven to SHIP; full suite passes.

Test plan

  • dotnet build src/CodeIndex — clean (0 warnings, 0 errors)
  • dotnet test — 1619 passed, 0 failed, 2 skipped (performance tests)
  • Extract_Java_DoesNotExtractMethodCallsAsEnumMembers pins tab-indented RED(); / GREEN(); inside a class body are not symbolized
  • Extract_Java_StopsEnumMembersAtSemicolon pins declarations after the first top-level ; are not added as enum members
  • Extract_Java_HandlesAnnotationWithQuotedParen / Extract_Java_HandlesBlockCommentBetweenAnnotationAndMember pin the lex-aware annotation skip
  • Extract_Java_RecoversMembersWhenAnnotationIsMalformed / Extract_Java_HandlesEmptyEnumBody / Extract_Java_HandlesEnumWithOnlySemicolon / Extract_Java_HandlesTrailingComma / Extract_Java_HandlesAnonymousMemberBody pin the bounded recovery and edge-case body shapes
  • Extract_Java_RecoveryIgnoresLinesInsideAnonymousMemberBody / Extract_Java_RecoveryDedupsByNameAcrossAnnotationStartLines pin the brace-depth-aware recovery and name-based dedup
  • Extract_Java_DetectsUnicodeEnumMembers pins Unicode member names per JLS §3.8
  • Extract_Java_HandlesTextBlockContainingBrace / Extract_Java_HandlesStringContainingBrace pin the lex-aware body-range resolution
  • Existing tab / 2-space / static-final / declaration coverage remains intact

Closes #364.
Closes #292.

🤖 Generated with Claude Code

Widthdom and others added 9 commits April 17, 2026 23:12
The C# side was already relaxed from `^\s{2,}` to `^\s+` via #214 (with the
existing Extract_CSharp_DetectsTabIndentedEnumMembers regression), but the
parallel Java enum-member regex still required two whitespace characters.
Single-tab indentation (EditorConfig `indent_style=tab` or legacy IDE
defaults) therefore silently dropped every Java enum member — the enum
shell surfaced as a symbol but RED/GREEN/BLUE were invisible to `symbols`,
`definition`, `references`, `callers`, and `callees`.

Java now uses `^\s+` so a single leading tab or space matches. The
existing anchored tail (`(?:,|\{|;)\s*$`) still excludes file-top
declarations because a non-indented `public class Foo {` ends with `{`
but has zero leading whitespace. Added a Java tab-indent regression
alongside the existing 2-space fixture, and updated the stale comment
claim that the old threshold supported tabs.

Closes #364

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the Java enum-member line regex with ExtractJavaEnumMembers, a
body-scoped scanner modeled on the C# implementation. The scanner walks
the enum body tracking strings, char literals, line/block comments,
Java 15+ text blocks, parens, brackets, and nested braces, emits each
member at top-level ',' boundaries, and stops at the first top-level
';'. Member-name extraction skips leading @annotation(...) forms.

This captures tab-indented enum members (#364) without introducing the
phantom-symbol regression that a simple regex relaxation would cause
for tab-indented UPPERCASE method calls (#292, e.g. \tRED();).

Regression coverage:
- Extract_Java_DetectsTabIndentedEnumMembers (already in place)
- Extract_Java_DoesNotExtractMethodCallsAsEnumMembers (new)
- Extract_Java_StopsEnumMembersAtSemicolon (new)

Closes #364, #292.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
SkipLeadingJavaAnnotations was not string/comment-aware, so
`@Label(")")` closed the paren-balance counter prematurely and
`@A /*note*/ B` stopped the skip on the `/`. Both inputs dropped the
enum member name silently.

Rewrite the skip on top of a multi-line-aware scanner
(TryConsumeJavaNonCodeAcrossLines) that tracks strings, char
literals, text blocks, and block/line comments. Pair helpers:
  - SkipJavaWhitespaceAndComments walks whitespace and comments
    between annotations and around identifiers/parens.
  - TryConsumeJavaNonCodeAcrossLines handles `\n` explicitly so
    line-comment / string / char literals close correctly in a
    multi-line span.

Regression tests:
- Extract_Java_HandlesAnnotationWithQuotedParen
- Extract_Java_HandlesBlockCommentBetweenAnnotationAndMember

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When ExtractJavaEnumMembersFromBody exits with unbalanced paren or
bracket depths, the body almost certainly contains an unterminated
annotation such as `@Ann(`. Without recovery, one unclosed annotation
suppresses all following enum members because the primary scanner
treats them as annotation arguments.

Add RecoverJavaEnumMembersByLine: a per-line regex pass that rescues
obvious uppercase-identifier members between the enum body's opening
brace and its terminating `;`. The fallback only runs on malformed
input (scanner depths > 0 at body end), so well-formed code retains
the primary scanner's stronger guarantees about strings, comments,
text blocks, and paren-balanced annotation arguments.

Regression tests:
- Extract_Java_RecoversMembersWhenAnnotationIsMalformed
- Extract_Java_HandlesEmptyEnumBody
- Extract_Java_HandlesEnumWithOnlySemicolon
- Extract_Java_HandlesTrailingComma
- Extract_Java_HandlesAnonymousMemberBody

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Round 4 adversarial review surfaced two blocking issues in
RecoverJavaEnumMembersByLine:

- B-1: The per-line regex was context-free, so uppercase call
  statements inside an anonymous member body (e.g. `ACTIVATE_HELPER();`)
  could be emitted as phantom enum members, and their trailing `;`
  would terminate recovery prematurely and drop subsequent real members.
- B-2: Dedup was keyed on StartLine. The primary scanner stamps
  StartLine at the annotation line while recovery stamps the
  member-name line, so the same member could be double-emitted.

Fix:
- Track brace depth across the enum body using TryConsumeJavaNonCode so
  strings, char literals, comments, and text blocks don't corrupt the
  depth. Only apply the line regex when the line starts at
  braceDepth == 0 and the scanner mode is Normal.
- Terminate recovery on a top-level `;` (braceDepth == 0) detected via
  the char scan, rather than a naive line-suffix check.
- Dedup by member name within the enum container. Java enum member
  names are unique by language rule, so this is both safer and simpler
  than aligning StartLine bases between the primary scanner and the
  recovery pass.

Regression tests:
- Extract_Java_RecoveryIgnoresLinesInsideAnonymousMemberBody
- Extract_Java_RecoveryDedupsByNameAcrossAnnotationStartLines

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
JavaEnumMemberNameRegex was ASCII-only ([A-Za-z_$][A-Za-z0-9_$]*), which
truncated enum member names like RÉSUMÉ or NAÏVE at the first non-ASCII
character — the extractor would report `R` or `NA` instead of the full
identifier. Java allows Unicode identifiers (JLS §3.8).

Widen the regex to match JLS identifier rules:
- Start: \p{L} (letter), \p{Nl} (letter number), underscore, dollar
- Continue: \p{L}\p{Nl}\p{Nd}\p{Mn}\p{Mc}\p{Pc} plus underscore and dollar

Regression test: Extract_Java_DetectsUnicodeEnumMembers covers RÉSUMÉ
and NAÏVE, and asserts that the pre-fix truncated names `R`/`NA` are
not emitted.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
FindBraceRange (the BodyStyle.Brace default) was a dumb {/}
counter with no string/comment/text-block awareness. A `}` inside
a Java text block (`"""..."""`) or a regular string literal would
be counted as a closing brace and truncate the enum body range
early, dropping every member declared after the literal.

Add FindJavaBraceRange, dispatched from ResolveRange for Java, that
reuses the TryConsumeJavaNonCode lexer state machine to skip
strings, char literals, line/block comments, and text blocks. Matches
the existing FindCSharpBraceRange / FindJavaScriptBraceRange pattern.

Regression tests:
- Extract_Java_HandlesTextBlockContainingBrace (text block with `}`)
- Extract_Java_HandlesStringContainingBrace (regular string with `}`)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The [Unreleased] entry for #364/#292 only mentioned the initial
body-scoped scanner swap. Document the follow-up refinements landed
during adversarial review:

- Lex-aware annotation skip across quoted parens / block comments
  / text blocks.
- Bounded recovery pass for malformed annotation bodies.
- Brace-depth-aware recovery with member-name dedup.
- Unicode (JLS §3.8) identifiers.
- FindJavaBraceRange for enum body ranges that survives `}` inside
  text blocks and quoted strings.

Expand both the English and Japanese test lists to name the new
regression tests (annotation lex-awareness, recovery edge cases,
anonymous-body guard, Unicode members, text block and quoted-brace
body-range checks).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…tab-indented-enum

# Conflicts:
#	CHANGELOG.md
@Widthdom
Widthdom merged commit ad02061 into main Apr 17, 2026
6 checks passed
@Widthdom
Widthdom deleted the fix/issue-364-csharp-tab-indented-enum branch April 17, 2026 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant