Fix #364, #292: capture Java tab-indented enum members without false positives - #401
Merged
Merged
Conversation
The C# side was already relaxed from `^\s{2,}` to `^\s+` via #214 (with the
existing Extract_CSharp_DetectsTabIndentedEnumMembers regression), but the
parallel Java enum-member regex still required two whitespace characters.
Single-tab indentation (EditorConfig `indent_style=tab` or legacy IDE
defaults) therefore silently dropped every Java enum member — the enum
shell surfaced as a symbol but RED/GREEN/BLUE were invisible to `symbols`,
`definition`, `references`, `callers`, and `callees`.
Java now uses `^\s+` so a single leading tab or space matches. The
existing anchored tail (`(?:,|\{|;)\s*$`) still excludes file-top
declarations because a non-indented `public class Foo {` ends with `{`
but has zero leading whitespace. Added a Java tab-indent regression
alongside the existing 2-space fixture, and updated the stale comment
claim that the old threshold supported tabs.
Closes #364
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the Java enum-member line regex with ExtractJavaEnumMembers, a body-scoped scanner modeled on the C# implementation. The scanner walks the enum body tracking strings, char literals, line/block comments, Java 15+ text blocks, parens, brackets, and nested braces, emits each member at top-level ',' boundaries, and stops at the first top-level ';'. Member-name extraction skips leading @annotation(...) forms. This captures tab-indented enum members (#364) without introducing the phantom-symbol regression that a simple regex relaxation would cause for tab-indented UPPERCASE method calls (#292, e.g. \tRED();). Regression coverage: - Extract_Java_DetectsTabIndentedEnumMembers (already in place) - Extract_Java_DoesNotExtractMethodCallsAsEnumMembers (new) - Extract_Java_StopsEnumMembersAtSemicolon (new) Closes #364, #292. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
SkipLeadingJavaAnnotations was not string/comment-aware, so
`@Label(")")` closed the paren-balance counter prematurely and
`@A /*note*/ B` stopped the skip on the `/`. Both inputs dropped the
enum member name silently.
Rewrite the skip on top of a multi-line-aware scanner
(TryConsumeJavaNonCodeAcrossLines) that tracks strings, char
literals, text blocks, and block/line comments. Pair helpers:
- SkipJavaWhitespaceAndComments walks whitespace and comments
between annotations and around identifiers/parens.
- TryConsumeJavaNonCodeAcrossLines handles `\n` explicitly so
line-comment / string / char literals close correctly in a
multi-line span.
Regression tests:
- Extract_Java_HandlesAnnotationWithQuotedParen
- Extract_Java_HandlesBlockCommentBetweenAnnotationAndMember
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When ExtractJavaEnumMembersFromBody exits with unbalanced paren or bracket depths, the body almost certainly contains an unterminated annotation such as `@Ann(`. Without recovery, one unclosed annotation suppresses all following enum members because the primary scanner treats them as annotation arguments. Add RecoverJavaEnumMembersByLine: a per-line regex pass that rescues obvious uppercase-identifier members between the enum body's opening brace and its terminating `;`. The fallback only runs on malformed input (scanner depths > 0 at body end), so well-formed code retains the primary scanner's stronger guarantees about strings, comments, text blocks, and paren-balanced annotation arguments. Regression tests: - Extract_Java_RecoversMembersWhenAnnotationIsMalformed - Extract_Java_HandlesEmptyEnumBody - Extract_Java_HandlesEnumWithOnlySemicolon - Extract_Java_HandlesTrailingComma - Extract_Java_HandlesAnonymousMemberBody Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Round 4 adversarial review surfaced two blocking issues in RecoverJavaEnumMembersByLine: - B-1: The per-line regex was context-free, so uppercase call statements inside an anonymous member body (e.g. `ACTIVATE_HELPER();`) could be emitted as phantom enum members, and their trailing `;` would terminate recovery prematurely and drop subsequent real members. - B-2: Dedup was keyed on StartLine. The primary scanner stamps StartLine at the annotation line while recovery stamps the member-name line, so the same member could be double-emitted. Fix: - Track brace depth across the enum body using TryConsumeJavaNonCode so strings, char literals, comments, and text blocks don't corrupt the depth. Only apply the line regex when the line starts at braceDepth == 0 and the scanner mode is Normal. - Terminate recovery on a top-level `;` (braceDepth == 0) detected via the char scan, rather than a naive line-suffix check. - Dedup by member name within the enum container. Java enum member names are unique by language rule, so this is both safer and simpler than aligning StartLine bases between the primary scanner and the recovery pass. Regression tests: - Extract_Java_RecoveryIgnoresLinesInsideAnonymousMemberBody - Extract_Java_RecoveryDedupsByNameAcrossAnnotationStartLines Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
JavaEnumMemberNameRegex was ASCII-only ([A-Za-z_$][A-Za-z0-9_$]*), which
truncated enum member names like RÉSUMÉ or NAÏVE at the first non-ASCII
character — the extractor would report `R` or `NA` instead of the full
identifier. Java allows Unicode identifiers (JLS §3.8).
Widen the regex to match JLS identifier rules:
- Start: \p{L} (letter), \p{Nl} (letter number), underscore, dollar
- Continue: \p{L}\p{Nl}\p{Nd}\p{Mn}\p{Mc}\p{Pc} plus underscore and dollar
Regression test: Extract_Java_DetectsUnicodeEnumMembers covers RÉSUMÉ
and NAÏVE, and asserts that the pre-fix truncated names `R`/`NA` are
not emitted.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
FindBraceRange (the BodyStyle.Brace default) was a dumb {/}
counter with no string/comment/text-block awareness. A `}` inside
a Java text block (`"""..."""`) or a regular string literal would
be counted as a closing brace and truncate the enum body range
early, dropping every member declared after the literal.
Add FindJavaBraceRange, dispatched from ResolveRange for Java, that
reuses the TryConsumeJavaNonCode lexer state machine to skip
strings, char literals, line/block comments, and text blocks. Matches
the existing FindCSharpBraceRange / FindJavaScriptBraceRange pattern.
Regression tests:
- Extract_Java_HandlesTextBlockContainingBrace (text block with `}`)
- Extract_Java_HandlesStringContainingBrace (regular string with `}`)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The [Unreleased] entry for #364/#292 only mentioned the initial body-scoped scanner swap. Document the follow-up refinements landed during adversarial review: - Lex-aware annotation skip across quoted parens / block comments / text blocks. - Bounded recovery pass for malformed annotation bodies. - Brace-depth-aware recovery with member-name dedup. - Unicode (JLS §3.8) identifiers. - FindJavaBraceRange for enum body ranges that survives `}` inside text blocks and quoted strings. Expand both the English and Japanese test lists to name the new regression tests (annotation lex-awareness, recovery edge cases, anonymous-body guard, Unicode members, text block and quoted-brace body-range checks). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…tab-indented-enum # Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
:127regex requires\s{2,}but a tab is only 1 character #364): the Java enum-member regex required^\s{2,}indentation, so a single tab of indentation (common with EditorConfigindent_style=tabor legacy IDE defaults) silently dropped every Java enum member. The enum shell surfaced as aColorsymbol butRED/GREEN/BLUEwere invisible tosymbols,definition,references,callers, andcallees.enummember pattern over-matches: anyUPPERCASE(...);call statement becomes a phantomfunctiondefinition — fires outside enum bodies and inside text blocks #292): naively relaxing the leading whitespace to^\s+would also start catching tab-indented method calls like\tRED();inside a class body as phantom enum members, worsening Javaenummember pattern over-matches: anyUPPERCASE(...);call statement becomes a phantomfunctiondefinition — fires outside enum bodies and inside text blocks #292.ExtractJavaEnumMembers) modeled on the existing C# approach. The scanner walks the enum body tracking strings, char literals, line/block comments, Java 15+ text blocks, parens, brackets, and nested braces (for anonymous member bodies), emits each member at top-level,boundaries, and stops at the first top-level;.@Annotation(...)prefixes are skipped using the same lexer state machine, so annotations with quoted parens / block comments / text blocks no longer derail member binding.@Deprecated A(1)still binds toA.FindJavaBraceRange): the enum body range itself is now resolved by a new Java-specific range resolver that reuses the same lexer state machine, so a}inside a text block or quoted string no longer prematurely closes the enum body and drops every member after the literal.@Ann(spanning a partial edit), a bounded line-regex recovery pass emits obvious uppercase-identifier members. Recovery tracks brace depth across the body so lines inside anonymous member bodies or methods aren't mis-emitted as phantom members, dedups by member name (the primary scanner stampsStartLineat the annotation line while recovery stamps the member-name line, so StartLine-based dedup would double-emit), and terminates on a top-level;.RÉSUMÉorNAÏVEare captured intact instead of truncated at the first non-ASCII character.Test plan
dotnet build src/CodeIndex— clean (0 warnings, 0 errors)dotnet test— 1619 passed, 0 failed, 2 skipped (performance tests)Extract_Java_DoesNotExtractMethodCallsAsEnumMemberspins tab-indentedRED();/GREEN();inside a class body are not symbolizedExtract_Java_StopsEnumMembersAtSemicolonpins declarations after the first top-level;are not added as enum membersExtract_Java_HandlesAnnotationWithQuotedParen/Extract_Java_HandlesBlockCommentBetweenAnnotationAndMemberpin the lex-aware annotation skipExtract_Java_RecoversMembersWhenAnnotationIsMalformed/Extract_Java_HandlesEmptyEnumBody/Extract_Java_HandlesEnumWithOnlySemicolon/Extract_Java_HandlesTrailingComma/Extract_Java_HandlesAnonymousMemberBodypin the bounded recovery and edge-case body shapesExtract_Java_RecoveryIgnoresLinesInsideAnonymousMemberBody/Extract_Java_RecoveryDedupsByNameAcrossAnnotationStartLinespin the brace-depth-aware recovery and name-based dedupExtract_Java_DetectsUnicodeEnumMemberspins Unicode member names per JLS §3.8Extract_Java_HandlesTextBlockContainingBrace/Extract_Java_HandlesStringContainingBracepin the lex-aware body-range resolutionCloses #364.
Closes #292.
🤖 Generated with Claude Code