Skip to content

Fix several Scala parse-print idempotency issues found on Apache Flink#8174

Merged
knutwannheden merged 4 commits into
mainfrom
dizzy-wombat
Jul 7, 2026
Merged

Fix several Scala parse-print idempotency issues found on Apache Flink#8174
knutwannheden merged 4 commits into
mainfrom
dizzy-wombat

Conversation

@knutwannheden

Copy link
Copy Markdown
Contributor

Motivation

Parsing the Apache Flink codebase (Scala 2) with rewrite-scala produced 44 parse failures ("is not print idempotent") across 42 files. Triaging them revealed five distinct parser bugs. All 1017 Scala files in apache/flink now round-trip with zero parse errors and zero print diffs.

Summary

  • Argument prefix doubling in new Foo(...) calls — whitespace after a comma was captured into argPrefix without advancing the cursor past it, so the argument's own visit consumed it again into a nested prefix. Ordinary arguments masked this because withPrefix overwrites the prefix at the same level, but for J.TypeCast arguments (asInstanceOf) the duplicate sits on the inner qualifier, printing new MyPojo(a.asInstanceOf[Int], b) with a doubled space (or a doubled newline+indent for multi-line argument lists). The cursor now advances to the argument start, as the first-argument branch already did.
  • Backticks dropped from package segmentspackage a.b.`trait` printed without backticks because the package expression was rebuilt from compiler name strings, which strip them. ScalaASTConverter now derives the package name from the pid source span, for both the file-header and braced-package forms.
  • Trailing ; dropped — two defects in visitBlock: (a) dotty extends some statement spans to include the trailing ; (e.g. when a val's rhs sits on its own line), which the forward scan couldn't see; the absorbed ; is now recognized the same way consumeTrailingSemicolon does, including a cursor catch-up so the ; can't leak into whitespace; (b) a braced block's final expression had no semicolon handling at all — its ; is now consumed and preserved via the Semicolon marker. Braceless blocks (e.g. match-case bodies) are excluded: there the ; is the enclosing construct's separator and is preserved there.
  • enum/given as identifiers produced error trees — valid Scala 2 names are hard keywords in Scala 3, so files like def qualifyEnum(enum: Enum[_]) printed with <error> fragments. ScalaCompilerBridge.parseOne now retries a file under -source 3.0-migration — where dotty's scanner demotes enum/given/erased to identifiers — whenever the default 3.6-migration parse has syntax errors, keeping the first attempt if the retry doesn't parse cleanly either. Each attempt parses under its own reporter so a failed attempt's errors don't surface on a file the retry parsed fine. A global source-version switch was not an option: 3.0-migration disables ≥3.1 parser features such as fewerBraces colon-arguments, which this parser supports.
  • Scala 2 do { ... } while (cond) printed garbled — dotty desugars do-while into WhileDo(Block(body, cond), ()), which printed as whiledo {{ ... } while (...)}do. The desugared shape (span starting at the do keyword) is now detected and mapped to J.DoWhileLoop.

Test plan

  • New round-trip tests for each bug: TypeCastTest#castAsFirstOfMultipleArguments, NewClassTest#newClassWithMultiLineChainedArgument, CompilationUnitTest#packageWithBacktickedSegment, BlockTest#trailingSemicolonAfterMultiLineInitializer, Scala2CompatTest#enumAsIdentifier / #givenAsIdentifier, ControlFlowTest#doWhileLoop
  • ./gradlew :rewrite-scala:test green
  • Round-tripped all 1017 Scala files in apache/flink: zero parse errors, zero print diffs

https://claude.ai/code/session_01R2LCngRXpEq7cuF6wu8XmE

Five distinct parser bugs surfaced when parsing the Apache Flink codebase
(Scala 2) with rewrite-scala:

- Argument prefix doubling in `new Foo(...)` calls: whitespace after a comma
  was captured into argPrefix without advancing the cursor past it, so the
  argument's visit consumed it again into a nested prefix. Visible for
  J.TypeCast arguments (`asInstanceOf`), where withPrefix can't replace the
  nested copy.
- Backticks dropped from package segments (`package a.b.`trait``): the package
  expression was rebuilt from compiler name strings, which strip backticks.
  Now derived from the source span.
- Trailing `;` dropped: dotty extends some statement spans over the trailing
  `;`, which visitBlock's forward scan couldn't see; and a braced block's
  final expression had no semicolon handling at all.
- `enum`/`given` used as identifiers (valid Scala 2) produced error trees:
  parseOne now retries a file under `-source 3.0-migration` (where dotty's
  scanner demotes enum/given/erased to identifiers) when the default
  3.6-migration parse has syntax errors.
- Scala 2 `do { ... } while (cond)` printed garbled: dotty desugars it to
  `WhileDo(Block(body, cond), ())`; now detected and mapped to J.DoWhileLoop.

Verified by round-tripping all 1017 Scala files in apache/flink: zero parse
errors, zero print diffs.

Claude-Session: https://claude.ai/code/session_01R2LCngRXpEq7cuF6wu8XmE
doesNotLeakTempDirectories compared before/after snapshots of the shared
java.io.tmpdir, so rewrite-scala temp dirs created (and later deleted) by
concurrently running test classes were flagged as leaks. Poll until
transient dirs disappear; genuine leaks persist and still fail the test.

Claude-Session: https://claude.ai/code/session_01R2LCngRXpEq7cuF6wu8XmE
A backticked package segment like `trait` is source syntax only — the
package name is `a.b.trait`. Build the package expression with
TypeTree.build's escape overload so the identifier's simple name is the
bare name with a Quoted marker, and render the backticks from the marker
in ScalaPrinter. J.Package.getPackageName() now returns the real package
name, so recipes matching on it work.

Claude-Session: https://claude.ai/code/session_01R2LCngRXpEq7cuF6wu8XmE
@knutwannheden knutwannheden merged commit 77bd1ee into main Jul 7, 2026
1 check passed
@knutwannheden knutwannheden deleted the dizzy-wombat branch July 7, 2026 21:39
@github-project-automation github-project-automation Bot moved this from In Progress to Done in OpenRewrite Jul 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant