[java] Breaking Java Grammar changes for PMD 7.0.0

[adangel]

The goal is, to resemble the official Java Language Specification with PMD's own java grammar.
This makes it easier to maintain the grammar in the future and avoids special solution inside PMD.

Scheduled

• Deal with ExplicitReceiverParameters explicitely in the grammar, see JLS §8.4.1. Refs [java] Better handle explicit receiver parameters #1017
  • [java] Separate receiver parameter from formal parameter  #1980
  • 121d4c7
• Add node for anonymous class declarations, refs [java] Add new node for anonymous class declaration #905
• Separate FormalParameter from catch clause formal parameters (see JLS §14.20) and other improvements around FormalParameter, refs [java] AST inconsistencies around FormalParameter #998
• Improvements to LocalVariableDeclaration and FieldDeclaration, refs [java] Improve ASTLocalVariableDeclaration #1128
• For ShiftExpression, it makes no sense to have these extra
  RUNSIGNEDSHIFT and RSIGNEDSHIFT, which only match the token anyway. They could
  set the image on their parent, but shouldn't be pushed on the stack... we can
  use a #void production for that, like in oowekyala/pmd@87f05fc
• Parsing ASTClassOrInterfaceType recursively to preserve annotation and type parameter location, refs [java] ClassOrInterfaceType AST improvements #1150. (Left recursive parsing is forbidden by javacc but we can build a "left-recursive node" by manipulating the jjtree stack
• Make ImportDeclaration regular node instead of TypeNode
  • [java] Remove Name nodes in Import- and PackageDeclaration #1888
  • 303b962
• Make FieldDeclaration regular node instead of TypeNode
  • [java] Leftovers about Dimensionable & type annotations #2203
  • 1fefbd2
• Make ASTType and ASTReferenceType interfaces, make new node ArrayType -> helps type resolution, easier to specialize the API
• Use an ArrayDimensions node to represent array dimensions on ArrayType, VariableDeclaratorId, MethodDeclarator and other contexts instead of Dimensionable
  • [java] New expression and type grammar #1759
  • [java] Leftovers about Dimensionable & type annotations #2203
  • 6378080

Proposed

• Split ForStatement into ForStatement and ForeachStatement
  • b03e233
  • [java] Improve statement grammar #2164
• Make ASTLiteral an interface, use a different node for each type of literal (integer,
• Merge UnaryExpression and UnaryExpressionNotPlusMinus. The two productions are needed for correct parsing, but there's no need to have two different nodes, especially since they have the same precedence.
• Add a VariableDeclaratorId under enum constant declaration. This would consistently make VariableDeclaratorId represent a declaration of any value (field, parameter, local var, enum constant, etc).
• Clarify parsing of PrimaryPrefix/Suffix when using "this" or "super". See also [java] New rule: AvoidReassigningLoopVariables #1530 (comment)

Remaining open tasks

• Remove setters from AccessNode interface and make FormalParameter, Resource and LocalVariableDeclaration not implement it anymore
  • tracked in [java] AccessNode API changes #1307

[jsotuyod]

@oowekyala should we merge #905 into this?

[oowekyala]

@jsotuyod I think adding that issue to the checklist is a good idea, but I'd prefer the issue to stay open. It gives room for clearer discussions/ easier referencing. Same for #998. I'll do that

I'd really love the grammar for expressions to be revamped for 7.0.0 too (#497), but I get a feeling this would have been a task for a GSoC given the number of rules and other clients to update. It also interacts with possible typeres GSoCs so it's hard to implement before the end of the summer

[adangel]

I'm closing this - the tasks are collected on https://github.com/pmd/pmd/wiki/PMD-7.0.0-Java