[core] Clarify API for ANTLR based languages

[adangel]

Part of #2499

• We should not expose antlr specific API into the PMD's node API (this is addressed by [core,swift] Refactor antlr implementation #2463)

• AntlrTokenizer is internal but should be impl, because it is needed to implement a language
  based on antlr - at least AntlrTokenizer.getCharStreamFromSourceCode

• AntlrBaseRule uses already a different concept than JavaccBasedRules
  (where is the visitor or is the rule a visitor?). This indicates missing work on API design
  in general.

  • Those rules use a visitor instead of being a visitor. Having Javacc rules implement the interface is the bad design, which led to duplication of thousands of lines:

  pmd/pmd-java/src/main/java/net/sourceforge/pmd/lang/java/rule/AbstractJavaRule.java

  Lines 217 to 221 in ca0f653

  	//
  	// The following APIs are identical to those in JavaParserVisitorAdapter.
  	// Due to Java single inheritance, it preferred to extend from the more
  	// complex Rule base class instead of from relatively simple Visitor.
  	//

  Note that because of [core] Add generic visitor interface in pmd-core #2589, we don't actually need language-specific base rule classes anymore. We could just have an AbstractVisitorRule on the model of the Antlr one (the antlr one doesn't need to exist), with a buildVisitor() method that returns an AstVisitor<RuleContext, ?>.

• Open question: How can we extend the Antlr nodes (e.g. adding additional getters/attributes to expose some information for XPath)? The issue is, that they're all generated into the same file...

  • Moved to [doc] Verify ANTLR Documentation #2501 as this for now just needs to be documented

• How does the impl API look like for Antlr-based languages? -> that leads to the documentation

• This task probably depends on PMD 7.0.0 API

• This task creates another refactoring task (implementing the proposed API).