[core] Text documents epic

[oowekyala]

This is a task list for the series of PRs I mentioned in #693 (comment). These are for PMD 7

• [core] Analysis listeners #3692
  • [core] Deprecations for analysis listeners #3762 (merge this before)
• [core] Text documents #3893
  • [core] Add file collector and new programmatic API for PMD #3785 (merge this before)
  • What:
    • Replace the current document API which is dead code
    • Introduce TextFile to replace DataSource
    • Revamp initial file collection phase (internals of the FileProcessor of [core] Add file collector and new programmatic API for PMD #3785)
    • Introduce Chars, a string slice type. This allows us to share the char array of the source file, and not duplicate it when eg we parse a subfile in an embedded language. It also enables more efficient analysis of eg comments.
  • Why:
    • Bridge the gap between CPD's SourceCode and PMD's DataSource
    • Have a higher-level abstraction stack than DataSource (eg data source provides only byte streams, we want to work on character streams)
    • TextFile is both readable and writable, a prerequisite to [core] Autofixes feature development #693
    • FileLocation for [java] Single Line Warnings #730
    • Avoid unnecessary string copies
• [core] Text documents escapes #4044 Revamp Javacc infrastructure
  • What:
    • Rewrite CharStream so as not to rely on javacc-generated sources in pmd-core
    • Build CharStream on top of TextDocuments as a first-class API
  • Why:
    • Control the support for JavaCC-generated languages entirely, without relying on javacc-generated sources
    • Exploit TextDocuments and Chars correctly, to avoid string copies
    • Finish [core] Merging Javacc build scripts #2239
    • Fix some problems in CPD tokenizers related to string escape
• [java] Cleanup handling of Java comments #4237
  • What:
    • Stabilize how Comment nodes are represented
    • Use Chars properly in comments to avoid string copies in rules
  • Why:
    • A todo item leftover from the initial grammar revamp

Remaining tasks

• [java] Add Javadoc parsing #4325

[adangel]

I'm closing this as done. The remaining task (#4325) is tracked separately.