[doc] Document API evolution principles as ADR

[oowekyala]

The actual API development and marking some part of the API as internal or add new API is an ongoing task, that will need to be done everytime. We won't just define an API and then are done with it. The API will change as new features want to be implemented.

So we should only make sure we document

• what the criteria is for a public API
  • clear-cut between public APIs and internal
  • which packages to use, e.g. internal for internal "API"
  • when to use package impl
• how to deprecate and remove old APIs
  • how long to support a given API
  • with respect to semantic version
• when to define a new API as experimental
• when to promote a experimental API as stable
• annotations to use, see also https://github.com/pmd/pmd/blob/master/docs/pages/next_major_development.md#new-api-support-guidelines
• guidelines for AST classes (package private ctor, final, etc.)

The documentation should be part of the "Architecture Decision Records" -> https://pmd.github.io/latest/pmd_projectdocs_decisions.html

---

Old info

(Low priority for now, but should not be lost)

Original discussion, with the main ideas repeated here:

I think at some point we need to clear-cut which APIs are public and which are not (ie: by using internal packages as Gradle does). Up until then, developers may assume anything is at their disposition.

RxJava also uses some specific annotations (@Experimental, @Beta). Maybe we could even have a @ReservedSubclassing one, to express that a class or interface is not meant to be subclassed by external clients, and that its inheritance-specific members are private API, while public members are still public API. Which would be useful for all AST related interfaces for example.

maybe, concrete AST node classes should have a package-private constructor, and be final. They're not meant to be instantiated by hand, but that could break some tests and rules

Clearly defining the API PMD provides, makes sense and will help, if we at some time in the future want to modularize PMD using java9. That would prevent API leaking then at compile time already...

Edit: I think none of pmd-ui should be considered public API, for the unforeseable future

I think we could also talk about an approximate schedule, eg

• Do we announce which APIs will be made private and deprecate them in a minor release (eg 6.3.0) and make the actual changes in the next major release (7.0.0)?
• Do we announce which APIs will be made private and deprecate them in a major release (eg 7.0.0) and make the actual changes in the next major release (8.0.0)?
• A bit of both, depending on the level of "breaking" we're talking about? E.g. deprecating the subclassing of node classes and such in a minor release, and making related breaking changes in 7.0.0, while keeping the moving of private API into an internal package for 8.0.0

[jsotuyod]

Also of note, we should probably skip internal classes from javadoc altogether. The maven Javadoc plugin supports this easily https://maven.apache.org/plugins/maven-javadoc-plugin/examples/exclude-package-names.html

[adangel]

Discussion from today:

• Not public API:

  • Inheritance-specific members of AST related classes & interfaces. eg adding a member to an interface shouldn’t be considered api breaking
  • Setters only used in the parser
  • Exceptions… (I’ve seen JaxenException in the Node interface…)
  • XPath, AttributeAxisIterator
  • RuleSetFactory

• public API

  • AST structure
  • Language Symbol Table / Metrics / Type Resolution …(Not the implementation!)
  • Implementing custom Rules (AbstractRule, concrete Properties, …)
  • Renderers
  • Executing PMD, Ant Task, Configuration
  • RuleSet XML

• Javadoc documentation only for public API

• New Features, maybe @Incubating or @Experimental at first

• Use @Deprecated during the transition (from the point we announce decide which APIs will be non-public until 7.0.0). Update the 7.0.0 branch as we go along

Package structure:

• each public package has sub-package “internal” from 7.0.0 on
  Examples:
• net.sf.pmd
• net.sf.pmd.internal
• net.sf.pmd.ant
• net.sf.pmd.ant.internal (<- already exists)
• net.sf.pmd.lang.ast
• net.sf.pmd.lang.ast.internal

We'll try to use a separate branch for 7.0.0-SNAPSHOT

[adangel]

Discussion from 2018-07-28:

• Suggestion: Start deprecating some parts in master branch
• Maybe @Private annotation to differ between @Deprecated for removal and move to .impl.
• Document deprecated APIs
  • In release notes: only what “newly” deprecated
  • have an extra page in the docs for all deprecations of a major release
    • copy the API changes section of release notes to this page whenever we release (part of the release process)
  • Summary of major releases: big features (something worth to upgrade…), api changes
• Open a first PR adding the @PrivateApi annotation, document in the release notes what this is for, add that to the release process
  • Annotate incrementally
  • Preliminary thought: once we’ve annotated everything, we can start developing PMD 7.0.0

[jsotuyod]

Maybe @Private annotation to differ between @Deprecated for removal and move to .impl.

On the previous meeting we agreed on internal packages. Has that decision been revised, or is this just a wording issue?

[adangel]

Using internal packages is still the plan. We talked about the transition of the source code on the master branch, to get the whole thing started:

• On the master branch (PMD 6 development), we'll mark the classes, that will be internal in PMD 7 as @Deprecated and @PrivateApi. That way, we can in parallel build the API changes doc and sort out already on PMD 6.x, which classes will be gone. And API users get a heads up, what will be gone.
• Once we think, we have marked the majority of classes/packages, we can actually start moving the classes for real - and this will happen on the PMD 7 branch (or, depending on the situation, the master branch, if PMD 6 happens on a maintenance branch then).
• When moving the classes to internal, we can remove both annotations again (or we could keep @PrivateApi for documentation purposes). But since we'll use the javadoc/package configuration, we probably don't need any annotation anymore, just the package internal is then enough.

[adangel]

I'll reuse this issue for: "Document API evolution principles as ADR"

The actual API development and marking some part of the API as internal or add new API is an ongoing task, that will need to be done everytime. We won't just define an API and then are done with it. The API will change as new features want to be implemented.

So we should only make sure we document

• what the criteria is for a public API
• how to deprecate and remove old APIs
• when to define a new API as experimental
• when to promote a experimental API as stable

The documentation should be part of the "Architecture Decision Records" -> https://pmd.github.io/latest/pmd_projectdocs_decisions.html