JDK-8298405: Support Markdown in the standard doclet #11701

jonathan-gibbons · 2022-12-15T23:47:45Z

Support for Markdown comments in the standard doclet.

To enable Markdown in a comment, start the comment with /**md followed by whitespace. The syntax is as defined for CommonMark.

The work is in 3 parts:

Update the Compiler Tree API to support Markdown tree nodes, containing strings of (uninterpreted) Markdown source code.
Import commonmark-java into the jdk.javadoc module, to be able to convert Markdown strings to HTML.
Update the standard doclet, to leverage the preceding two parts, to translate Markdown in documentation comments to Content nodes.

There are new tests both for the low level work in the Compiler Tree API, and for the overall high-level work in the doclet.

Background info: https://mail.openjdk.org/pipermail/javadoc-dev/2023-January/005563.html

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue
Change requires CSR request JDK-8298687 to be approved

Issues

JDK-8298405: Support Markdown in the standard doclet
JDK-8298687: Support Markdown in the standard doclet (CSR)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk pull/11701/head:pull/11701
$ git checkout pull/11701

Update a local copy of the PR:
$ git checkout pull/11701
$ git pull https://git.openjdk.org/jdk pull/11701/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 11701

View PR using the GUI difftool:
$ git pr show -t 11701

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/11701.diff

bridgekeeper · 2022-12-15T23:48:39Z

👋 Welcome back jjg! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2022-12-15T23:51:28Z

@jonathan-gibbons The following labels will be automatically applied to this pull request:

compiler
javadoc

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

mlbridge · 2023-01-03T19:16:44Z

Webrevs

pavelrappo

This is a welcome change and impressive work! My initial comments are inline.

src/jdk.compiler/share/classes/com/sun/source/doctree/DocTree.java

src/jdk.compiler/share/classes/com/sun/source/doctree/InheritDocTree.java

src/jdk.compiler/share/classes/com/sun/source/doctree/MarkdownTree.java

src/jdk.compiler/share/classes/com/sun/source/doctree/package-info.java

src/jdk.compiler/share/classes/com/sun/tools/javac/tree/DocPretty.java

src/jdk.javadoc/share/classes/jdk/internal/org/commonmark/internal/util/Html5Entities.java

test/langtools/tools/javac/doctree/MarkdownTest.java

src/jdk.javadoc/share/classes/jdk/javadoc/internal/doclets/formats/html/HtmlDocletWriter.java

jonathan-gibbons · 2023-01-04T18:37:56Z

This is a welcome change and impressive work! My initial comments are inline.

Thanks; I think of it as the icing on the cake of the technical-debt cleanup we have been doing over the past few releases.

pavelrappo · 2023-01-04T22:05:10Z

In case you've missed test failures in your GitHub Actions (GHA): test/langtools/tools/javac/lib/DPrinter.java. A trivial fix, really.

src/jdk.compiler/share/classes/com/sun/source/doctree/DocTreeVisitor.java

AlanBateman · 2023-01-05T17:11:29Z

This seems a significant feature with several design choices that probably should be captured somewhere. Is is significant enough to have a JEP?

Add legal header to imported CommonMark source files Always use Text nodes inside AttributeTree values Unwrap <p> from "simple" paragraphs

jonathan-gibbons · 2023-01-07T01:05:40Z

In case you've missed test failures in your GitHub Actions (GHA): test/langtools/tools/javac/lib/DPrinter.java. A trivial fix, really.

Fixed.

Share Markdown parser and renderer in instance of MarkdownHandler

src/jdk.javadoc/share/classes/jdk/internal/org/commonmark/internal/Bracket.java

pavelrappo · 2023-01-09T17:55:05Z

src/jdk.internal.md/share/classes/jdk/internal/org/commonmark/internal/util/Html5Entities.java

+public class Html5Entities {
+
+    private static final Map<String, String> NAMED_CHARACTER_REFERENCES = readEntities();
+    private static final String ENTITY_PATH = "/org/commonmark/internal/util/entities.txt";


I see that you've added the missing entities.properties file, but renamed it to entities.txt. IIRC from our offline chat, it's due to how JDK build treats .properties files.

I wonder what would be better: (a) to keep the original extension, but amend the build to ignore this file, or (b) to do what you've done and possibly suggest a PR for CommonMark to do the same.

On the one hand, it's not a true Java .properties file as one might've though from its extension. On the other hand, this change of yours diverge us from the original snapshot of CommonMark.

I like your option (b)

retain the change to use entities.txt

report the issue to CommonMark, if not actually suggest a PR.

FWIW, the rename is handled automatically by the process I use to import the source code. I think that automated fix-up is acceptable; manual fix-up that would be needed every time we update the source would not be acceptable.

wimdeblauwe · 2023-01-21T11:55:15Z

Is the work here done in a way that asciidoc ( https://docs.asciidoctor.org/ ) support is also possible?

ExE-Boss · 2023-01-31T06:47:55Z

There should probably be an option to make Markdown the default.

pavelrappo · 2023-01-31T23:26:41Z

@wimdeblauwe, not sure if you've noticed it, but your comment was hidden: #11701 (comment). To make the comment visible and response possible, consider accepting TOU suggested by the bot. Thanks.

wimdeblauwe · 2023-02-01T06:41:51Z

@wimdeblauwe, not sure if you've noticed it, but your comment was hidden: #11701 (comment). To make the comment visible and response possible, consider accepting TOU suggested by the bot. Thanks.

Thanks for letting me know! Should be ok now.

jonathan-gibbons · 2023-02-07T23:27:31Z

FYI, draft JEP: https://openjdk.org/jeps/8299906

jonathan-gibbons · 2023-02-07T23:31:30Z

There should probably be an option to make Markdown the default.

That would lead to configuration complexity in any environment that had to support different kinds of comments in different libraries/modules/packages/files

jonathan-gibbons · 2023-02-07T23:58:16Z

Is the work here done in a way that asciidoc ( https://docs.asciidoctor.org/ ) support is also possible?

Yes, and no. But mostly no.

Yes, in the sense that there are aspects of this work that might allow asciidoclet to be enhanced. And, the general methodology of this work is not specific to the details of Markdown.

No, in the sense that the proposal does not include the Standard Doclet supporting pluggable backends for different markup languages, and I don't think we want to go there, at least to begin with. I think it is more interesting to focus on getting broader support across the ecosystem for a limited set of markup languages than an open-ended set.

jimisola · 2023-02-08T06:04:51Z

Would also really like to see AsciiDoc support. Our development team use it for everything. Markdown is ok but AsciiDoc is much better for technical documentation.

What is necessary for AsciiDoc to be considered? Or rather, what is the process for it being taken into consideration?

pavelrappo · 2023-02-08T10:16:12Z

@jimisola, not sure if you've noticed it, but your comment was hidden: #11701 (comment). To make the comment visible and response possible, consider accepting TOU suggested by the bot. Thanks.

Have you seen that notification from the bot? Regardless, the bot might need to use a better message/format for such communication.

wimdeblauwe · 2023-02-08T11:09:49Z

I had the same issue with not noticing the question from the bot. I think the main problem is that there is no email notification of that message. So unless you manually check the ticket, you don't notice it.

magicus · 2023-02-08T13:21:29Z

@wimdeblauwe Let's not turn this PR into a discussion about Skara (the Github bots of OpenJDK), but just a short explanation: There is no way we can get the email address of a random user on Github coming by and making a comment; users emails are generally not available through Github for privacy reasons. So we just can't do that.

However, we do tag the user, and they should get a Github notification for that. If they have "send email on notification" turned on, they will get an email as well.

But I think the message could perhaps be a bit more explicit about their original content being hidden. This is not immediately clear if you just read the start of the message. I'll open an enhancement request on Skara for that.

jimisola · 2023-02-21T05:22:18Z

Have you seen that notification from the bot? Regardless, the bot might need to use a better message/format for such communication.

No, thanks. I missed that. Accepted now.

Great that this got implemented. Would have been even better with asciidoc support 😉

jonathan-gibbons · 2023-02-21T17:06:59Z

FYI, the PR has been withdrawn for now, pending additional design discussions.

lfvjimisola · 2023-05-30T13:31:05Z

Hi @lfvjimisola, thanks for making a comment in an OpenJDK project!

All comments and discussions in the OpenJDK Community must be made available under the OpenJDK Terms of Use. If you already are an OpenJDK Author, Committer or Reviewer, please click here to open a new issue so that we can record that fact. Please Use "Add GitHub user lfvjimisola for the summary.

If you are not an OpenJDK Author, Committer or Reviewer, simply check the box below to accept the OpenJDK Terms of Use for your comments.

I agree to the OpenJDK Terms of Use for all comments I make in a project in the OpenJDK GitHub organization.

Your comment will be automatically restored once you have accepted the OpenJDK Terms of Use.

koppor · 2024-01-02T14:38:02Z

Thank you so much for the work @jonathan-gibbons and @lahodaj (for the jshell part).

I can see the need for an universal solution allow to handle both Markdown and ASCIIDoc. However, offering Markdown would be a huge improvement compared to the existing solution (having to use HTML and IDE support took a long time to fix "trivial" things - e.g., proper HTML reformat https://youtrack.jetbrains.com/issue/IDEA-147601).

JDK-8298405: Markdown support in the standard doclet

2462e77

jonathan-gibbons marked this pull request as draft December 15, 2022 23:47

openjdk bot added javadoc javadoc-dev@openjdk.org compiler compiler-dev@openjdk.org csr Pull request needs approved CSR before integration labels Dec 15, 2022

fix whitespace

f9e2453

jonathan-gibbons marked this pull request as ready for review January 3, 2023 19:08

openjdk bot added the rfr Pull request is ready for review label Jan 3, 2023

pavelrappo reviewed Jan 4, 2023

View reviewed changes

pavelrappo reviewed Jan 5, 2023

View reviewed changes

src/jdk.compiler/share/classes/com/sun/source/doctree/DocTreeVisitor.java Outdated Show resolved Hide resolved

Update to CommonMark 0.21.

fc64496

jonathan-gibbons changed the title ~~JDK-8298405: Markdown support in the standard doclet~~ JDK-8298405: Support Markdown in the standard doclet Jan 5, 2023

jonathan-gibbons added 2 commits January 6, 2023 15:56

Always use Text nodes inside AttributeTree values

4df36c9

Move CommonMark to new internal module.

b801b47

Add legal header to imported CommonMark source files Always use Text nodes inside AttributeTree values Unwrap <p> from "simple" paragraphs

jonathan-gibbons added 2 commits January 7, 2023 08:44

Rename FFFC variable

88373a1

Share Markdown parser and renderer in instance of MarkdownHandler

Update copyright years

82e363a

turbanoff reviewed Jan 9, 2023

View reviewed changes

src/jdk.javadoc/share/classes/jdk/internal/org/commonmark/internal/Bracket.java Outdated Show resolved Hide resolved

pavelrappo reviewed Jan 9, 2023

View reviewed changes

openjdk bot mentioned this pull request Jan 11, 2023

8299902: Support for MarkDown javadoc in JShell #11936

Closed

3 tasks

Merge with upstream/master

3f45c98

Rename MarkdownTree to RawTextTree

432ae0b

Merge with upstream/master

3acc758

jonathan-gibbons added 2 commits February 8, 2023 14:15

Merge with upstream/master

59719f1

Revert changes to Object.hashCode

e1f2958

jonathan-gibbons closed this Feb 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JDK-8298405: Support Markdown in the standard doclet #11701

JDK-8298405: Support Markdown in the standard doclet #11701

jonathan-gibbons commented Dec 15, 2022 •

edited by openjdk bot

Loading

bridgekeeper bot commented Dec 15, 2022

openjdk bot commented Dec 15, 2022

mlbridge bot commented Jan 3, 2023 •

edited

Loading

pavelrappo left a comment

jonathan-gibbons commented Jan 4, 2023

pavelrappo commented Jan 4, 2023

AlanBateman commented Jan 5, 2023

jonathan-gibbons commented Jan 7, 2023

pavelrappo Jan 9, 2023 •

edited

Loading

jonathan-gibbons Jan 9, 2023 •

edited

Loading

wimdeblauwe commented Jan 21, 2023 •

edited by bridgekeeper bot

Loading

ExE-Boss commented Jan 31, 2023

pavelrappo commented Jan 31, 2023

wimdeblauwe commented Feb 1, 2023

jonathan-gibbons commented Feb 7, 2023

jonathan-gibbons commented Feb 7, 2023

jonathan-gibbons commented Feb 7, 2023 •

edited

Loading

jimisola commented Feb 8, 2023 •

edited by bridgekeeper bot

Loading

pavelrappo commented Feb 8, 2023

wimdeblauwe commented Feb 8, 2023

magicus commented Feb 8, 2023

jimisola commented Feb 21, 2023 •

edited

Loading

jonathan-gibbons commented Feb 21, 2023

lfvjimisola commented May 30, 2023 •

edited by bridgekeeper bot

Loading

koppor commented Jan 2, 2024

JDK-8298405: Support Markdown in the standard doclet #11701

JDK-8298405: Support Markdown in the standard doclet #11701

Conversation

jonathan-gibbons commented Dec 15, 2022 • edited by openjdk bot Loading

Progress

Issues

Reviewing

bridgekeeper bot commented Dec 15, 2022

openjdk bot commented Dec 15, 2022

mlbridge bot commented Jan 3, 2023 • edited Loading

Webrevs

pavelrappo left a comment

Choose a reason for hiding this comment

jonathan-gibbons commented Jan 4, 2023

pavelrappo commented Jan 4, 2023

AlanBateman commented Jan 5, 2023

jonathan-gibbons commented Jan 7, 2023

pavelrappo Jan 9, 2023 • edited Loading

Choose a reason for hiding this comment

jonathan-gibbons Jan 9, 2023 • edited Loading

Choose a reason for hiding this comment

wimdeblauwe commented Jan 21, 2023 • edited by bridgekeeper bot Loading

ExE-Boss commented Jan 31, 2023

pavelrappo commented Jan 31, 2023

wimdeblauwe commented Feb 1, 2023

jonathan-gibbons commented Feb 7, 2023

jonathan-gibbons commented Feb 7, 2023

jonathan-gibbons commented Feb 7, 2023 • edited Loading

jimisola commented Feb 8, 2023 • edited by bridgekeeper bot Loading

pavelrappo commented Feb 8, 2023

wimdeblauwe commented Feb 8, 2023

magicus commented Feb 8, 2023

jimisola commented Feb 21, 2023 • edited Loading

jonathan-gibbons commented Feb 21, 2023

lfvjimisola commented May 30, 2023 • edited by bridgekeeper bot Loading

koppor commented Jan 2, 2024

jonathan-gibbons commented Dec 15, 2022 •

edited by openjdk bot

Loading

mlbridge bot commented Jan 3, 2023 •

edited

Loading

pavelrappo Jan 9, 2023 •

edited

Loading

jonathan-gibbons Jan 9, 2023 •

edited

Loading

wimdeblauwe commented Jan 21, 2023 •

edited by bridgekeeper bot

Loading

jonathan-gibbons commented Feb 7, 2023 •

edited

Loading

jimisola commented Feb 8, 2023 •

edited by bridgekeeper bot

Loading

jimisola commented Feb 21, 2023 •

edited

Loading

lfvjimisola commented May 30, 2023 •

edited by bridgekeeper bot

Loading