8248001: javadoc generates invalid HTML pages whose ftp:// links are broken #5198

masyano · 2021-08-20T10:12:11Z

Could you please review the 8248001 bug fixes?

The problem is that javadoc generates invalid HTML pages whose ftp:// links are broken. The fix changes not to use protocol names directly, but to use regular expression.

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8248001: javadoc generates invalid HTML pages whose ftp:// links are broken

Reviewers

Hannes Wallnöfer (@hns - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/5198/head:pull/5198
$ git checkout pull/5198

Update a local copy of the PR:
$ git checkout pull/5198
$ git pull https://git.openjdk.java.net/jdk pull/5198/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 5198

View PR using the GUI difftool:
$ git pr show -t 5198

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/5198.diff

…broken

bridgekeeper · 2021-08-20T10:13:10Z

👋 Welcome back myano! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2021-08-20T10:15:09Z

@masyano The following label will be automatically applied to this pull request:

javadoc

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

…broken

mlbridge · 2021-08-20T10:31:46Z

Webrevs

pavelrappo · 2021-08-20T10:48:10Z

src/jdk.javadoc/share/classes/jdk/javadoc/internal/doclets/formats/html/HtmlDocletWriter.java

-                || lower.startsWith("http:")
-                || lower.startsWith("https:")
-                || lower.startsWith("file:")) {
+        if (text.matches("^[^:/?#]+:.+$")) {


Why not use java.net.URI API to determine if the link contains a scheme?

I assume the link is in an HTML document and goes in an HTML document. If you wanted to use java.net.URI, depending on where from text comes from and whereto it goes, you might need first to decode it using URLDecoder, and then you might need to re-encode it before spitting it out... That's a lot of operations where things could go wrong, especially if the link contains a query string.

That said a stricter regexp (unless I'm mistaken) could be: ^[a-zA-Z][a-zA-Z0-9+\-\.]*:.+$
[ from RFC 2396: scheme = alpha *( alpha | digit | "+" | "-" | "." ) ]

My only concerns were correctness and code reuse. Using an API doesn't require one to read through RFC.

I'd argue for simply adding ftp: as an additional condition (unless there's other interesting URI schemes I'm not thinking of?). If a regex is to be used, I agree it should be much stricter (and defined in a constant, so that the Pattern doesn't need to be compiled on each invocation?).

I would normally opt for a generic regexp-based solution such as proposed by @dfuch, but there is a security aspect to this as well (e.g. script invocation), so I'd go with the more conservative approach here to just add ftp: protocol to the list.

I decided the regex ^[^:/?#]+:.+$ from the description in RFC 2396.

B. Parsing a URI Reference with a Regular Expression The following line is the regular expression for breaking-down a URI reference into its components. ^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))? 12 3 4 5 6 7 8 9 ... Therefore, we can determine the value of the four components and fragment as ... scheme = $2

I agree that adding ftp: is better for the viewpoint of security. However, in addition to ftp, schemes such as javascript and git may be specified, so it's difficult to cover all commonly used schemes.

That regexp will correctly break the URI into its different components but it doesn't guarantee that each of the component is syntactically correct - as further syntax restriction may apply on each of the components.

…broken

masyano · 2021-09-07T12:23:23Z

Thank you for your commnets. I would argue for simply adding ftp:.

masyano · 2021-09-13T11:06:52Z

I pushed the fix, can someone review it?

hns

Thank you for choosing the conservative approach! The fix looks good.

Using actual domain names in the test is problematic. We should use example.com, which has been reserved explicitly for this kind of purpose.

Also, I'm not convinced we have to introduced a new test for this, especially with the rather generic name of "TestHtmlDocletWriter". It seems there is an existing test for the feature in test/langtools/jdk/javadoc/doclet/testHrefInDocComment/TestHrefInDocComment.java. Would it be possible to update/enhance the existing test? If there is a good reason to introduce a new test, it should have a more telling name.

jonathan-gibbons · 2021-09-13T16:00:28Z

src/jdk.javadoc/share/classes/jdk/javadoc/internal/doclets/formats/html/HtmlDocletWriter.java

+                || lower.startsWith("file:")
+                || lower.startsWith("ftp:")) {


Adding ftp: is OK, but given that the method is about modifying relative URLs, a reasonable/preferable alternative would be to use URI.isAbsolute

If a URISyntaxException occurs while creating the URI, I would suggest it should simply return text and not try and modify the text.

The way to use java.net.URI API has already been discussed above. #5198 (comment)

The method using regex has a security problem because it matches patterns other than ftp. So I adopted @hns 's suggestion.

@jonathan-gibbons Could you reply to the above comment?

@masyano I think it's ok this way. After all, this is a simple bug and we have spent more than enough time on bike-shedding. If we want to improve the mechanism by using java.net.URI we can file a separate issue for it.

…broken

masyano · 2021-09-24T11:12:24Z

@hns As you commented, I changed to add test codes in test/langtools/jdk/javadoc/doclet/testHrefInDocComment,
and not to introduce a new test. I modified the test to use www.example.com instead of www.domain.com.

…broken

hns · 2021-09-27T12:34:06Z

@masyano thanks, this helps to not inflate the size of our test suite more than necessary.

What is missing from the updated test is something like the checkHtml method in your original test that verifies the correct links in the generated files. Also, the copyright date in the TestHrefInDocComment.java should be updated to 2021.

@hns As you commented, I changed to add test codes in test/langtools/jdk/javadoc/doclet/testHrefInDocComment,
and not to introduce a new test. I modified the test to use www.example.com instead of www.domain.com.

…broken

masyano · 2021-09-28T08:24:22Z

I updated the copyright date to 2021.

I think checkHtml is not necessary because JavadocTester checks links by checkLinks(). The following is log when the fix has not been applyed.

  Checked 11 files.
  Found 189 references to 54 anchors in 16 files and 6 other URIs.
*      1 missing files
       0 duplicate ids
       0 missing ids
  Schemes
       1 file
       1 ftp
       1 http
       2 https
       1 mailto
  Hosts
       1 www.example.com
       2 docs.oracle.com
       1 www.exsample.com
FAILED: 1 errors found when checking links
        at javadoc.tester.JavadocTester.checkLinks(JavadocTester.java:565)
        at javadoc.tester.JavadocTester.javadoc(JavadocTester.java:384)
        at TestHrefInDocComment.test(TestHrefInDocComment.java:46)

hns

Looks good!

openjdk · 2021-09-30T09:06:56Z

@masyano This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8248001: javadoc generates invalid HTML pages whose ftp:// links are broken

Reviewed-by: hannesw

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 315 new commits pushed to the master branch:

2f955d6: 8273142: Remove dependancy of TestHttpServer, HttpTransaction, HttpCallback from open/test/jdk/sun/net/www/protocol/http/ tests
94e31e5: 8274506: TestPids.java and TestPidsLimit.java fail with podman run as root
a8210c5: 8274401: C2: GraphKit::load_array_element bypasses Access API
dfc557c: 8274406: RunThese30M.java failed "assert(!LCA_orig->dominates(pred_block) || early->dominates(pred_block)) failed: early is high enough"
c0533ef: 8274522: java/lang/management/ManagementFactory/MXBeanException.java test fails with Shenandoah
f8415a9: 8274523: java/lang/management/MemoryMXBean/MemoryTest.java test should handle Shenandoah
355356c: 8273435: Remove redundant zero-length check in ClassDesc.of
97385d4: 8274405: Suppress warnings on non-serializable non-transient instance fields in javac and javadoc
79cebe2: 8274050: Unnecessary Vector usage in javax.crypto
97b2874: 8274509: Remove stray * and stylistic . from doc comments
... and 305 more: https://git.openjdk.java.net/jdk/compare/a522d6b53cd841b4bfe87eac5778c9e5cdf5e90f...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

As you do not have Committer status in this project an existing Committer must agree to sponsor your change. Possible candidates are the reviewers of this PR (@hns) but any other Committer may sponsor as well.

➡️ To flag this PR as ready for integration with the above commit message, type /integrate in a new comment. (Afterwards, your sponsor types /sponsor in a new comment to perform the integration).

hns · 2021-09-30T09:27:01Z

@masyano thanks for the fix, and your patience in revising it. I can sponsor the PR for you.

masyano · 2021-09-30T09:42:09Z

/integrate

openjdk · 2021-09-30T09:44:37Z

@masyano
Your change (at version b68c760) is now ready to be sponsored by a Committer.

masyano · 2021-09-30T09:45:10Z

@hns Thank you for approving and sponsoring.

hns

Sorry, I have to revoke my approval. I just saw there are several unclosed tags in the new test file, and the test fails.

hns · 2021-09-30T10:44:44Z

test/langtools/jdk/javadoc/doclet/testHrefInDocComment/pkg/J1.java

+package pkg;
+
+/**
+ *This class has <a href="{@docRoot}/pkg/J1.html#functions">various functions</


Unclosed a element

Sorry, the last few characters were trimmed. I fixed it.

hns · 2021-09-30T10:45:17Z

test/langtools/jdk/javadoc/doclet/testHrefInDocComment/pkg/J1.java

+    protected Object field1;
+
+    /**
+     *<a href="{@docRoot}/pkg/J1.html#functions">Creates an instance which has


Another unclosed a element

Sorry, the last few characters were trimmed. I fixed it.

…broken

hns

Thanks, looks good now!

masyano · 2021-09-30T11:28:10Z

/integrate

openjdk · 2021-09-30T11:31:17Z

@masyano
Your change (at version c9d6489) is now ready to be sponsored by a Committer.

hns · 2021-09-30T11:32:14Z

/sponsor

openjdk · 2021-09-30T11:33:36Z

Going to push as commit bb95dda.
Since your change was applied there have been 315 commits pushed to the master branch:

2f955d6: 8273142: Remove dependancy of TestHttpServer, HttpTransaction, HttpCallback from open/test/jdk/sun/net/www/protocol/http/ tests
94e31e5: 8274506: TestPids.java and TestPidsLimit.java fail with podman run as root
a8210c5: 8274401: C2: GraphKit::load_array_element bypasses Access API
dfc557c: 8274406: RunThese30M.java failed "assert(!LCA_orig->dominates(pred_block) || early->dominates(pred_block)) failed: early is high enough"
c0533ef: 8274522: java/lang/management/ManagementFactory/MXBeanException.java test fails with Shenandoah
f8415a9: 8274523: java/lang/management/MemoryMXBean/MemoryTest.java test should handle Shenandoah
355356c: 8273435: Remove redundant zero-length check in ClassDesc.of
97385d4: 8274405: Suppress warnings on non-serializable non-transient instance fields in javac and javadoc
79cebe2: 8274050: Unnecessary Vector usage in javax.crypto
97b2874: 8274509: Remove stray * and stylistic . from doc comments
... and 305 more: https://git.openjdk.java.net/jdk/compare/a522d6b53cd841b4bfe87eac5778c9e5cdf5e90f...master

Your commit was automatically rebased without conflicts.

openjdk · 2021-09-30T11:33:52Z

@hns @masyano Pushed as commit bb95dda.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

8248001: javadoc generates invalid HTML pages whose ftp:// links are …

8df89de

…broken

openjdk bot added the javadoc javadoc-dev@openjdk.org label Aug 20, 2021

Yano, Masanori added 2 commits August 20, 2021 19:16

8248001: javadoc generates invalid HTML pages whose ftp:// links are …

e5401a7

…broken

8248001: javadoc generates invalid HTML pages whose ftp:// links are …

6121ce4

…broken

openjdk bot added the rfr Pull request is ready for review label Aug 20, 2021

pavelrappo reviewed Aug 20, 2021

View reviewed changes

Yano, Masanori added 2 commits September 7, 2021 21:01

Merge branch 'master' of https://github.com/masyano/jdk into 8248001

77da523

8248001: javadoc generates invalid HTML pages whose ftp:// links are …

7639e7d

…broken

hns suggested changes Sep 13, 2021

View reviewed changes

jonathan-gibbons reviewed Sep 13, 2021

View reviewed changes

8248001: javadoc generates invalid HTML pages whose ftp:// links are …

1d501a0

…broken

8248001: javadoc generates invalid HTML pages whose ftp:// links are …

f336b86

…broken

8248001: javadoc generates invalid HTML pages whose ftp:// links are …

b68c760

…broken

hns approved these changes Sep 30, 2021

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Sep 30, 2021

openjdk bot added the sponsor Pull request is ready to be sponsored label Sep 30, 2021

hns suggested changes Sep 30, 2021

View reviewed changes

openjdk bot removed sponsor Pull request is ready to be sponsored ready Pull request is ready to be integrated labels Sep 30, 2021

hns suggested changes Sep 30, 2021

View reviewed changes

8248001: javadoc generates invalid HTML pages whose ftp:// links are …

c9d6489

…broken

hns approved these changes Sep 30, 2021

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Sep 30, 2021

openjdk bot added the sponsor Pull request is ready to be sponsored label Sep 30, 2021

openjdk bot closed this Sep 30, 2021

openjdk bot added integrated Pull request has been integrated and removed ready Pull request is ready to be integrated rfr Pull request is ready for review sponsor Pull request is ready to be sponsored labels Sep 30, 2021

8248001: javadoc generates invalid HTML pages whose ftp:// links are broken #5198

8248001: javadoc generates invalid HTML pages whose ftp:// links are broken #5198

Conversation

masyano commented Aug 20, 2021 • edited by openjdk bot

Progress

Issue

Reviewers

Reviewing

bridgekeeper bot commented Aug 20, 2021

openjdk bot commented Aug 20, 2021

mlbridge bot commented Aug 20, 2021 • edited

Webrevs

Choose a reason for hiding this comment

dfuch Aug 20, 2021 • edited

Choose a reason for hiding this comment

dfuch Aug 20, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anthonyvdotbe Aug 21, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

masyano commented Sep 7, 2021

masyano commented Sep 13, 2021

hns left a comment

Choose a reason for hiding this comment

jonathan-gibbons Sep 13, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

masyano commented Sep 24, 2021

hns commented Sep 27, 2021

masyano commented Sep 28, 2021

hns left a comment

Choose a reason for hiding this comment

openjdk bot commented Sep 30, 2021 • edited

hns commented Sep 30, 2021

masyano commented Sep 30, 2021

openjdk bot commented Sep 30, 2021

masyano commented Sep 30, 2021

hns left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hns left a comment

Choose a reason for hiding this comment

masyano commented Sep 30, 2021

openjdk bot commented Sep 30, 2021

hns commented Sep 30, 2021

openjdk bot commented Sep 30, 2021

openjdk bot commented Sep 30, 2021

masyano commented Aug 20, 2021 •

edited by openjdk bot

mlbridge bot commented Aug 20, 2021 •

edited

dfuch Aug 20, 2021 •

edited

dfuch Aug 20, 2021 •

edited

anthonyvdotbe Aug 21, 2021 •

edited

jonathan-gibbons Sep 13, 2021 •

edited

openjdk bot commented Sep 30, 2021 •

edited