8301119: Support for GB18030-2022 #45

gnu-andrew · 2023-06-23T01:39:00Z

This is being proposed for inclusion in 8u382 during rampdown, so that the changes are in place for when GB18030-2022 enforcement begins in August. It modifies GB18030 to handle both the 2000 and the 2022 variant. The 2000 variant is available by setting -Djdk.charset.GB18030=2000.

With the preceding test changes in place (#43 and #44), the changes needed for this are fairly minimal. The biggest divergence from 11u is in the character set providers. The changes in the make directory are not needed as 8u never moved to using a template for GB18030 in the first place (the 11u changes revert it back to being source-based). The change in the SPI.java generator tool moves into ExtendedCharsets.java in the class library, as the file is not auto-generated in 8u. The change to StandardCharsets.java.template lands in AbstractCharsetProvider.java.

In 8u, the standard charsets are generated from a text file by a shell script, while the extended charsets are handled by a standard class. 11u moves GB18030 from extended to standard. I experimented with this in 8u, but it seemed more problematic than just keeping it in the extended set. The only reason I can see for moving it in 11u is it allows IS_2000 to be package-private to sun.nio.cs, whereas we need to make it public in sun.nio.cs.ext so it can be accessed from sun.nio.cs.

To use the 11u solution would mean major rewrites to the shell script or bringing over the whole change in how the standard charset provider is generated from 11u, which I think, along with moving the package the character set is in, is too risky and unnecessary for this change. The generation changes are necessary because the GB18030 character set needs to provide a different alias, depending on whether it is the 2000 or 2002 variant. The genCharsetProvider.sh would need the alterations we have added to ExtendedCharsets.java to handle this, but converted to awk.

The only adjustment to GB18030.java, other than copyright headers, is to replace the use of jdk.internal.misc.VM.initLevel with that of sun.misc.VM.isBooted. 8u does not provide as fine-grained access to the initialisation status as 11u, and so may force the use of the 2022 standard until a later stage in the bootup (BOOTED is initLevel() = 4 in 11u).

With the tests, the adjustments are just due to differing bug IDs, the absence of @modules and the use of constructs (var) and library calls (Set.of) that don't exist in 8u. The List.of and Set.of calls are frequent issues in backports, so I used this as an opportunity to introduce a full set of equivalents into the test library. It should now be possible to just rewrite Set.of to Utils.setOf and List.of to Utils.listOf. The returned collections are expected to be unmodifiable, not contain null and (in the case of sets) not contain duplicates. Simple replacement with a newly constructed ArrayList or HashSet would not ensure this. While this test does not rely on this, others may, so it seemed worth providing a closer replacement for use in future backports.

All sun.nio.cs tests pass with this patch applied.

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue

Issue

JDK-8301119: Support for GB18030-2022 (Enhancement - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk8u.git pull/45/head:pull/45
$ git checkout pull/45

Update a local copy of the PR:
$ git checkout pull/45
$ git pull https://git.openjdk.org/jdk8u.git pull/45/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 45

View PR using the GUI difftool:
$ git pr show -t 45

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk8u/pull/45.diff

Webrev

Link to Webrev Comment

bridgekeeper · 2023-06-23T01:40:37Z

👋 Welcome back andrew! A progress list of the required criteria for merging this PR into pr/44 will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2023-06-23T01:41:45Z

This backport pull request has now been updated with issue from the original commit.

mlbridge · 2023-06-23T01:46:21Z

Webrevs

02: Full (4f38ca44)
01: Full (aac31dd2)
00: Full (6aa546aa)

openjdk · 2023-06-23T02:46:29Z

This backport pull request has now been updated with issue from the original commit.

openjdk · 2023-06-23T10:14:18Z

@gnu-andrew Please do not rebase or force-push to an active PR as it invalidates existing review comments. Note for future reference, the bots always squash all changes into a single commit automatically as part of the integration. See OpenJDK Developers’ Guide for more information.

openjdk-notifier · 2023-06-23T12:24:50Z

The parent pull request that this pull request depends on has now been integrated and the target branch of this pull request has been updated. This means that changes from the dependent pull request can start to show up as belonging to this pull request, which may be confusing for reviewers. To remedy this situation, simply merge the latest changes from the new target branch into this pull request by running commands similar to these in the local repository for your personal fork:

git checkout JDK-8301119
git fetch https://git.openjdk.org/jdk8u.git master
git merge FETCH_HEAD
# if there are conflicts, follow the instructions given by git merge
git commit -m "Merge master"
git push

openjdk · 2023-06-23T12:26:46Z

@gnu-andrew this pull request can not be integrated into master due to one or more merge conflicts. To resolve these merge conflicts and update this pull request you can run the following commands in the local repository for your personal fork:

git checkout JDK-8301119
git fetch https://git.openjdk.org/jdk8u.git master
git merge FETCH_HEAD
# resolve conflicts and follow the instructions given by git merge
git commit -m "Merge master"
git push

openjdk · 2023-06-23T12:52:44Z

@gnu-andrew Please do not rebase or force-push to an active PR as it invalidates existing review comments. Note for future reference, the bots always squash all changes into a single commit automatically as part of the integration. See OpenJDK Developers’ Guide for more information.

gnu-andrew · 2023-06-23T14:54:37Z

I'm not able to reproduce these failures locally, which makes it hard to diagnose what's going on here.

I can try and move the character set to sun.nio.cs after all and see if it makes a difference for you.

…64cpp Fix inter_masm_riscv64.cpp by fixing mdp->mdx and adding annotation

jerboaa · 2023-06-27T08:32:39Z

These 4 tests consistently fail for me with a fastdebug build:

FAILED: java/nio/charset/Charset/RegisteredCharsets.java
FAILED: sun/nio/cs/mapping/CoderTest.java
FAILED: sun/nio/cs/mapping/TestConv.java
FAILED: sun/nio/cs/TestGB18030.java

That was my bad. Ran on the wrong JDK build (so have hidden the comment). Sorry. I'll post a review of the patch today.

jerboaa

I'd suggest to use the late init hook in AbstractCharsetProvider.charsetForName() instead which would make this patch cleaner IMO. See gnu-andrew#1

jerboaa · 2023-06-27T12:49:51Z

jdk/src/share/classes/sun/nio/cs/AbstractCharsetProvider.java

+    private String canonicalize(String csn) {
+        if (csn.startsWith("gb18030-")) {
+            return csn.equals("gb18030-2022") && !GB18030.IS_2000 ||
+                   csn.equals("gb18030-2000") && GB18030.IS_2000 ? "gb18030" : csn;
+        } else {
+            String acn = aliasMap.get(csn);
+            return (acn != null) ? acn : csn;
+        }


I believe if we hooked into the late initialization hook instead we wouldn't need this canonicalization as the aliasMap would map it back to the correct class name.

jerboaa · 2023-06-27T12:51:07Z

jdk/src/share/classes/sun/nio/cs/AbstractCharsetProvider.java

 import java.util.Map;
 import sun.misc.ASCIICaseInsensitiveComparator;
-
+import sun.nio.cs.ext.GB18030;


Depending on GB18030 class (sun.nio.cs.ext. in charsets.jar) in package sun.nio.cs (rt.jar) seems worrisome.

jerboaa · 2023-06-27T12:54:20Z

jdk/src/share/classes/sun/nio/cs/ext/ExtendedCharsets.java

        charset("GB18030", "GB18030",
                new String[] {
-                    "gb18030-2000"
+                    GB18030.IS_2000 ? "gb18030-2000" : "gb18030-2022"


This initialization will be wrong when run with LANG=zh_CN.GB18030 and -Djdk.charset.GB18030=2000as in that case the GB18030.IS_2000 will be false as the JVM won't have properly initialized yet. Again a suggestion to use the late init hook instead. See https://bugs.openjdk.org/browse/JDK-8310947

gnu-andrew · 2023-07-12T11:23:19Z

Replaced by openjdk/jdk8u-dev#339

gnu-andrew changed the base branch from master to pr/44 June 23, 2023 01:39

openjdk bot changed the title ~~Backport cfe34ed89c4f6ef9a49dceef30da1e43b418b152~~ 8186801: Add regression test to test mapping based charsets (generated at build time) Jun 23, 2023

openjdk bot added backport Port of a pull request already in a different code base rfr Pull request is ready for review labels Jun 23, 2023

gnu-andrew changed the title ~~8186801: Add regression test to test mapping based charsets (generated at build time)~~ Backport 5c4e744dabcf7785c35168db5d0458ccebfd41e6 Jun 23, 2023

openjdk bot changed the title ~~Backport 5c4e744dabcf7785c35168db5d0458ccebfd41e6~~ 8301119: Support for GB18030-2022 Jun 23, 2023

gnu-andrew mentioned this pull request Jun 23, 2023

8186801: Add regression test to test mapping based charsets (generated at build time) #43

Closed

3 tasks

gnu-andrew force-pushed the JDK-8301119 branch from 6aa546a to aac31dd Compare June 23, 2023 10:13

openjdk-notifier bot changed the base branch from pr/44 to master June 23, 2023 12:24

openjdk bot added the merge-conflict Pull request has merge conflict with target branch label Jun 23, 2023

Backport 5c4e744dabcf7785c35168db5d0458ccebfd41e6

4f38ca4

gnu-andrew force-pushed the JDK-8301119 branch from aac31dd to 4f38ca4 Compare June 23, 2023 12:51

openjdk bot removed the merge-conflict Pull request has merge conflict with target branch label Jun 23, 2023

This comment was marked as outdated.

Sign in to view

DingliZhang pushed a commit to DingliZhang/jdk8u that referenced this pull request Jun 25, 2023

Merge pull request openjdk#45 from zhangxiang-plct/fix_interp_masm_rv…

52aeed4

…64cpp Fix inter_masm_riscv64.cpp by fixing mdp->mdx and adding annotation

jerboaa reviewed Jun 27, 2023

View reviewed changes

gnu-andrew mentioned this pull request Jul 12, 2023

8301119: Support for GB18030-2022 openjdk/jdk8u-dev#339

Closed

3 tasks

gnu-andrew closed this Jul 13, 2023

8301119: Support for GB18030-2022 #45

8301119: Support for GB18030-2022 #45

Uh oh!

Conversation

gnu-andrew commented Jun 23, 2023 • edited by openjdk bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Progress

Issue

Reviewing

Webrev

Uh oh!

bridgekeeper bot commented Jun 23, 2023

Uh oh!

openjdk bot commented Jun 23, 2023

Uh oh!

mlbridge bot commented Jun 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Webrevs

Uh oh!

openjdk bot commented Jun 23, 2023

Uh oh!

openjdk bot commented Jun 23, 2023

Uh oh!

openjdk-notifier bot commented Jun 23, 2023

Uh oh!

openjdk bot commented Jun 23, 2023

Uh oh!

openjdk bot commented Jun 23, 2023

Uh oh!

This comment was marked as outdated.

Uh oh!

gnu-andrew commented Jun 23, 2023

Uh oh!

jerboaa commented Jun 27, 2023

Uh oh!

jerboaa left a comment

Choose a reason for hiding this comment

Uh oh!

jerboaa Jun 27, 2023

Choose a reason for hiding this comment

Uh oh!

jerboaa Jun 27, 2023

Choose a reason for hiding this comment

Uh oh!

jerboaa Jun 27, 2023

Choose a reason for hiding this comment

Uh oh!

gnu-andrew commented Jul 12, 2023

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

gnu-andrew commented Jun 23, 2023 •

edited by openjdk bot

Loading

mlbridge bot commented Jun 23, 2023 •

edited

Loading