8306031: Update IANA Language Subtag Registry to Version 2023-04-13 #13501

justin-curtis-lu · 2023-04-17T19:48:10Z

Update the registry and accompanying tests with the IANA 4/13/2022 update.

This update introduces the case where an IANA entry can have a preferred value, but that preferred value has a preferred value as well.

This causes unexpected failures in JDK tests because of how locale equivalencies are created.

eg: ar-ajp has a preferred value of ajp but ajp has a preferred value of apc

Normally, when the JDK is built, LocaleEquivlalentMaps.java generates the following

...
singleEquivMap.put("ar-ajp", "ajp");
singleEquivMap.put("ajp", "ar-ajp");
...
multiEquivsMap.put("ajp", new String[] {"apc", "ar-apc"});
multiEquivsMap.put("apc", new String[] {"ajp", "ar-apc"});
multiEquivsMap.put("ar-apc", new String[] {"apc", "ajp"});
...

When LocaleMatcher.parse(ACCEPT_LANGUAGE) is called with ACCEPT_LANGUAGE containing apc and ajp in that order, the following occurs:

apc is found, apc is added, all of apc's equivalencies are added: ajp and ar-apc

When parse iterates to ajp, it finds that it is already added to the list, and does not add it's equivalency ar-ajp.

To address this, the build process must be adjusted so that the equivalencies are built as

...
multiEquivsMap.put("ajp", new String[] {"apc", "ar-ajp", "ar-apc"});
multiEquivsMap.put("apc", new String[] {"ajp", "ar-ajp", "ar-apc"});
multiEquivsMap.put("ar-ajp", new String[] {"apc", "ajp", "ar-apc"});
multiEquivsMap.put("ar-apc", new String[] {"apc", "ajp", "ar-ajp"});
...

As, if ar-ajp has a preferred value of ajp, and ajp has a preferred value of apc, this technically means that ar-ajp is equivalent to apc and its equivalencies as well. This way, when LocaleMatcher.parse(ACCEPT_LANGUAGE) iterates to apc, it will add all of it's equivalencies including ar-ajp.

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue

Issue

JDK-8306031: Update IANA Language Subtag Registry to Version 2023-04-13

Reviewers

Naoto Sato (@naotoj - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/13501/head:pull/13501
$ git checkout pull/13501

Update a local copy of the PR:
$ git checkout pull/13501
$ git pull https://git.openjdk.org/jdk.git pull/13501/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 13501

View PR using the GUI difftool:
$ git pr show -t 13501

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/13501.diff

Webrev

Link to Webrev Comment

bridgekeeper · 2023-04-17T19:48:54Z

👋 Welcome back jlu! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2023-04-17T19:50:36Z

@justin-curtis-lu The following labels will be automatically applied to this pull request:

core-libs
i18n

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

mlbridge · 2023-04-19T19:59:36Z

Webrevs

naotoj · 2023-04-19T22:46:44Z

make/jdk/src/classes/build/tools/generatelsrequivmaps/EquivMapsGenerator.java

+                final String finalPref = ","+preferred;
+                final String inbtwnPref = ","+preferred+",";


This could utilize regex?

Much better that way, fixed

naotoj · 2023-04-19T22:49:36Z

make/jdk/src/classes/build/tools/generatelsrequivmaps/EquivMapsGenerator.java

+                final String finalPref = ","+preferred;
+                final String inbtwnPref = ","+preferred+",";
+                // Check if current pref exists inside a value for another pref
+                List<StringBuilder> doublePrefs = initialLanguageMap.entrySet()


values() fits here

naotoj · 2023-04-19T22:50:30Z

make/jdk/src/classes/build/tools/generatelsrequivmaps/EquivMapsGenerator.java

+                        .stream().filter(e -> (e.getValue().toString().endsWith(finalPref) ||
+                                e.getValue().toString().contains(inbtwnPref)))
+                        .map(Map.Entry::getValue)
+                        .collect(Collectors.toList());


Can replace collect() with toList()

Changed this, as well as the other existing occurrence

naotoj · 2023-04-19T22:51:58Z

make/jdk/src/classes/build/tools/generatelsrequivmaps/EquivMapsGenerator.java

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2012, 2021, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2012, 2023, Oracle and/or its affiliates. All rights reserved.


Cannot comment on unmodified lines, but instead of calculating the initial load itself, HashMap.newHashMap() can be used for initializing maps.

Fixed, so they are created with numMappings and not initialLoad

naotoj

Much better. Some comments follow.

naotoj · 2023-04-20T17:00:50Z

make/jdk/src/classes/build/tools/generatelsrequivmaps/EquivMapsGenerator.java

@@ -37,9 +37,9 @@
 import java.util.List;
 import java.util.Locale;
 import java.util.Map;
+import java.util.regex.Pattern;


Nit: sort the imports

naotoj · 2023-04-20T17:06:26Z

make/jdk/src/classes/build/tools/generatelsrequivmaps/EquivMapsGenerator.java

+                // can have a preferred value itself.
+                // eg: ar-ajp has pref ajp which has pref apc
+                boolean foundInOther = false;
+                Pattern pattern = Pattern.compile("\\b"+preferred+"\\b");


I think we should explicitly find a pattern with , instead of the word boundary. Otherwise, it could match - in a tag.

Good point, changed it to match starting with , and ending with , or end of string boundary

naotoj · 2023-04-20T17:08:49Z

make/jdk/src/classes/build/tools/generatelsrequivmaps/EquivMapsGenerator.java

+                + sortedLanguageMap1.size() + ");\n"
+                + "        multiEquivsMap = HashMap.newHashMap("
+                + sortedLanguageMap2.size() + ");\n"
+                + "        regionVariantEquivMap = HashMap.newHashMap("
+                + sortedRegionVariantMap.size() + ");\n\n"


It'd be nice if these messy concatenated strings be converted to a text block, but that would be for another day.

Filed a separate JBS issue for this here

naotoj

LGTM

openjdk · 2023-04-21T17:37:54Z

@justin-curtis-lu This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8306031: Update IANA Language Subtag Registry to Version 2023-04-13

Reviewed-by: naoto

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 150 new commits pushed to the master branch:

88d9ebf: 8306752: Open source several container and component AWT tests
1c2dadc: 8306683: Open source several clipboard and color AWT tests
b372f28: 8306753: Open source several container AWT tests
e3ccaa6: 8306623: (bf) CharBuffer::allocate throws unexpected exception type with some CharSequences
d819deb: 8304423: Refactor FdLibm.java
28829f3: 8306734: Shenandoah: Missing barriers on deoptimization path
9beae21: 8306718: Optimize and opensource some old AWT tests
36d61c3: 8306765: Some client related jtreg problem list entries are malformed
5f4965d: 8306687: Relax memory ordering constraints on metaspace atomic counters
33d6614: 8304845: Update PCSC-Lite for Suse Linux to 1.9.9 and fix incomplete license wording
... and 140 more: https://git.openjdk.org/jdk/compare/1fd400608e9ea423a6e4f1797652bd02f027da1b...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

As you do not have Committer status in this project an existing Committer must agree to sponsor your change. Possible candidates are the reviewers of this PR (@naotoj) but any other Committer may sponsor as well.

➡️ To flag this PR as ready for integration with the above commit message, type /integrate in a new comment. (Afterwards, your sponsor types /sponsor in a new comment to perform the integration).

justin-curtis-lu · 2023-04-25T20:28:45Z

/integrate

openjdk · 2023-04-25T20:30:15Z

@justin-curtis-lu
Your change (at version b03a51c) is now ready to be sponsored by a Committer.

naotoj · 2023-04-25T20:30:33Z

/sponsor

openjdk · 2023-04-25T20:32:36Z

Going to push as commit 00b1eac.
Since your change was applied there have been 150 commits pushed to the master branch:

88d9ebf: 8306752: Open source several container and component AWT tests
1c2dadc: 8306683: Open source several clipboard and color AWT tests
b372f28: 8306753: Open source several container AWT tests
e3ccaa6: 8306623: (bf) CharBuffer::allocate throws unexpected exception type with some CharSequences
d819deb: 8304423: Refactor FdLibm.java
28829f3: 8306734: Shenandoah: Missing barriers on deoptimization path
9beae21: 8306718: Optimize and opensource some old AWT tests
36d61c3: 8306765: Some client related jtreg problem list entries are malformed
5f4965d: 8306687: Relax memory ordering constraints on metaspace atomic counters
33d6614: 8304845: Update PCSC-Lite for Suse Linux to 1.9.9 and fix incomplete license wording
... and 140 more: https://git.openjdk.org/jdk/compare/1fd400608e9ea423a6e4f1797652bd02f027da1b...master

Your commit was automatically rebased without conflicts.

openjdk · 2023-04-25T20:32:54Z

@naotoj @justin-curtis-lu Pushed as commit 00b1eac.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

justin-curtis-lu added 3 commits April 17, 2023 09:39

Update registry

83469c3

Update test with preffered value: ajp

5aa2586

Fix order of langs

2296fa6

openjdk bot added core-libs core-libs-dev@openjdk.org i18n i18n-dev@openjdk.org labels Apr 17, 2023

justin-curtis-lu added 2 commits April 17, 2023 13:41

ajp should follow apc

3b6cd3e

Adjust build and test to account for new case

30e0ef8

justin-curtis-lu marked this pull request as ready for review April 19, 2023 19:53

openjdk bot added the rfr Pull request is ready for review label Apr 19, 2023

Copyright

e5d332e

naotoj reviewed Apr 19, 2023

View reviewed changes

justin-curtis-lu added 2 commits April 19, 2023 23:10

Improve stream processing and hash map creation

4a75417

Hash map should be initialized with numMappings

7d3021a

naotoj reviewed Apr 20, 2023

View reviewed changes

justin-curtis-lu added 2 commits April 20, 2023 13:11

Match on exact pattern

c304662

Import ordering

b03a51c

naotoj approved these changes Apr 21, 2023

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Apr 21, 2023

openjdk bot added the sponsor Pull request is ready to be sponsored label Apr 25, 2023

openjdk bot added the integrated Pull request has been integrated label Apr 25, 2023

openjdk bot closed this Apr 25, 2023

openjdk bot removed the ready Pull request is ready to be integrated label Apr 25, 2023

openjdk bot removed rfr Pull request is ready for review sponsor Pull request is ready to be sponsored labels Apr 25, 2023

		final String finalPref = ","+preferred;
		final String inbtwnPref = ","+preferred+",";

8306031: Update IANA Language Subtag Registry to Version 2023-04-13 #13501

8306031: Update IANA Language Subtag Registry to Version 2023-04-13 #13501

Uh oh!

Conversation

justin-curtis-lu commented Apr 17, 2023 • edited by openjdk bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Progress

Issue

Reviewers

Reviewing

Webrev

Uh oh!

bridgekeeper bot commented Apr 17, 2023

Uh oh!

openjdk bot commented Apr 17, 2023

Uh oh!

mlbridge bot commented Apr 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Webrevs

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justin-curtis-lu Apr 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

naotoj left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justin-curtis-lu Apr 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

naotoj left a comment

Choose a reason for hiding this comment

Uh oh!

openjdk bot commented Apr 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

justin-curtis-lu commented Apr 25, 2023

Uh oh!

openjdk bot commented Apr 25, 2023

Uh oh!

naotoj commented Apr 25, 2023

Uh oh!

openjdk bot commented Apr 25, 2023

Uh oh!

openjdk bot commented Apr 25, 2023

Uh oh!

Uh oh!

justin-curtis-lu commented Apr 17, 2023 •

edited by openjdk bot

Loading

mlbridge bot commented Apr 19, 2023 •

edited

Loading

justin-curtis-lu Apr 20, 2023 •

edited

Loading

justin-curtis-lu Apr 20, 2023 •

edited

Loading

openjdk bot commented Apr 21, 2023 •

edited

Loading