8260265: UTF-8 by Default #4733

naotoj · 2021-07-08T21:23:00Z

This is an implementation for the JEP 400: UTF-8 by Default. The gist of the changes is Charset.defaultCharset() returning UTF-8 and file.encoding system property being added in the spec, but another notable modification is in java.io.PrintStream where it continues to use the Console encoding as the default charset instead of UTF-8. Other changes are mostly clarification of the term "default charset" and their links. Corresponding CSR has also been drafted.

JEP 400: https://bugs.openjdk.java.net/browse/JDK-8187041
CSR: https://bugs.openjdk.java.net/browse/JDK-8260266

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8260265: UTF-8 by Default

Reviewers

Alan Bateman (@AlanBateman - Reviewer)
Roger Riggs (@RogerRiggs - Reviewer) ⚠️ Review applies to 6f9e5eb

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/4733/head:pull/4733
$ git checkout pull/4733

Update a local copy of the PR:
$ git checkout pull/4733
$ git pull https://git.openjdk.java.net/jdk pull/4733/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 4733

View PR using the GUI difftool:
$ git pr show -t 4733

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/4733.diff

bridgekeeper · 2021-07-08T21:23:56Z

👋 Welcome back naoto! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

naotoj · 2021-07-08T21:24:03Z

/csr

openjdk · 2021-07-08T21:25:25Z

@naotoj this pull request will not be integrated until the CSR request JDK-8260266 for issue JDK-8260265 has been approved.

openjdk · 2021-07-08T21:28:12Z

@naotoj The following labels will be automatically applied to this pull request:

core-libs
hotspot-runtime
net

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

mlbridge · 2021-07-08T22:07:14Z

Webrevs

gbaso · 2021-07-14T12:39:46Z

Consider an application that creates a java.io.FileWriter with its one-argument constructor and then uses it to write some text to a file. The resulting file will contain a sequence of bytes encoded using the default charset of the JDK running the application. A second application, run on a different machine or by a different user on the same machine, creates a java.io.FileReader with its one-argument constructor and uses it to read the bytes in that file. The resulting text contains a sequence of characters decoded using the default charset of the JDK running the second application. If the default charset differs between the JDK of the first application and the JDK of the second application, then the resulting text may be silently corrupted or incomplete, since these APIs replace erroneous input rather than fail.

It's even worse than that, because many OpenSSH installs are configured by default to forward and accept the user locale (see e.g. for RHEL 7).

So a single application, on a single remote machine, can be unknowingly started by a single user with different locales, and therefore different encodings, depending on how the user connected to the remote machine. For example, on Windows connecting via powershell results in LANG=en_US.UTF-8, while using WSL2 results in LANG=C.UTF-8. On Java 11 in a RHEL7 machine, file.encoding results in UTF-8 in the first case, but ANSI_X3.4-1968 in the second, leading to a default charset ASCII.

Worth mentioning is also that Charset.forName("default") is just an alias to ASCII, per sun.nio.cs.StandardCharsets$Aliases.

RogerRiggs · 2021-07-14T14:50:13Z

src/java.base/share/classes/java/io/ByteArrayOutputStream.java

-     * or the {@code toString()} method, which uses the platform's default
-     * character encoding.
+     * or the {@code toString()} method, which uses the default
+     * charset.


Fold to previous line.

RogerRiggs · 2021-07-14T14:55:39Z

src/java.base/share/classes/java/io/Console.java

+            if (csname != null) {
+                try {
+                    cs = Charset.forName(csname);
+                } catch (Exception ignored) { }


A separate enhancement...
I've long thought that should be a way to avoid the exception here.
For example, a Charset.forName(csname, default);
The caller might have a default in mind or supply null and then be able to test for null.

Agreed. Will file an RFE for this.

https://bugs.openjdk.java.net/browse/JDK-8270490

RogerRiggs · 2021-07-14T14:59:15Z

src/java.base/share/classes/java/io/FileReader.java

 *
 * <p>
 * The {@code FileReader} is meant for reading streams of characters. For reading
 * streams of raw bytes, consider using a {@code FileInputStream}.
 *
 * @see InputStreamReader
 * @see FileInputStream
+ * @see java.nio.charset.Charset#defaultCharset()


The @ see duplicates the link above, the javadoc can do without the @ see.

If I remove that @see, I don't see the link in See Also section. Am I missing something?

In my view the @ linkplain is sufficient to allow the reader to navigate; but YMMV.

RogerRiggs · 2021-07-14T15:03:14Z

src/java.base/share/classes/java/io/InputStreamReader.java

@@ -35,7 +35,7 @@
 * An InputStreamReader is a bridge from byte streams to character streams: It
 * reads bytes and decodes them into characters using a specified {@link
 * java.nio.charset.Charset charset}.  The charset that it uses
- * may be specified by name or may be given explicitly, or the platform's
+ * may be specified by name or may be given explicitly, or the
 * {@link Charset#defaultCharset() default charset} may be accepted.


"may be accepted" seems like the API has some choice in the matter.
Perhaps "accepted" -> "used".
And in other classes below if there's a suitable replacement.

RogerRiggs · 2021-07-14T15:09:41Z

src/java.base/share/classes/java/io/PrintStream.java

- * bytes using the given encoding or charset, or the platform's default
- * character encoding if not specified.
+ * bytes using the given encoding or charset, or the default
+ * console charset if not specified.


JEP 400 doesn't give a rationale for using the console charset for PrintStream.
PrintStreams are used for output to files and other media other than just a tty/console.
The charset of system.out/err should use the console charset.

This was my thinking in #4733 (comment).

OK, I am now conviced. Modified not to default to Console.charset() for generic PrintStream w/o charset constructor.

RogerRiggs · 2021-07-14T15:25:44Z

src/java.base/share/classes/java/lang/System.java

@@ -797,6 +797,15 @@ public static native void arraycopy(Object src,  int  srcPos,
     *     <td>The module name of the initial/main module</td></tr>
     * <tr><th scope="row">{@systemProperty jdk.module.main.class}</th>
     *     <td>The main class name of the initial module</td></tr>
+     * <tr><th scope="row">{@systemProperty file.encoding}</th>
+     *     <td>The name of the default charset. Users may specify
+     *     {@code UTF-8} or {@code COMPAT} on the command line to the value.


The wording could imply that only those two values can be supplied.
It could be rephrased to say that if the property is supplied on the command line
it overrides the default UTF-8.

That was intentional. Only those two are supported, others continue to work as before (but not supported).

Still it leaves an uncomfortable feeling, perhaps remedied by an "other values have unspecified behavior"
or the "other values are implementation specific".

Added the clarifying sentence to the spec.

RogerRiggs · 2021-07-14T15:32:00Z

src/java.base/share/classes/java/net/URLDecoder.java

@@ -86,17 +88,17 @@
     */
    private URLDecoder() {}

-    // The platform default encoding
+    // The default charset
    static String dfltEncName = URLEncoder.dfltEncName;


Perhaps add the value of file.encoding to the StaticProperties either as a string or as the Charset.
That would allow a few different lookups of the property to be simplified.

RogerRiggs · 2021-07-14T15:34:20Z

src/java.base/share/classes/java/net/URLEncoder.java

@@ -161,7 +163,7 @@ public static String encode(String s) {
        try {
            str = encode(s, dfltEncName);
        } catch (UnsupportedEncodingException e) {


Perhaps a separate cleanup, the Charset should be cached, not just the name and use the encode(s, charset) method.

RogerRiggs · 2021-07-14T15:40:10Z

src/java.base/share/classes/java/nio/charset/Charset.java

+     * the system property {@code file.encoding} on the command line. If the
+     * value designates {@code COMPAT}, the default charset is derived from
+     * the {@code native.encoding} system property, which typically depends
+     * upon the locale and charset of the underlying operating system.


The description in java.lang.System of the file.encoding property does not indicate it is 'implementation specific'.
In that context, it appears to be part of the JavaSE spec.
Having the spec in a single place with references to it from others could avoid duplication.

file.encoding is listed under @implNote tag in System.getProperties(), so it is implementation-specific.

naotoj · 2021-07-14T20:51:53Z

Consider an application that creates a java.io.FileWriter with its one-argument constructor and then uses it to write some text to a file. The resulting file will contain a sequence of bytes encoded using the default charset of the JDK running the application. A second application, run on a different machine or by a different user on the same machine, creates a java.io.FileReader with its one-argument constructor and uses it to read the bytes in that file. The resulting text contains a sequence of characters decoded using the default charset of the JDK running the second application. If the default charset differs between the JDK of the first application and the JDK of the second application, then the resulting text may be silently corrupted or incomplete, since these APIs replace erroneous input rather than fail.

It's even worse than that, because many OpenSSH installs are configured by default to forward and accept the user locale (see e.g. for RHEL 7).

So a single application, on a single remote machine, can be unknowingly started by a single user with different locales, and therefore different encodings, depending on how the user connected to the remote machine. For example, on Windows connecting via powershell results in LANG=en_US.UTF-8, while using WSL2 results in LANG=C.UTF-8. On Java 11 in a RHEL7 machine, file.encoding results in UTF-8 in the first case, but ANSI_X3.4-1968 in the second, leading to a default charset ASCII.

Worth mentioning is also that Charset.forName("default") is just an alias to ASCII, per sun.nio.cs.StandardCharsets$Aliases.

Thanks. Updated the JEP.

AlanBateman

Good work!

One thing we missed when adding Console.charset() was the code example in OutputStreamWriter's class description. It uses System.out and should have been changed to "anOutputStream" to avoid people copying this usage. We can do it as part of this PR or another but would be good to get it consistent with InputStreamReader.

AlanBateman · 2021-07-20T11:23:55Z

src/java.base/share/classes/java/nio/charset/Charset.java

+     *
+     * @implNote An implementation may override the default charset with
+     * the system property {@code file.encoding} on the command line. If the
+     * value designates {@code COMPAT}, the default charset is derived from


It might be simpler to say "If the value is COMPAT" and avoid the word "designates" here.

AlanBateman · 2021-07-20T11:25:35Z

src/java.base/share/classes/java/io/FileReader.java

@@ -47,40 +47,40 @@

   /**
    * Creates a new {@code FileReader}, given the name of the file to read,
-    * using the platform's
-    * {@linkplain java.nio.charset.Charset#defaultCharset() default charset}.
+    * using the {@linkplain java.nio.charset.Charset#defaultCharset() default charset}.


The java.io classes import java.nio.charset.Charset so don't need to use the fully qualified class name everywhere.

Corrected, as well as other locations.

AlanBateman · 2021-07-20T11:28:23Z

test/jdk/tools/launcher/UnicodeTest.java

@@ -163,7 +163,7 @@ static String generateSources() throws Exception {
        return commandLineClassName;
    }

-    private static final String defaultEncoding = Charset.defaultCharset().name();
+    private static final String defaultEncoding = System.getProperty("sun.jnu.encoding");


It's a bit confusing to have "defaultEncoding" set to the this property. Maybe the test should be changed to run with -Dfile.encoding=COMPAT or else change the two usages of defaultEncoding in the test.

Since the test is asserting UTF-8 path names, it is not affected by file.encoding being COMPAT or not. To make it less confusing, I renamed the field name to filePathEncoding.

naotoj · 2021-07-20T20:37:50Z

Good work!

One thing we missed when adding Console.charset() was the code example in OutputStreamWriter's class description. It uses System.out and should have been changed to "anOutputStream" to avoid people copying this usage. We can do it as part of this PR or another but would be good to get it consistent with InputStreamReader.

I included the leftover fix as well.

AlanBateman · 2021-07-21T09:46:20Z

src/java.base/share/classes/java/lang/System.java

+     *     one from {@code native.encoding} system property during runtime startup.
+     *     Specifying it to {@code UTF-8}, or no value is set, defaults to use
+     *     {@code UTF-8}. Other values have unspecified behavior.
+     *     </td></tr>


Here's a suggesting re-wording to consider:

The name of the default charset, defaults to "UTF-8". The property may be set on the command line to the value "UTF-8" or "COMPAT". If set on the command line to the value "COMPAT" then the value is replaced with the value of the native.encoding property during startup. Setting the property to a value other than "UFT-8" or "COMPAT" leads to unspecified behavior.

Thanks, Alan. Updated the description as suggested.

AlanBateman

Thanks for incorporating the suggestion for the getProperties text. I think it looks good now.

openjdk · 2021-07-22T21:37:26Z

@naotoj This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8260265: UTF-8 by Default

Reviewed-by: alanb, rriggs

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 2 new commits pushed to the master branch:

1fb798d: 8272915: (doc) package-info typo in extLink
5116784: 8273091: Doc of [Strict]Math.floorDiv(long,int) erroneously documents int in @return tag

Please see this link for an up-to-date comparison between the source branch of this pull request and the master branch.
As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

naotoj · 2021-08-30T21:12:17Z

/integrate

openjdk · 2021-08-30T21:14:27Z

Going to push as commit 7fc8540.
Since your change was applied there have been 17 commits pushed to the master branch:

3204853: 8272343: Remove MetaspaceClosure::FLAG_MASK
fecefb8: 8271302: Regex Test Refresh
f18c0fa: 8271560: sun/security/ssl/DHKeyExchange/LegacyDHEKeyExchange.java still fails due to "An established connection was aborted by the software in your host machine"
5aaa20f: 8272861: Add a micro benchmark for vector api
7a01ba6: 8272093: Extract evacuation failure injection from G1CollectedHeap
98b9d98: 8272797: Mutex with rank safepoint_check_never imply allow_vm_block
f11e099: 8272651: G1 heap region info print order changed by JDK-8269914
fbffa54: 8270438: "Cores to use" output in configure is misleading
5185dbd: 8273098: Unnecessary Vector usage in java.naming
276b07b: 8271490: [ppc] [s390]: Crash in JavaThread::pd_get_top_frame_for_profiling
... and 7 more: https://git.openjdk.java.net/jdk/compare/e66c8afb59b57c4546656efa97f723f084964330...master

Your commit was automatically rebased without conflicts.

openjdk · 2021-08-30T21:14:38Z

@naotoj Pushed as commit 7fc8540.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

8260265: UTF-8 by Default

107210c

openjdk bot added the csr Pull request needs approved CSR before integration label Jul 8, 2021

openjdk bot added hotspot-runtime hotspot-runtime-dev@openjdk.org core-libs core-libs-dev@openjdk.org net net-dev@openjdk.org labels Jul 8, 2021

naotoj marked this pull request as ready for review July 8, 2021 22:02

openjdk bot added the rfr Pull request is ready for review label Jul 8, 2021

This comment has been minimized.

Sign in to view

RogerRiggs reviewed Jul 14, 2021

View reviewed changes

naotoj added 2 commits July 14, 2021 10:27

Merge branch 'master' into JDK-8260265

118bff8

Reflects review comments

981200f

naotoj added 2 commits July 14, 2021 14:56

Reflects comments

182dcb6

Removed leftover console references in PrintStream

38b8d98

gbaso mentioned this pull request Jul 17, 2021

Logback should default to JVM's default charset instead of ASCII spring-projects/spring-boot#27230

Closed

AlanBateman reviewed Jul 20, 2021

View reviewed changes

naotoj added 2 commits July 20, 2021 13:19

Reflects comments

07dffb9

Merge branch 'master' into JDK-8260265

a7c1ed4

AlanBateman approved these changes Jul 21, 2021

View reviewed changes

AlanBateman reviewed Jul 21, 2021

View reviewed changes

naotoj added 2 commits July 21, 2021 10:18

Refined file.encoding description

0e1dcc2

Merge branch 'master' into JDK-8260265

6f9e5eb

RogerRiggs approved these changes Jul 21, 2021

View reviewed changes

AlanBateman approved these changes Jul 22, 2021

View reviewed changes

openjdk bot removed the csr Pull request needs approved CSR before integration label Jul 22, 2021

openjdk bot added the ready Pull request is ready to be integrated label Jul 22, 2021

naotoj added 7 commits July 26, 2021 10:31

Merge branch 'master' into JDK-8260265

b5a0ee5

Merge branch 'master' into JDK-8260265

4610fe4

Merge branch 'master' into JDK-8260265

7d5137d

Merge branch 'master' into JDK-8260265

5434073

Merge branch 'master' into JDK-8260265

b946cd2

Removed "default" alias

67e1d4a

Fixed a failing test

28e79d4

AlanBateman approved these changes Aug 28, 2021

View reviewed changes

openjdk bot closed this Aug 30, 2021

openjdk bot added integrated Pull request has been integrated and removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Aug 30, 2021

naotoj mentioned this pull request Oct 20, 2021

8270490: Charset.forName() taking fallback default value #6045

Closed

3 tasks

jglick mentioned this pull request Oct 21, 2021

Write SSH keys in default encoding jenkinsci/credentials-binding-plugin#146

Draft

jglick mentioned this pull request Jan 4, 2022

Fix more DM_DEFAULT_ENCODING SpotBugs violations jenkinsci/jenkins#6098

Merged

10 tasks

8260265: UTF-8 by Default #4733

8260265: UTF-8 by Default #4733

Conversation

naotoj commented Jul 8, 2021 • edited by openjdk bot

Progress

Issue

Reviewers

Reviewing

bridgekeeper bot commented Jul 8, 2021

naotoj commented Jul 8, 2021

openjdk bot commented Jul 8, 2021

openjdk bot commented Jul 8, 2021

mlbridge bot commented Jul 8, 2021 • edited

Webrevs

This comment has been minimized.

gbaso commented Jul 14, 2021 • edited by bridgekeeper bot

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

naotoj commented Jul 14, 2021

AlanBateman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

naotoj commented Jul 20, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlanBateman left a comment

Choose a reason for hiding this comment

openjdk bot commented Jul 22, 2021 • edited

naotoj commented Aug 30, 2021

openjdk bot commented Aug 30, 2021

openjdk bot commented Aug 30, 2021

naotoj commented Jul 8, 2021 •

edited by openjdk bot

mlbridge bot commented Jul 8, 2021 •

edited

gbaso commented Jul 14, 2021 •

edited by bridgekeeper bot

openjdk bot commented Jul 22, 2021 •

edited