Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8231640: (prop) Canonical property storage #5372

Closed
wants to merge 28 commits into from

Conversation

jaikiran
Copy link
Member

@jaikiran jaikiran commented Sep 4, 2021

The commit in this PR implements the proposal for enhancement that was discussed in the core-libs-dev mailing list recently[1], for https://bugs.openjdk.java.net/browse/JDK-8231640

At a high level - the store() APIs in Properties have been modified to now look for the SOURCE_DATE_EPOCH environment variable[2]. If that env variable is set, then instead of writing out the current date time as a date comment, the store() APIs instead will use the value set for this env variable to parse it to a Date and write out the string form of such a date. The implementation here uses the d MMM yyyy HH:mm:ss 'GMT' date format and Locale.ROOT to format and write out such a date. This should provide reproducibility whenever the SOURCE_DATE_EPOCH is set. Furthermore, intentionally, no changes in the date format of the "current date" have been done.

These modified store() APIs work in the presence of the SecurityManager too. The caller is expected to have a read permission on the SOURCE_DATE_EPOCH environment variable. If the caller doesn't have that permission, then the implementation of these store() APIs will write out the "current date" and will ignore any value that has been set for the SOURCE_DATE_EPOCH env variable. This should allow for backward compatibility of existing applications, where, when they run under a SecurityManager and perhaps with an existing restrictive policy file, the presence of SOURCE_DATE_EPOCH shouldn't impact their calls to the store() APIs.

The modified store() APIs will also ignore any value for SOURCE_DATE_EPOCH that cannot be parsed to an long value. In such cases, the store() APIs will write out the "current date" and ignore the value set for this environment variable. No exceptions will be thrown for such invalid values. This is an additional backward compatibility precaution to prevent any rogue value for SOURCE_DATE_EPOCH from breaking applications.

An additional change in the implementation of these store() APIs and unrelated to the date comment, is that these APIs will now write out the property keys in a deterministic order. The keys will be written out in the natural ordering as specified by java.lang.String#compareTo() API.

The combination of the ordering of the property keys when written out and the usage of SOURCE_DATE_EPOCH environment value to determine the date comment should together allow for reproducibility of the output generated by these store() APIs.

New jtreg test classes have been introduced to verify these changes. The primary focus of PropertiesStoreTest is the ordering aspects of the property keys that are written out. On the other hand StoreReproducibilityTest focuses on the reproducibility of the output generated by these APIs. The StoreReproducibilityTest runs these tests both in the presence and absence of SecurityManager. Plus, in the presence of SecurityManager, it tests both the scenarios where the caller is granted the requisite permission and in other case not granted that permission.

These new tests and existing tests under test/jdk/java/util/Properties/ pass with these changes.

[1] https://mail.openjdk.java.net/pipermail/core-libs-dev/2021-August/080758.html
[2] https://reproducible-builds.org/specs/source-date-epoch/


Progress

  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue
  • Change must be properly reviewed

Issue

Reviewers

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/5372/head:pull/5372
$ git checkout pull/5372

Update a local copy of the PR:
$ git checkout pull/5372
$ git pull https://git.openjdk.java.net/jdk pull/5372/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 5372

View PR using the GUI difftool:
$ git pr show -t 5372

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/5372.diff

@jaikiran
Copy link
Member Author

jaikiran commented Sep 4, 2021

I haven't yet created a CSR for this and will do that next week once the initial reviews and changes are sorted out.

@bridgekeeper
Copy link

bridgekeeper bot commented Sep 4, 2021

👋 Welcome back jpai! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@openjdk openjdk bot added the rfr Pull request is ready for review label Sep 4, 2021
@openjdk
Copy link

openjdk bot commented Sep 4, 2021

@jaikiran The following label will be automatically applied to this pull request:

  • core-libs

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

@openjdk openjdk bot added the core-libs core-libs-dev@openjdk.org label Sep 4, 2021
synchronized (this) {
for (Map.Entry<Object, Object> e : entrySet()) {
for (Map.Entry<Object, Object> e : new TreeMap<>(map).entrySet()) {
Copy link
Member

@turbanoff turbanoff Sep 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this sorting intentionally added? It's not clear from issue description or PR description that order of properties should be changed too.
Anyway I think copying entrySet() to array and then sorting should be faster, than creating a TreeMap

Copy link

@rfscholte rfscholte Sep 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case of reproducibility it should be at least ordered, i.e. keep original input order.

@mlbridge
Copy link

mlbridge bot commented Sep 5, 2021

Mailing list message from Alan Bateman on core-libs-dev:

On 04/09/2021 16:50, Jaikiran Pai wrote:

The commit in this PR implements the proposal for enhancement that was discussed in the core-libs-dev mailing list recently[1], for https://bugs.openjdk.java.net/browse/JDK-8231640

At a high level - the `store()` APIs in `Properties` have been modified to now look for the `SOURCE_DATE_EPOCH` environment variable[2]. If that env variable is set, then instead of writing out the current date time as a date comment, the `store()` APIs instead will use the value set for this env variable to parse it to a `Date` and write out the string form of such a date. The implementation here uses the `d MMM yyyy HH:mm:ss 'GMT'` date format and `Locale.ROOT` to format and write out such a date. This should provide reproducibility whenever the `SOURCE_DATE_EPOCH` is set. Furthermore, intentionally, no changes in the date format of the "current date" have been done.

These modified `store()` APIs work in the presence of the `SecurityManager` too. The caller is expected to have a read permission on the `SOURCE_DATE_EPOCH` environment variable. If the caller doesn't have that permission, then the implementation of these `store()` APIs will write out the "current date" and will ignore any value that has been set for the `SOURCE_DATE_EPOCH` env variable. This should allow for backward compatibility of existing applications, where, when they run under a `SecurityManager` and perhaps with an existing restrictive policy file, the presence of `SOURCE_DATE_EPOCH` shouldn't impact their calls to the `store()` APIs.

The modified `store()` APIs will also ignore any value for `SOURCE_DATE_EPOCH` that cannot be parsed to an `long` value. In such cases, the `store()` APIs will write out the "current date" and ignore the value set for this environment variable. No exceptions will be thrown for such invalid values. This is an additional backward compatibility precaution to prevent any rogue value for `SOURCE_DATE_EPOCH` from breaking applications.

In the discussion on this option then I think we were talking about
changing the specification of the store methods to allow for an
implementation specific means to override the date and put the
discussion on SOURCE_DATE_EPOCH in an @implNote.

The SM case probably needs discussion as I was expecting to see
something like this:

PrivilegedAction<String> pa = () -> System.getenv("SOURCE_DATE_EPOCH");
String value = AccessController.doPrivileged(pa);

as a caller of the store method (in a library for example) is unlikely
to have this permission. I assume your concern is that untrusted code
would have a way to read this specific env variable without a permission
(by way of creating and storing an empty Properties)?? In this case I'm
mostly wondering if it's really worth having a behavior difference in
this mode.

An additional change in the implementation of these `store()` APIs and unrelated to the date comment, is that these APIs will now write out the property keys in a deterministic order. The keys will be written out in the natural ordering as specified by `java.lang.String#compareTo()` API.

The store methods are already specified to throw CCE if the properties
has non-String keys so this should be okay.

-Alan

@mlbridge
Copy link

mlbridge bot commented Sep 5, 2021

Mailing list message from Jaikiran Pai on core-libs-dev:

Hello Alan,

On 05/09/21 1:46 pm, Alan Bateman wrote:

On 04/09/2021 16:50, Jaikiran Pai wrote:

The commit in this PR implements the proposal for enhancement that
was discussed in the core-libs-dev mailing list recently[1], for
https://bugs.openjdk.java.net/browse/JDK-8231640

At a high level - the `store()` APIs in `Properties` have been
modified to now look for the `SOURCE_DATE_EPOCH` environment
variable[2]. If that env variable is set, then instead of writing out
the current date time as a date comment, the `store()` APIs instead
will use the value set for this env variable to parse it to a `Date`
and write out the string form of such a date. The implementation here
uses the `d MMM yyyy HH:mm:ss 'GMT'` date format and `Locale.ROOT` to
format and write out such a date. This should provide reproducibility
whenever the `SOURCE_DATE_EPOCH` is set. Furthermore, intentionally,
no changes in the date format of the "current date" have been done.

These? modified `store()` APIs work in the presence of the
`SecurityManager` too. The caller is expected to have a read
permission on the `SOURCE_DATE_EPOCH` environment variable. If the
caller doesn't have that permission, then the implementation of these
`store()` APIs will write out the "current date" and will ignore any
value that has been set for the `SOURCE_DATE_EPOCH` env variable.
This should allow for backward compatibility of existing
applications, where, when they run under a `SecurityManager` and
perhaps with an existing restrictive policy file, the presence of
`SOURCE_DATE_EPOCH` shouldn't impact their calls to the `store()` APIs.

The modified `store()` APIs will also ignore any value for
`SOURCE_DATE_EPOCH` that? cannot be parsed to an `long` value. In
such cases, the `store()` APIs will write out the "current date" and
ignore the value set for this environment variable. No exceptions
will be thrown for such invalid values. This is an additional
backward compatibility precaution to prevent any rogue value for
`SOURCE_DATE_EPOCH` from breaking applications.
In the discussion on this option then I think we were talking about
changing the specification of the store methods to allow for an
implementation specific means to override the date and put the
discussion on SOURCE_DATE_EPOCH in an @implNote.

I will move the updated javadoc to an @implNote then. I guess, the
existing part where it explains how the current date comment is written
out, should stay where it is currently?

The SM case probably needs discussion as I was expecting to see
something like this:

PrivilegedAction<String> pa = () -> System.getenv("SOURCE_DATE_EPOCH");
String value = AccessController.doPrivileged(pa);

as a caller of the store method (in a library for example) is unlikely
to have this permission. I assume your concern is that untrusted code
would have a way to read this specific env variable without a
permission (by way of creating and storing an empty Properties)?? In
this case I'm mostly wondering if it's really worth having a behavior
difference in this mode.

I had initially started off with using a doPrivileged code in this
change. However, I soon realized that it might be a security hole, like
in the example you state. Unlike other parts of the JDK code where the
doPrivileged makes sense, since the "action" part of it does things that
are internal implementation details to the JDK, this new piece of code
that we are introducing, does a System.getenv(...) in its "action" but
will then additionally hand out the value (in a formatted manner of
course) of that environment variable to the callers, through the
OutputStream/Writer instances that the callers pass in to the store()
APIs. Effectively, this means that even when getenv.SOURCE_DATE_EPOCH
(or getenv.* for that matter) haven't been granted to the callers, they
will _always_ be able to get hold of that environment variable's value
through this approach. Like you state, they could just call
Properties.store() (which by the way, doesn't necessarily have to be
empty) to bypass the security checks that are put in place for the
direct System.getenv(...) calls from their code.

In such a doPrivelged case, the only way the administrator can prevent
this security bypass, would then be to _not_ grant this permission to
the JDK code itself, which then effectively means that this enhancement
for using SOURCE_DATE_EPOCH will not take effect at all and the date
comment will always write out the current date.

So the doPriveleged case, IMO, will not allow selective granting of
permissions to callers. The alternate approach that's used in this PR,
on the other hand allows for selective granting of permissions. I'm not
too aware of how things stand with the application server and security
manager integration these days, but I would guess that in such use cases
where you probably would want to control which deployed applications
(within the same JVM) are granted what permissions, this alternate
approach of not using the doPriveleged would make it feasible to control
these permissions.

-Jaikiran

@mlbridge
Copy link

mlbridge bot commented Sep 5, 2021

Mailing list message from Jaikiran Pai on core-libs-dev:

Hello Andrey,

On 05/09/21 12:02 am, Andrey Turbanov wrote:

src/java.base/share/classes/java/util/Properties.java line 924:

922: writeDateComment(bw);
923: synchronized (this) {
924: for (Map.Entry<Object, Object> e : new TreeMap<>(map).entrySet()) {
Is this sorting intentionally added? It's not clear from issue description or PR description that order of properties should be changed too.
Anyway I think copying to array and then sorting should be faster, that creating TreeMap

Yes, the ordering of the properties is intentional as noted in the
description of this PR as well as the linked mail discussion thread.

As for the usage of TreeMap, I had looked around the JDK code to see if
there are more performant ways to order that existing Map, but I
couldn't find any. Do you mean that converting the keySet() of an
existing Map into an array and then sorting that array and then using
that sorted array to iterate and using these keys to do an additional
lookup for value against the original Map would be more efficient in
this case? I can experiment with it in a simple JMH benchmark with some
decent/regular sized Map and see how it performs.

-Jaikiran

@jaikiran
Copy link
Member Author

jaikiran commented Sep 5, 2021

Off topic - the tier1 failures that are being reported in GitHub action jobs seem genuine but unrelated to this change. The failure is consistently in a single test InfoOptsTest and it fails with the following error when running the testUniqueInfoOpts test case. Looks like a recent issue in some commit. I'll sort that out separately.
Here's the error from the logs:

2021-09-04T16:37:24.7528210Z Running test testUniqueInfoOpts
2021-09-04T16:37:24.7630470Z Main [--help, --help] []
2021-09-04T16:37:24.7635480Z rc:0
2021-09-04T16:37:24.7736940Z javac/DIRECT:
2021-09-04T16:37:24.7762440Z Usage: javac <options> <source files>
2021-09-04T16:37:24.7863980Z where possible options include:
2021-09-04T16:37:24.7892480Z   @<filename>                  Read options and filenames from file
2021-09-04T16:37:24.7986700Z   -Akey[=value]                Options to pass to annotation processors
2021-09-04T16:37:24.8003100Z   --add-modules <module>(,<module>)*
2021-09-04T16:37:24.8030720Z         Root modules to resolve in addition to the initial modules, or all modules
2021-09-04T16:37:24.8106550Z         on the module path if <module> is ALL-MODULE-PATH.
2021-09-04T16:37:24.8108330Z   --boot-class-path <path>, -bootclasspath <path>
2021-09-04T16:37:24.8110830Z         Override location of bootstrap class files
2021-09-04T16:37:24.8112470Z   --class-path <path>, -classpath <path>, -cp <path>
2021-09-04T16:37:24.8113590Z         Specify where to find user class files and annotation processors
2021-09-04T16:37:24.8115140Z   -d <directory>               Specify where to place generated class files
2021-09-04T16:37:24.8116530Z   -deprecation
2021-09-04T16:37:24.8117490Z         Output source locations where deprecated APIs are used
2021-09-04T16:37:24.8118770Z   --enable-preview
2021-09-04T16:37:24.8120330Z         Enable preview language features. To be used in conjunction with either -source or --release.
2021-09-04T16:37:24.8129810Z   -encoding <encoding>         Specify character encoding used by source files
2021-09-04T16:37:24.8136440Z   -endorseddirs <dirs>         Override location of endorsed standards path
2021-09-04T16:37:24.8239190Z   -extdirs <dirs>              Override location of installed extensions
2021-09-04T16:37:24.8251400Z   -g                           Generate all debugging info
2021-09-04T16:37:24.8290250Z   -g:{lines,vars,source}       Generate only some debugging info
2021-09-04T16:37:24.8291910Z   -g:none                      Generate no debugging info
2021-09-04T16:37:24.8293200Z   -h <directory>
2021-09-04T16:37:24.8294170Z         Specify where to place generated native header files
2021-09-04T16:37:24.8296270Z   --help, -help, -?            Print this help message
2021-09-04T16:37:24.8297720Z   --help-extra, -X             Print help on extra options
2021-09-04T16:37:24.8299160Z   -implicit:{none,class}
2021-09-04T16:37:24.8300250Z         Specify whether or not to generate class files for implicitly referenced files
2021-09-04T16:37:24.8301820Z   -J<flag>                     Pass <flag> directly to the runtime system
2021-09-04T16:37:24.8303230Z   --limit-modules <module>(,<module>)*
2021-09-04T16:37:24.8304210Z         Limit the universe of observable modules
2021-09-04T16:37:24.8305570Z   --module <module>(,<module>)*, -m <module>(,<module>)*
2021-09-04T16:37:24.8306630Z         Compile only the specified module(s), check timestamps
2021-09-04T16:37:24.8307980Z   --module-path <path>, -p <path>
2021-09-04T16:37:24.8309050Z         Specify where to find application modules
2021-09-04T16:37:24.8310500Z   --module-source-path <module-source-path>
2021-09-04T16:37:24.8311660Z         Specify where to find input source files for multiple modules
2021-09-04T16:37:24.8313050Z   --module-version <version>
2021-09-04T16:37:24.8314040Z         Specify version of modules that are being compiled
2021-09-04T16:37:24.8315390Z   -nowarn                      Generate no warnings
2021-09-04T16:37:24.8316640Z   -parameters
2021-09-04T16:37:24.8317620Z         Generate metadata for reflection on method parameters
2021-09-04T16:37:24.8361300Z   -proc:{none,only}
2021-09-04T16:37:24.8362510Z         Control whether annotation processing and/or compilation is done.
2021-09-04T16:37:24.8364150Z   -processor <class1>[,<class2>,<class3>...]
2021-09-04T16:37:24.8365410Z         Names of the annotation processors to run; bypasses default discovery process
2021-09-04T16:37:24.8366910Z   --processor-module-path <path>
2021-09-04T16:37:24.8367990Z         Specify a module path where to find annotation processors
2021-09-04T16:37:24.8369540Z   --processor-path <path>, -processorpath <path>
2021-09-04T16:37:24.8370620Z         Specify where to find annotation processors
2021-09-04T16:37:24.8371840Z   -profile <profile>
2021-09-04T16:37:24.8372800Z         Check that API used is available in the specified profile
2021-09-04T16:37:24.8374080Z   --release <release>
2021-09-04T16:37:24.8375110Z         Compile for the specified Java SE release. Supported releases: 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18
2021-09-04T16:37:24.8377060Z   -s <directory>               Specify where to place generated source files
2021-09-04T16:37:24.8378940Z   --source <release>, -source <release>
2021-09-04T16:37:24.8380150Z         Provide source compatibility with the specified Java SE release. Supported releases: 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18
2021-09-04T16:37:24.8381690Z   --source-path <path>, -sourcepath <path>
2021-09-04T16:37:24.8382670Z         Specify where to find input source files
2021-09-04T16:37:24.8383990Z   --system <jdk>|none          Override location of system modules
2021-09-04T16:37:24.8385370Z   --target <release>, -target <release>
2021-09-04T16:37:24.8386570Z         Generate class files suitable for the specified Java SE release. Supported releases: 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18
2021-09-04T16:37:24.8388070Z   --upgrade-module-path <path>
2021-09-04T16:37:24.8389040Z         Override location of upgradeable modules
2021-09-04T16:37:24.8390480Z   -verbose                     Output messages about what the compiler is doing
2021-09-04T16:37:24.8391930Z   --version, -version          Version information
2021-09-04T16:37:24.8393380Z   -Werror                      Terminate compilation if warnings occur
2021-09-04T16:37:24.8394150Z 
2021-09-04T16:37:24.8395160Z Main [-X, -X] []
2021-09-04T16:37:24.8395860Z rc:0
2021-09-04T16:37:24.8396520Z javac/DIRECT:
2021-09-04T16:37:24.8397830Z   --add-exports <module>/<package>=<other-module>(,<other-module>)*
2021-09-04T16:37:24.8399140Z         Specify a package to be considered as exported from its defining module
2021-09-04T16:37:24.8400780Z         to additional modules, or to all unnamed modules if <other-module> is ALL-UNNAMED.
2021-09-04T16:37:24.8403050Z   --add-reads <module>=<other-module>(,<other-module>)*
2021-09-04T16:37:24.8404200Z         Specify additional modules to be considered as required by a given module.
2021-09-04T16:37:24.8405750Z         <other-module> may be ALL-UNNAMED to require the unnamed module.
2021-09-04T16:37:24.8407430Z   --default-module-for-created-files <module-name>
2021-09-04T16:37:24.8408890Z         Fallback target module for files created by annotation processors, if none specified or inferred.
2021-09-04T16:37:24.8410640Z   -Djava.endorsed.dirs=<dirs>  Override location of endorsed standards path
2021-09-04T16:37:24.8412280Z   -Djava.ext.dirs=<dirs>       Override location of installed extensions
2021-09-04T16:37:24.8413820Z   --help-lint                  Print the supported keys for -Xlint
2021-09-04T16:37:24.8415200Z   --patch-module <module>=<file>(:<file>)*
2021-09-04T16:37:24.8416210Z         Override or augment a module with classes and resources
2021-09-04T16:37:24.8417110Z         in JAR files or directories
2021-09-04T16:37:24.8418510Z   -Xbootclasspath:<path>       Override location of bootstrap class files
2021-09-04T16:37:24.8420080Z   -Xbootclasspath/a:<path>     Append to the bootstrap class path
2021-09-04T16:37:24.8421630Z   -Xbootclasspath/p:<path>     Prepend to the bootstrap class path
2021-09-04T16:37:24.8423170Z   -Xdiags:{compact,verbose}    Select a diagnostic mode
2021-09-04T16:37:24.8424430Z   -Xdoclint
2021-09-04T16:37:24.8425370Z         Enable recommended checks for problems in javadoc comments
2021-09-04T16:37:24.8426730Z   -Xdoclint:(all|none|[-]<group>)[/<access>]
2021-09-04T16:37:24.8427770Z         Enable or disable specific checks for problems in javadoc comments,
2021-09-04T16:37:24.8428920Z         where <group> is one of accessibility, html, missing, reference, or syntax,
2021-09-04T16:37:24.8430170Z         and <access> is one of public, protected, package, or private.
2021-09-04T16:37:24.8431760Z   -Xdoclint/package:[-]<packages>(,[-]<package>)*
2021-09-04T16:37:24.8432880Z         Enable or disable checks in specific packages. Each <package> is either the
2021-09-04T16:37:24.8434310Z         qualified name of a package or a package name prefix followed by .*, which
2021-09-04T16:37:24.8435940Z         expands to all sub-packages of the given package. Each <package> can be prefixed
2021-09-04T16:37:24.8438060Z         with - to disable checks for the specified package or packages.
2021-09-04T16:37:24.8439610Z   -Xlint                       Enable recommended warnings
2021-09-04T16:37:24.8440840Z   -Xlint:<key>(,<key>)*
2021-09-04T16:37:24.8441750Z         Warnings to enable or disable, separated by comma.
2021-09-04T16:37:24.8443150Z         Precede a key by - to disable the specified warning.
2021-09-04T16:37:24.8444570Z         Use --help-lint to see the supported keys.
2021-09-04T16:37:24.8446010Z   -Xmaxerrs <number>           Set the maximum number of errors to print
2021-09-04T16:37:24.8447550Z   -Xmaxwarns <number>          Set the maximum number of warnings to print
2021-09-04T16:37:24.8550380Z   -Xpkginfo:{always,legacy,nonempty}
2021-09-04T16:37:24.8636310Z         Specify handling of package-info files
2021-09-04T16:37:24.8639600Z   -Xplugin:"name args"
2021-09-04T16:37:24.8663470Z         Name and optional arguments for a plug-in to be run
2021-09-04T16:37:24.8665480Z   -Xprefer:{source,newer}
2021-09-04T16:37:24.8683700Z         Specify which file to read when both a source file and class file are found for an implicitly compiled class
2021-09-04T16:37:24.8728280Z   -Xprint
2021-09-04T16:37:24.8729520Z         Print out a textual representation of specified types
2021-09-04T16:37:24.8730900Z   -XprintProcessorInfo
2021-09-04T16:37:24.8731960Z         Print information about which annotations a processor is asked to process
2021-09-04T16:37:24.8733270Z   -XprintRounds
2021-09-04T16:37:24.8734200Z         Print information about rounds of annotation processing
2021-09-04T16:37:24.8736230Z   -Xstdout <filename>          Redirect standard output
2021-09-04T16:37:24.8736940Z 
2021-09-04T16:37:24.8737760Z These extra options are subject to change without notice.
2021-09-04T16:37:24.8739170Z Main [--help-lint, --help-lint] []
2021-09-04T16:37:24.8739960Z rc:0
2021-09-04T16:37:24.8740890Z javac/DIRECT:
2021-09-04T16:37:24.8742040Z The supported keys for -Xlint are:
2021-09-04T16:37:24.8742920Z     all                  Enable all warnings
2021-09-04T16:37:24.8743980Z     auxiliaryclass       Warn about an auxiliary class that is hidden in a source file, and is used from other files.
2021-09-04T16:37:24.8744800Z     cast                 Warn about use of unnecessary casts.
2021-09-04T16:37:24.8745500Z     classfile            Warn about issues related to classfile contents.
2021-09-04T16:37:24.8746200Z     deprecation          Warn about use of deprecated items.
2021-09-04T16:37:24.8747550Z     dep-ann              Warn about items marked as deprecated in JavaDoc but not using the @Deprecated annotation.
2021-09-04T16:37:24.8748410Z     divzero              Warn about division by constant integer 0.
2021-09-04T16:37:24.8749030Z     empty                Warn about empty statement after if.
2021-09-04T16:37:24.8749680Z     exports              Warn about issues regarding module exports.
2021-09-04T16:37:24.8750470Z     fallthrough          Warn about falling through from one case of a switch statement to the next.
2021-09-04T16:37:24.8751300Z     finally              Warn about finally clauses that do not terminate normally.
2021-09-04T16:37:24.8752780Z     missing-explicit-ctor Warn about missing explicit constructors in public and protected classes in exported packages.
2021-09-04T16:37:24.8753780Z     module               Warn about module system related issues.
2021-09-04T16:37:24.8754410Z     opens                Warn about issues regarding module opens.
2021-09-04T16:37:24.8755110Z     options              Warn about issues relating to use of command line options.
2021-09-04T16:37:24.8755850Z     overloads            Warn about issues regarding method overloads.
2021-09-04T16:37:24.8756560Z     overrides            Warn about issues regarding method overrides.
2021-09-04T16:37:24.8757270Z     path                 Warn about invalid path elements on the command line.
2021-09-04T16:37:24.8758010Z     processing           Warn about issues regarding annotation processing.
2021-09-04T16:37:24.8759040Z     rawtypes             Warn about use of raw types.
2021-09-04T16:37:24.8759730Z     removal              Warn about use of API that has been marked for removal.
2021-09-04T16:37:24.8761030Z     requires-automatic   Warn about use of automatic modules in the requires clauses.
2021-09-04T16:37:24.8762640Z     requires-transitive-automatic Warn about automatic modules in requires transitive.
2021-09-04T16:37:24.8763680Z     serial               Warn about Serializable classes that do not provide a serial version ID. 
2021-09-04T16:37:24.8765050Z                          Also warn about access to non-public members from a serializable element.
2021-09-04T16:37:24.8765870Z     static               Warn about accessing a static member using an instance.
2021-09-04T16:37:24.8766610Z     strictfp             Warn about unnecessary use of the strictfp modifier.
2021-09-04T16:37:24.8767930Z     synchronization      Warn about synchronization attempts on instances of value-based classes.
2021-09-04T16:37:24.8769460Z     text-blocks          Warn about inconsistent white space characters in text block indentation.
2021-09-04T16:37:24.8770810Z     try                  Warn about issues relating to use of try blocks (i.e. try-with-resources).
2021-09-04T16:37:24.8771580Z     unchecked            Warn about unchecked operations.
2021-09-04T16:37:24.8772250Z     varargs              Warn about potentially unsafe vararg methods.
2021-09-04T16:37:24.8772950Z     preview              Warn about use of preview language features.
2021-09-04T16:37:24.8773540Z     none                 Disable all warnings
2021-09-04T16:37:24.8774850Z Main [-version, -version] []
2021-09-04T16:37:24.8775370Z rc:0
2021-09-04T16:37:24.8775690Z javac/DIRECT:
2021-09-04T16:37:24.8776460Z javac 18-internal
2021-09-04T16:37:24.8777290Z Main [-fullversion, -fullversion] []
2021-09-04T16:37:24.8777740Z rc:0
2021-09-04T16:37:24.8778060Z javac/DIRECT:
2021-09-04T16:37:24.8779450Z javac full version "18-internal+0-jaikiran-85748cf4a8efb69cbe69667851a14321804a51b6"
2021-09-04T16:37:24.8780590Z >>>>> Expected string appears more than once: 18
2021-09-04T16:37:24.8780950Z 
2021-09-04T16:37:24.8781400Z java.lang.Exception: 1 errors occurred
2021-09-04T16:37:24.8782230Z 	at OptionModesTester.runTests(OptionModesTester.java:81)
2021-09-04T16:37:24.8783100Z 	at InfoOptsTest.main(InfoOptsTest.java:41)
2021-09-04T16:37:24.8784350Z 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2021-09-04T16:37:24.8786310Z 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
2021-09-04T16:37:24.8788620Z 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2021-09-04T16:37:24.8791250Z 	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
2021-09-04T16:37:24.8792700Z 	at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
2021-09-04T16:37:24.8794000Z 	at java.base/java.lang.Thread.run(Thread.java:833)
2021-09-04T16:37:24.8794420Z 
2021-09-04T16:37:24.8794980Z JavaTest Message: Test threw exception: java.lang.Exception
2021-09-04T16:37:24.8795660Z JavaTest Message: shutting down test
2021-09-04T16:37:24.8795990Z 
2021-09-04T16:37:24.8796190Z 
2021-09-04T16:37:24.8797640Z TEST RESULT: Failed. Execution failed: `main' threw exception: java.lang.Exception: 1 errors occurred

@jaikiran
Copy link
Member Author

jaikiran commented Sep 5, 2021

The failure is consistently in a single test InfoOptsTest and it fails with the following error when running the testUniqueInfoOpts test case. Looks like a recent issue in some commit. I'll sort that out separately.

Created https://bugs.openjdk.java.net/browse/JDK-8273361

@mlbridge
Copy link

mlbridge bot commented Sep 7, 2021

Mailing list message from Jaikiran Pai on core-libs-dev:

On 05/09/21 6:01 pm, Jaikiran Pai wrote:

Hello Alan,

On 05/09/21 1:46 pm, Alan Bateman wrote:

On 04/09/2021 16:50, Jaikiran Pai wrote:

The commit in this PR implements the proposal for enhancement that
was discussed in the core-libs-dev mailing list recently[1], for
https://bugs.openjdk.java.net/browse/JDK-8231640

At a high level - the `store()` APIs in `Properties` have been
modified to now look for the `SOURCE_DATE_EPOCH` environment
variable[2]. If that env variable is set, then instead of writing
out the current date time as a date comment, the `store()` APIs
instead will use the value set for this env variable to parse it to
a `Date` and write out the string form of such a date. The
implementation here uses the `d MMM yyyy HH:mm:ss 'GMT'` date format
and `Locale.ROOT` to format and write out such a date. This should
provide reproducibility whenever the `SOURCE_DATE_EPOCH` is set.
Furthermore, intentionally, no changes in the date format of the
"current date" have been done.

These? modified `store()` APIs work in the presence of the
`SecurityManager` too. The caller is expected to have a read
permission on the `SOURCE_DATE_EPOCH` environment variable. If the
caller doesn't have that permission, then the implementation of
these `store()` APIs will write out the "current date" and will
ignore any value that has been set for the `SOURCE_DATE_EPOCH` env
variable. This should allow for backward compatibility of existing
applications, where, when they run under a `SecurityManager` and
perhaps with an existing restrictive policy file, the presence of
`SOURCE_DATE_EPOCH` shouldn't impact their calls to the `store()` APIs.

The modified `store()` APIs will also ignore any value for
`SOURCE_DATE_EPOCH` that? cannot be parsed to an `long` value. In
such cases, the `store()` APIs will write out the "current date" and
ignore the value set for this environment variable. No exceptions
will be thrown for such invalid values. This is an additional
backward compatibility precaution to prevent any rogue value for
`SOURCE_DATE_EPOCH` from breaking applications.
In the discussion on this option then I think we were talking about
changing the specification of the store methods to allow for an
implementation specific means to override the date and put the
discussion on SOURCE_DATE_EPOCH in an @implNote.

I will move the updated javadoc to an @implNote then. I guess, the
existing part where it explains how the current date comment is
written out, should stay where it is currently?

I've now updated the PR to move the new text of this javadoc into a
@implNote.

-Jaikiran

@turbanoff
Copy link
Member

turbanoff commented Sep 7, 2021

Do you mean that converting the keySet() of an
existing Map into an array and then sorting that array and then using
that sorted array to iterate and using these keys to do an additional
lookup for value against the original Map would be more efficient in
this case?

You can convert entrySet() to array. Not a keySet. In this case there is no need to lookup for values.

@mlbridge
Copy link

mlbridge bot commented Sep 7, 2021

Mailing list message from Roger Riggs on core-libs-dev:

Hi,

The value of SOURCE_DATE_EPOCH is not so sensitive that it needs the
protections you are applying.
The doPriv only exposes the value of that specific environment variable
and in the usual case, it is undefined.

The complexity in the specification and implementation seem unnecessary
in this case.

Though java.util.Date is used in the current implementation, its use is
discouraged in new code
in favor of java.time.ZonedDateTime.
Consider if java.time.format.DateTimeFormatter.RFC_1123_DATE_TIME can be
used to parse and format the time.

Thanks, Roger

On 9/5/21 8:31 AM, Jaikiran Pai wrote:

Hello Alan,

On 05/09/21 1:46 pm, Alan Bateman wrote:

On 04/09/2021 16:50, Jaikiran Pai wrote:

The commit in this PR implements the proposal for enhancement that
was discussed in the core-libs-dev mailing list recently[1], for
https://bugs.openjdk.java.net/browse/JDK-8231640

At a high level - the `store()` APIs in `Properties` have been
modified to now look for the `SOURCE_DATE_EPOCH` environment
variable[2]. If that env variable is set, then instead of writing
out the current date time as a date comment, the `store()` APIs
instead will use the value set for this env variable to parse it to
a `Date` and write out the string form of such a date. The
implementation here uses the `d MMM yyyy HH:mm:ss 'GMT'` date format
and `Locale.ROOT` to format and write out such a date. This should
provide reproducibility whenever the `SOURCE_DATE_EPOCH` is set.
Furthermore, intentionally, no changes in the date format of the
"current date" have been done.

These? modified `store()` APIs work in the presence of the
`SecurityManager` too. The caller is expected to have a read
permission on the `SOURCE_DATE_EPOCH` environment variable. If the
caller doesn't have that permission, then the implementation of
these `store()` APIs will write out the "current date" and will
ignore any value that has been set for the `SOURCE_DATE_EPOCH` env
variable. This should allow for backward compatibility of existing
applications, where, when they run under a `SecurityManager` and
perhaps with an existing restrictive policy file, the presence of
`SOURCE_DATE_EPOCH` shouldn't impact their calls to the `store()` APIs.

The modified `store()` APIs will also ignore any value for
`SOURCE_DATE_EPOCH` that? cannot be parsed to an `long` value. In
such cases, the `store()` APIs will write out the "current date" and
ignore the value set for this environment variable. No exceptions
will be thrown for such invalid values. This is an additional
backward compatibility precaution to prevent any rogue value for
`SOURCE_DATE_EPOCH` from breaking applications.
In the discussion on this option then I think we were talking about
changing the specification of the store methods to allow for an
implementation specific means to override the date and put the
discussion on SOURCE_DATE_EPOCH in an @implNote.

I will move the updated javadoc to an @implNote then. I guess, the
existing part where it explains how the current date comment is
written out, should stay where it is currently?

The SM case probably needs discussion as I was expecting to see
something like this:

PrivilegedAction<String> pa = () -> System.getenv("SOURCE_DATE_EPOCH");
String value = AccessController.doPrivileged(pa);

as a caller of the store method (in a library for example) is
unlikely to have this permission. I assume your concern is that
untrusted code would have a way to read this specific env variable
without a permission (by way of creating and storing an empty
Properties)?? In this case I'm mostly wondering if it's really worth
having a behavior difference in this mode.

I had initially started off with using a doPrivileged code in this
change. However, I soon realized that it might be a security hole,
like in the example you state. Unlike other parts of the JDK code
where the doPrivileged makes sense, since the "action" part of it does
things that are internal implementation details to the JDK, this new
piece of code that we are introducing, does a System.getenv(...) in
its "action" but will then additionally hand out the value (in a
formatted manner of course) of that environment variable to the
callers, through the OutputStream/Writer instances that the callers
pass in to the store() APIs. Effectively, this means that even when
getenv.SOURCE_DATE_EPOCH (or getenv.* for that matter) haven't been
granted to the callers, they will _always_ be able to get hold of that
environment variable's value through this approach. Like you state,
they could just call Properties.store() (which by the way, doesn't
necessarily have to be empty) to bypass the security checks that are
put in place for the direct System.getenv(...) calls from their code.

In such a doPrivelged case, the only way the administrator can prevent
this security bypass, would then be to _not_ grant this permission to
the JDK code itself, which then effectively means that this
enhancement for using SOURCE_DATE_EPOCH will not take effect at all
and the date comment will always write out the current date.

So the doPriveleged case, IMO, will not allow selective granting of
permissions to callers. The alternate approach that's used in this PR,
on the other hand allows for selective granting of permissions. I'm
not too aware of how things stand with the application server and
security manager integration these days, but I would guess that in
such use cases where you probably would want to control which deployed
applications (within the same JVM) are granted what permissions, this
alternate approach of not using the doPriveleged would make it
feasible to control these permissions.

-Jaikiran

@mlbridge
Copy link

mlbridge bot commented Sep 7, 2021

Mailing list message from Jaikiran Pai on core-libs-dev:

On 07/09/21 8:35 pm, Roger Riggs wrote:

Hi,

The value of SOURCE_DATE_EPOCH is not so sensitive that it needs the
protections you are applying.
The doPriv only exposes the value of that specific environment
variable and in the usual case, it is undefined.

The complexity in the specification and implementation seem
unnecessary in this case.

Given the inputs so far, the doPriveleged approach appears to be
acceptable. So I'll go ahead and update the PR shortly to change this part.

Though java.util.Date is used in the current implementation, its use
is discouraged in new code
in favor of java.time.ZonedDateTime.
Consider if java.time.format.DateTimeFormatter.RFC_1123_DATE_TIME can
be used to parse and format the time.

Noted. I'll take a look at these and update the PR as necessary.

Thank you all for the inputs so far.

-Jaikiran

@mlbridge
Copy link

mlbridge bot commented Sep 7, 2021

Mailing list message from Alan Bateman on core-libs-dev:

On 07/09/2021 16:05, Roger Riggs wrote:

Hi,

The value of SOURCE_DATE_EPOCH is not so sensitive that it needs the
protections you are applying.
The doPriv only exposes the value of that specific environment
variable and in the usual case, it is undefined.

The complexity in the specification and implementation seem
unnecessary in this case.

I agree.? Given the complexity then it makes your suggestion/option to
just drop the date from the comment somewhat tempting.

-Alan

@mlbridge
Copy link

mlbridge bot commented Sep 8, 2021

Mailing list message from Stuart Marks on core-libs-dev:

On 9/7/21 8:27 AM, Jaikiran Pai wrote:

On 07/09/21 8:35 pm, Roger Riggs wrote:

Though java.util.Date is used in the current implementation, its use is
discouraged in new code
in favor of java.time.ZonedDateTime.
Consider if java.time.format.DateTimeFormatter.RFC_1123_DATE_TIME can be used to
parse and format the time.

Noted. I'll take a look at these and update the PR as necessary.

Unless there's an overriding reason, it might be nice to have the output format
match the format used in the Debian patch that adds SOURCE_DATE_EPOCH:

https://salsa.debian.org/openjdk-team/openjdk/-/blob/master/debian/patches/reproducible-properties-timestamp.diff

(See JDK-8272157 for the journey that led us here, in particular, lack of
doPrivileged when reading the environment. The Debian patch also doesn't provide a
reproducible order, which we've already agreed is necessary.)

s'marks

@mlbridge
Copy link

mlbridge bot commented Sep 8, 2021

Mailing list message from Jaikiran Pai on core-libs-dev:

Hello Robert,

On 07/09/21 11:24 pm, Robert Scholte wrote:

On Sat, 4 Sep 2021 18:30:06 GMT, Andrey Turbanov <github.com+741251+turbanoff at openjdk.org> wrote:

Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision:

use @implNote to explain the use of the environment variable
src/java.base/share/classes/java/util/Properties.java line 924:

922: writeDateComment(bw);
923: synchronized (this) {
924: for (Map.Entry<Object, Object> e : new TreeMap<>(map).entrySet()) {
Is this sorting intentionally added? It's not clear from issue description or PR description that order of properties should be changed too.
Anyway I think copying `entrySet()` to array and then sorting should be faster, than creating a TreeMap
In case of reproducibility it should be at least ordered, i.e. keep original input order.

As discussed in the mailing list, it is agreed upon that these property
keys will be oredered when they are written out by the store() APIs.
Thus providing reproducibility. However, the order will not be the
insertion order, instead it will be the natural order of the property
keys and this order will only be applicable/maintained when using the
store() APIs. Trying to store them in a original input order will be a
much bigger change and won't just be applicable for the store() APIs but
the entire internal implementation of the Properties class itself.

-Jaikiran

@mlbridge
Copy link

mlbridge bot commented Sep 8, 2021

Mailing list message from Jaikiran Pai on core-libs-dev:

Hello Stuart,

On 08/09/21 6:49 am, Stuart Marks wrote:

On 9/7/21 8:27 AM, Jaikiran Pai wrote:

On 07/09/21 8:35 pm, Roger Riggs wrote:

Though java.util.Date is used in the current implementation, its use
is discouraged in new code
in favor of java.time.ZonedDateTime.
Consider if java.time.format.DateTimeFormatter.RFC_1123_DATE_TIME
can be used to parse and format the time.

Noted. I'll take a look at these and update the PR as necessary.

Unless there's an overriding reason, it might be nice to have the
output format match the format used in the Debian patch that adds
SOURCE_DATE_EPOCH:

https://salsa.debian.org/openjdk-team/openjdk/-/blob/master/debian/patches/reproducible-properties-timestamp.diff

So the current patch implementation uses the format "d MMM yyyy HH:mm:ss
'GMT'", with a Locale.ROOT (for locale neutral formatting). I chose this
format since that was the one that the (deprecated)
java.util.Date#toGMTString() was using.

Roger's suggestion is to use DateTimeFormatter#RFC_1123_DATE_TIME date
format which is "dow, d MMM yyyy HH:mm:ss GMT" (where dow == day of week)

IMO, either of these formats are "well known", since they are/were used
within the JDK, especially the DateTimeFormatter#RFC_1123_DATE_TIME
which Roger suggested, since that's even a public spec.

The one in the debian patch is "yyyy-MM-dd HH:mm:ss z" which although is
fine to use, it however feels a bit "less known".

I was leaning towards Roger's suggestion to use the RFC_1123_DATE_TIME
in my upcoming patch update. Is there a reason why the one in debian's
patch is preferable compared to a spec backed format?

-Jaikiran

jaikiran added 2 commits Sep 8, 2021
 - Use doPriveleged instead of explicit permission checks, to reduce complexity
 - Use DateTimeFormatter and ZonedDateTime instead of Date.toString()
 - Use DateTimeFormatter.RFC_1123_DATE_TIME for formatting SOURCE_DATE_EPOCH dates
 - Use Arrays.sort instead of a TreeMap for ordering property keys
@mlbridge
Copy link

mlbridge bot commented Sep 8, 2021

Mailing list message from Jaikiran Pai on core-libs-dev:

On 07/09/21 9:02 pm, Alan Bateman wrote:

On 07/09/2021 16:05, Roger Riggs wrote:

Hi,

The value of SOURCE_DATE_EPOCH is not so sensitive that it needs the
protections you are applying.
The doPriv only exposes the value of that specific environment
variable and in the usual case, it is undefined.

The complexity in the specification and implementation seem
unnecessary in this case.

I agree.? Given the complexity then it makes your suggestion/option to
just drop the date from the comment somewhat tempting.

-Alan

I've now updated the PR to take into account the inputs that were
provided so far. More specifically, the PR has been updated to:

?- remove the complexity around SecurityManager usage and now just uses
a doPriveleged block to get the SOURCE_EPOCH_DATE.

?- use Arrays.sort(...) instead of a TreeMap to write out the sorted
properties. This was a suggestion from Andrey and based on my JMH
testing (which I will post separately), the Arrays.sort(...) did indeed
perform better.

?- use DateTimeFormatter.RFC_1123_DATE_TIME while formatting and
writing the reproducible SOURCE_DATE_EPOCH value. There isn't a general
agreement yet on what format should be used. Stuart has suggested using
a different format (the one in the debian patch). So this part of the
change could still undergo further change.

?- use ZonedDateTime along with a DateTimeFormatter which matches the
format used by java.util.Date.toString(), instead of using a
java.util.Date() instance when writing out the current date.

The new tests that have been introduced in this PR have been adjusted to
verify these new expectations. The existing and these new tests continue
to pass with these changes.

-Jaikiran

bw.newLine();
synchronized (this) {
for (Map.Entry<Object, Object> e : entrySet()) {
var entries = map.entrySet().toArray(new Map.Entry<?, ?>[0]);
Arrays.sort(entries, new Comparator<Map.Entry<?, ?>>() {
Copy link
Member Author

@jaikiran jaikiran Sep 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part here, intentionally doesn't use a lambda, since from what I remember seeing in some mail discussion, it was suggested that using lambda in core parts which get used very early during JVM boostrap, should be avoided. If that's not a concern here, do let me know and I can change it to a lambda.

Copy link
Member

@stuart-marks stuart-marks Sep 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a fair concern, but writing out a properties file is a pretty high-level operation that doesn't seem likely to be used during bootstrap. Also, I observe that the doPrivileged block above uses a lambda. So I think we're probably ok to use a lambda here. But if you get an inexplicable error at build time or at startup time, this would be the reason why.

Assuming we're ok with lambdas, it might be easier to use collections instead of arrays in order to preserve generic types. Unfortunately the map is Map<Object, Object> so we have to do some fancy casting to get the right type. But then we can use Map.Entry.comparingByKey() as the comparator. (Note that this uses lambda internally.)

Something like this might work:

        @SuppressWarnings("unchecked")
        var entries = new ArrayList<>(((Map<String, String>)(Map)map).entrySet());
        entries.sort(Map.Entry.comparingByKey());

@mlbridge
Copy link

mlbridge bot commented Sep 8, 2021

Mailing list message from Jaikiran Pai on core-libs-dev:

Hello Andrey,

On 07/09/21 7:50 pm, Andrey Turbanov wrote:

On Sun, 5 Sep 2021 12:38:20 GMT, Jaikiran Pai <jai.forums2013 at gmail.com> wrote:

Do you mean that converting the keySet() of an
existing Map into an array and then sorting that array and then using
that sorted array to iterate and using these keys to do an additional
lookup for value against the original Map would be more efficient in
this case?
You can convert entrySet() to array. Not a keySet. In this case there is no need to lookup for values.

I experimented this in a JMH test and the results matched your
expectations. So, I've updated the PR to use array sorting instead of
creating a TreeMap. For reference, here's the JMH benchmark code and the
results:

package org.myapp;

import org.openjdk.jmh.annotations.Benchmark;
import java.util.*;
import java.util.concurrent.*;
import org.openjdk.jmh.annotations.*;

public class MyBenchmark {

??? @State(Scope.Thread)
??? public static class TestData {
??????? static final Map<Object, Object> tenItems;
??????? static final Map<Object, Object> hundredItems;
??????? static final Map<Object, Object> thousandItems;

??????? static {
??????????? tenItems = new ConcurrentHashMap<>(8);
??????????? hundredItems = new ConcurrentHashMap<>(8);
??????????? thousandItems = new ConcurrentHashMap<>(8);
??????????? for (int i = 0; i < 1000; i++) {
??????????????? thousandItems.put("foo" + i, "bar");
??????????????? if (i < 100) {
??????????????????? hundredItems.put("hello" + i, "world");
??????????????? }
??????????????? if (i < 10) {
??????????????????? tenItems.put("good" + i, "morning");
??????????????? }
??????????? }
??????????? System.out.println("Test data created with " +
tenItems.size() + ", "
??????????????? + hundredItems.size() + " and " + thousandItems.size()
+ " Map keys");
??????? }
??? }

??? @Benchmark
??? @BenchmarkMode(Mode.AverageTime)
??? @OutputTimeUnit(TimeUnit.MICROSECONDS)
??? public void testTenItemsTreeMapSorting(TestData testData) {
??????? final Map<Object, Object> sorted = new TreeMap(testData.tenItems);
??? }

??? @Benchmark
??? @BenchmarkMode(Mode.AverageTime)
??? @OutputTimeUnit(TimeUnit.MICROSECONDS)
??? public void testHundredItemsTreeMapSorting(TestData testData) {
??????? final Map<Object, Object> sorted = new
TreeMap(testData.hundredItems);
??? }

??? @Benchmark
??? @BenchmarkMode(Mode.AverageTime)
??? @OutputTimeUnit(TimeUnit.MICROSECONDS)
??? public void testThousandItemsTreeMapSorting(TestData testData) {
??????? final Map<Object, Object> sorted = new
TreeMap(testData.thousandItems);
??? }

??? @Benchmark
??? @BenchmarkMode(Mode.AverageTime)
??? @OutputTimeUnit(TimeUnit.MICROSECONDS)
??? public void testTenItemsArraySorting(TestData testData) {
??????? var entries = testData.tenItems.entrySet().toArray(new
Map.Entry<?, ?>[0]);
??????? Arrays.sort(entries, new Comparator<Map.Entry<?, ?>>() {
??????????? @Override
??????????? public int compare(Map.Entry<?, ?> o1, Map.Entry<?, ?> o2) {
??????????????? return ((String) o1.getKey()).compareTo((String)
o2.getKey());
??????????? }
??????? });
??? }

??? @Benchmark
??? @BenchmarkMode(Mode.AverageTime)
??? @OutputTimeUnit(TimeUnit.MICROSECONDS)
??? public void testHundredItemsArraySorting(TestData testData) {
??????? var entries = testData.hundredItems.entrySet().toArray(new
Map.Entry<?, ?>[0]);
??????? Arrays.sort(entries, new Comparator<Map.Entry<?, ?>>() {
??????????? @Override
??????????? public int compare(Map.Entry<?, ?> o1, Map.Entry<?, ?> o2) {
??????????????? return ((String) o1.getKey()).compareTo((String)
o2.getKey());
??????????? }
??????? });
??? }

??? @Benchmark
??? @BenchmarkMode(Mode.AverageTime)
??? @OutputTimeUnit(TimeUnit.MICROSECONDS)
??? public void testThousandItemsArraySorting(TestData testData) {
??????? var entries = testData.thousandItems.entrySet().toArray(new
Map.Entry<?, ?>[0]);
??????? Arrays.sort(entries, new Comparator<Map.Entry<?, ?>>() {
??????????? @Override
??????????? public int compare(Map.Entry<?, ?> o1, Map.Entry<?, ?> o2) {
??????????????? return ((String) o1.getKey()).compareTo((String)
o2.getKey());
??????????? }
??????? });
??? }

}

Results:

Benchmark??????????????????????????????????? Mode? Cnt??? Score Error? Units
MyBenchmark.testHundredItemsArraySorting???? avgt?? 25??? 8.330 ? 0.147?
us/op
MyBenchmark.testHundredItemsTreeMapSorting?? avgt?? 25??? 8.637 ? 0.333?
us/op
MyBenchmark.testTenItemsArraySorting???????? avgt?? 25??? 0.261 ? 0.006?
us/op
MyBenchmark.testTenItemsTreeMapSorting?????? avgt?? 25??? 0.422 ? 0.007?
us/op
MyBenchmark.testThousandItemsArraySorting??? avgt?? 25? 151.566 ? 1.660?
us/op
MyBenchmark.testThousandItemsTreeMapSorting? avgt?? 25? 163.767 ? 1.911?
us/op

-Jaikiran

@magicus
Copy link
Member

magicus commented Sep 8, 2021

Am I the only one thinking there should also be a way for developers to explicitly disable timestamps from the API?

I think the current iteration looks okay (but the core-libs guys of course has the say in this), but I still think we need a new method (or overload) to allow for a timestamp-free to be generated, always, independent of environment variables.

@mlbridge
Copy link

mlbridge bot commented Sep 8, 2021

Mailing list message from Stuart Marks on core-libs-dev:

Unless there's an overriding reason, it might be nice to have the output format
match the format used in the Debian patch that adds SOURCE_DATE_EPOCH:

https://salsa.debian.org/openjdk-team/openjdk/-/blob/master/debian/patches/reproducible-properties-timestamp.diff

So the current patch implementation uses the format "d MMM yyyy HH:mm:ss 'GMT'",
with a Locale.ROOT (for locale neutral formatting). I chose this format since that
was the one that the (deprecated) java.util.Date#toGMTString() was using.

Roger's suggestion is to use DateTimeFormatter#RFC_1123_DATE_TIME date format which
is "dow, d MMM yyyy HH:mm:ss GMT" (where dow == day of week)

IMO, either of these formats are "well known", since they are/were used within the
JDK, especially the DateTimeFormatter#RFC_1123_DATE_TIME which Roger suggested,
since that's even a public spec.

The one in the debian patch is "yyyy-MM-dd HH:mm:ss z" which although is fine to
use, it however feels a bit "less known".

I was leaning towards Roger's suggestion to use the RFC_1123_DATE_TIME in my
upcoming patch update. Is there a reason why the one in debian's patch is preferable
compared to a spec backed format?

My point in bringing this is up is to consider interoperability. I don't have a
strong preference over the particular date format. As far as I can see, there are
currently two behaviors "in the wild":

1) Baseline OpenJDK 17 behavior:

dow mon dd hh:mm:ss zzz yyyy

This is the behavior provided by "new Date().toString()" and has likely not changed
in many years. Of course, the actual values reflect the current time and locale,
which hurts reproducibility, but the format itself hasn't changed.

2) Debian's OpenJDK with SOURCE_DATE_EPOCH set:

yyyy-MM-dd HH:mm:ss z

The question is, what format should the JDK-8231640 use?

I had said earlier that it might be a good idea to match the Debian format. But
thinking about this further, I think sticking with the original JDK format would be
preferable. The Debian change is after all an outlier.

So the more specific question is, should we try to continue with the original JDK
format or choose a format that's "better" in some sense? It seems to me that if
there's something out there that parses the date from a properties file, we'd want
to avoid breaking this code if the environment variable is set. So maybe stick with
the original format in all cases. But of course for reproducibility use the epoch
value from the environment and set the locale and zone offset to known values.

s'marks

 - Implement Roger's suggestion to explicitly state that the system property should be set on command line
 - Change @implNote to @implSpec based on inputs being provided on CSR issue
 - Use Roger's and Daniel's suggestions to reword the @implSpec text to explain how overridding of entrySet() can impact the order of the written out properties
 - Improve the internal code comment (meant for maintainers) to be more clear on what kind of check we are doing before deciding the order of properties to write
@mlbridge
Copy link

mlbridge bot commented Sep 16, 2021

Mailing list message from Jaikiran Pai on core-libs-dev:

On 16/09/21 4:05 am, Roger Riggs wrote:

On Wed, 15 Sep 2021 21:46:59 GMT, Stuart Marks <smarks at openjdk.org> wrote:

src/java.base/share/classes/java/util/Properties.java line 819:

817: * <p>
818: * If the {@systemProperty java.util.Properties.storeDate} is set and
819: * is non-empty (as determined by {@link String#isEmpty() String.isEmpty}),
"is set **on the command line** and non-empty"...

Following from a comment on the CSR, it should be clear that the property value used can only be set on the command line.
This is a clever way to detect whether the `entrySet()` method has been overridden to return something other than the entry-set provided by the Properties class... but I'm wondering if it's too clever. Does anyone who overrides entrySet() care about a specific order, or do they simply sort it in order to get reproducible output? If the latter, then sorting by default hasn't really broken anything.

Also, it was never specified that the properties are written based on what's returned by a self-call to `entrySet()`. So this was never guaranteed, though we do want to avoid gratuitous breakage.

I would also wager that anybody who overrides entrySet() so that they can control the ordering of the entries is probably breaking the contract of Map::entrySet, which says that it's mutable (a mapping can be removed by removing its entry from the entry-set, or the underlying value can be changed by calling setValue on an entry). This is admittedly pretty obscure, but it tells me that trying to customize the output of Properties::store by overriding entrySet() is a pretty fragile hack.

If people really need to control the order of output, they need to iterate the properties themselves instead of overriding entrySet(). I think the only difficulty in doing so is properly escaping the keys and values, as performed by saveConvert(). If this is a use case we want to support, then maybe we should expose a suitable API for escaping properties keys and values. That would be a separate enhancement, though.

Note some history in this Stack Overflow question and answers:

https://stackoverflow.com/questions/10275862/how-to-sort-properties-in-java

Basically they mostly describe overriding `keys()`, but we broke that in Java 9, and now the advice is to override `entrySet()`. But the goal is to sort the properties, which we're doing anyway!
One part of what store() does is annoying to replicate, the encoding that `saveConvert` does is non-trivial.
Other hacks based on returning a different entrySet might be to filter the set either keep some entries or ignore them.
Both unlikely, but hacks are frequently fragile. And we've been very cautious in this discussion to avoid
breaking things, in part, because there is so little info about how it is used.

To summarize the options that we have discussed for this entrySet() part:

- Do nothing specific for subclasses that override entrySet() method.
This would mean that the store() method would write out the properties
in the natural sort order, but also has a tiny possibility that it will
break backward compatibility if some code out there that was returning a
differently ordered set. Given how we have tried to prevent backward
compatibility issues in this PR plus the fact that we might have a
possible way to prevent this, I think we can rule out this option.

- Check the Properties object instance type to see if it has been
subclassed. If yes, then don't store the properties in the natural sort
order. I personally think we should rule this option out because this
option prevents this enhancement from being usable by any subclass of
Properties even if those subclasses do nothing related to entrySet() and
could merely have been subclassed for completely unrelated things.

- Detect that the entrySet() method is overridden by the subclass of
Properties and might have done something with the returned
instance/type. It's just a co-incidence that the Properties.entrySet()
already returns a internal private class from that method. This does
allow us to introduce a check to decide whether or not to use the new
natural sort order for writing the properties. It relies on an internal
knowledge plus the internal impl detail of the Property.entrySet() but,
IMO, it might be good enough for us to use to be sure that we don't
break backward compatibility and yet at the same time, introduce this
natural sorted order by default for most of the subclasses out there.
The other aspect to consider here is the future maintainability of such
check that is being proposed here. What this check would effectively
mean is that the implementation in Property.entrySet() and this store()
method will have to be closely "held together" so that if we do change
the implementation of Property.entrySet() to return something else at a
later point, we would still have to return a type which is private to
this Properties class (or some internal marker interface type) so that
the store() methods can continue to employ a check like this one. IMO,
that shouldn't be too hard to do if/when such a change happens in
Properties.entrySet(). So I am in favour of this option to get past this
entrySet() overridding issue.

-Jaikiran

jaikiran added 3 commits Sep 17, 2021
…storeDate system property to java.properties.date

- Tests updated accordingly
…e in the list of system properties listed in System::getProperties()
dfuch
dfuch approved these changes Sep 20, 2021
@jaikiran
Copy link
Member Author

jaikiran commented Sep 21, 2021

What would be the next step to move the CSR forward? Should I be changing it's status or do something else?

@mlbridge
Copy link

mlbridge bot commented Sep 21, 2021

Mailing list message from Joseph D. Darcy on core-libs-dev:

Hello Jaikiran,

The CSR is in Draft state. As discussed in the CSR wiki
(https://wiki.openjdk.java.net/display/csr/Main), the request needs to
be moved by the assignee to either Finalized or Proposed state to
request the CSR review the request.

HTH,

-Joe

On 9/20/2021 8:46 PM, Jaikiran Pai wrote:

@mlbridge
Copy link

mlbridge bot commented Sep 21, 2021

Mailing list message from Jaikiran Pai on core-libs-dev:

Thank you Joe. That link helped. I have now moved the CSR to the next state.

-Jaikiran

On 21/09/21 9:28 am, Joseph D. Darcy wrote:

Hello Jaikiran,

The CSR is in Draft state. As discussed in the CSR wiki
(https://wiki.openjdk.java.net/display/csr/Main), the request needs to
be moved by the assignee to either Finalized or Proposed state to
request the CSR review the request.

HTH,

-Joe

On 9/20/2021 8:46 PM, Jaikiran Pai wrote:

// returned by the Properties.entrySet() implementation. If yes, then we sort those entries
// in the natural order of their key. Else, we consider that the subclassed implementation may potentially
// have returned a differently ordered entries and so we just use the iteration order of the returned instance.
if (entries instanceof Collections.SynchronizedSet<?> ss
Copy link
Contributor

@AlanBateman AlanBateman Sep 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you mind re-formatting the comment to keep the line length a bit more consistent with the rest of the code, is's just a bit annoying to have it wrapping.

Copy link
Member Author

@jaikiran jaikiran Sep 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. I've updated the PR to reduce the line length of that code comment.

@@ -761,6 +761,9 @@ public static native void arraycopy(Object src, int srcPos,
* <tr><th scope="row">{@systemProperty native.encoding}</th>
* <td>Character encoding name derived from the host environment and/or
* the user's settings. Setting this system property has no effect.</td></tr>
* <tr><th scope="row">{@systemProperty java.properties.date}</th>
* <td>Text for the comment that must replace the default date comment
* written out by {@code Properties.store()} methods <em>(optional)</em> </td></tr>
Copy link
Contributor

@AlanBateman AlanBateman Sep 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To date, the table in getProperties has listed the supported system properties that the runtime makes available to applications. It hasn't historically listed the many other standard properties that can be set on the command line. So I'm sure about adding this one to the table as it opens the door to expanding the table to all the other system properties that are documented elsewhere in the API docs. Note that javadoc creates a table of system properties from usages of @systemProperty so there is already a more complete table in the javadoc build.

Copy link
Member Author

@jaikiran jaikiran Sep 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After this change to include this property in System::getProperties() javadoc was added, there have been inputs (here and on the CSR) which suggest that we probably shouldn't include it here. I've now updated this PR to revert this specific change.

jaikiran added 2 commits Sep 22, 2021
…ties.date in the list of system properties listed in System::getProperties()"

Additional inputs since this specific commit was introduced have leaned towards not listing this property in System::getProperties()

This reverts commit 458c1fd.
@openjdk openjdk bot removed the csr Pull request needs approved CSR before integration label Sep 26, 2021
@openjdk
Copy link

openjdk bot commented Sep 27, 2021

@jaikiran This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8231640: (prop) Canonical property storage

Reviewed-by: rriggs, smarks, dfuchs, ihse

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 173 new commits pushed to the master branch:

  • b0983df: 8274074: SIGFPE with C2 compiled code with -XX:+StressGCM
  • 7436a77: 8274317: Unnecessary reentrant synchronized block in java.awt.Cursor
  • 7426fd4: 8274325: C4819 warning at vm_version_x86.cpp on Windows after JDK-8234160
  • e3aff8f: 8274289: jdk/jfr/api/consumer/TestRecordedFrameType.java failed with "RuntimeException: assertNotEquals: expected Interpreted to not equal Interpreted"
  • 252aaa9: 8274293: Build failure on macOS with Xcode 13.0 as vfork is deprecated
  • 7700b25: 8273401: Disable JarIndex support in URLClassPath
  • 5ec1cdc: 8274321: Standardize values of @SInCE tags in javax.lang.model
  • 4838a2c: 8274143: Disable "invalid entry for security.provider.X" error message in log file when security.provider.X is empty
  • ab28db1: 8274312: ProblemList 2 serviceability/dcmd/gc tests with ZGC on macos-all
  • 8c122af: 8274314: Typo in WatchService#poll(long timeout, TimeUnit unit) javadoc
  • ... and 163 more: https://git.openjdk.java.net/jdk/compare/c54a918a0e526403a395ad76c1dd0519be136ac7...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

@openjdk openjdk bot added the ready Pull request is ready to be integrated label Sep 27, 2021
@jaikiran
Copy link
Member Author

jaikiran commented Sep 27, 2021

The CSR for this has been approved and there are no pending changes in this PR. I am thinking of integrating this PR in around 24 hours from now if this looks good.

Copy link
Contributor

@RogerRiggs RogerRiggs left a comment

Looks good, thanks for the initiative, the followup on the comments, and finding the sweet spot in the goals, compatibility, and implementation constraints.

@magicus
Copy link
Member

magicus commented Sep 27, 2021

I agree with Roger. It's hard to understand the amount of work you have put into this when you only look at the small, resulting patch. Thank you for pulling this through!

@jaikiran
Copy link
Member Author

jaikiran commented Sep 28, 2021

It was a pleasure working on this one due to the timely and valuable reviews, inputs and suggestions. Thanks to everyone for helping in getting this done.

@jaikiran
Copy link
Member Author

jaikiran commented Sep 28, 2021

/integrate

@openjdk
Copy link

openjdk bot commented Sep 28, 2021

Going to push as commit af50772.
Since your change was applied there have been 185 commits pushed to the master branch:

  • ddc2627: 8273790: Potential cyclic dependencies between Gregorian and CalendarSystem
  • 633eab2: 8174819: java/nio/file/WatchService/LotsOfEvents.java fails intermittently
  • 8876eae: 8269685: Optimize HeapHprofBinWriter implementation
  • c880b87: 8274367: Re-indent stack-trace examples for Throwable.printStackTrace
  • c4b52c7: 8271303: jcmd VM.cds {static, dynamic}_dump should print more info
  • 5b660f3: 8274392: Suppress more warnings on non-serializable non-transient instance fields in java.sql.rowset
  • 0865120: 8274345: make build-test-lib is broken
  • 75404ea: 8267636: Bump minimum boot jdk to JDK 17
  • 14100d5: 8274170: Add hooks for custom makefiles to augment jtreg test execution
  • daaa47e: 8274311: Make build.tools.jigsaw.GenGraphs more configurable
  • ... and 175 more: https://git.openjdk.java.net/jdk/compare/c54a918a0e526403a395ad76c1dd0519be136ac7...master

Your commit was automatically rebased without conflicts.

@openjdk openjdk bot closed this Sep 28, 2021
@openjdk openjdk bot added integrated Pull request has been integrated and removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Sep 28, 2021
@jaikiran jaikiran deleted the 8231640 branch Sep 28, 2021
@openjdk
Copy link

openjdk bot commented Sep 28, 2021

@jaikiran Pushed as commit af50772.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core-libs core-libs-dev@openjdk.org integrated Pull request has been integrated
8 participants