[GR-10004] Make --strict-compliance default and the only supported mode.

PullRequest: mx/663

Author: dougxc

README.md

@@ -2,9 +2,13 @@
 
 `mx` is a command line based tool for managing the development of (primarily) Java code. It includes a mechanism for specifying the dependencies as well as making it simple to build, test, run, update, etc the code and built artifacts. `mx` contains support for developing code spread across multiple source repositories. `mx` is written in Python (version 2.7) and is extensible.
 
-The organizing principle of `mx` is a _suite_. A _suite_ is a directory containing one or more _projects_ and also under the control of a version control system. A suite may import one or more dependent suites. One suite is designated as the primary suite. This is normally the suite in whose directory `mx` is executed. The set of suites that are reachable from the primary suite by transitive closure of the imports relation form the set that `mx` operates on. The set of suites implicitly defines the set of projects. The action of building a suite is to compile the code in the projects and generate one or more distributions which are 'jar' files containing the compiled classes and related metadata.
+The organizing principle of `mx` is a _suite_. A _suite_ is both a directory and the container for the components of the suite.
+A suite component is either a _project_, _library_ or _distribution_. There are various flavors of each of these.
+A suite may import and depend on other suites. For an execution of mx, exactly one suite is the primary suite.
+This is either the suite in whose directory `mx` is executed or the value of the global `-p` mx option.
+The set of suites reachable from the primary suite by transitive closure of the imports relation form the set that `mx` operates on.
 
-### Running mx ###
+### Running mx
 
 `mx` can be run directly (i.e., `python2.7 mx/mx.py ...`), but is more commonly invoked via the `mx/mx` bash script (which includes a Python version check). Adding the `mx/` directory to your PATH simplifies executing `mx`. The `mx/mx.cmd` script should be used on Windows.
 
@@ -21,6 +25,55 @@ For an example of `mx` usage, see the [README][1] for the Graal project.
 Note: There is a Bash completion script for global options and commands, located in `bash_completion` directory. Install it for example by `source`ing this script in your `~/.bashrc` file. If used, a temporary file `/tmp/mx-bash-completion-<project-path-hash>` is created and used for better performance. This should be OK since the `/tmp` directory is usually cleaned on every system startup.  
 [mx-honey](https://github.com/mukel/mx-honey) provides richer completions for `zsh` users.
 
+### Suites
+
+The definition of a suite and its components is in a file named `suite.py` in the _mx metadata directory_ of the
+primary suite. This is the directory named `mx.<suite name>` in the suite's top level directory. For example,
+for the `compiler` suite, it is `mx.compiler`. The format of `suite.py` is JSON with the following extensions:
+* Python multi-line and single-quoted strings are supported
+* Python hash comments are supported
+
+### Java projects
+
+Java source code is contained in a `project`. Here's an example of two [Graal compiler projects](https://github.com/oracle/graal/blob/b95d8827609d8b28993bb4468f5daa128a614e52/compiler/mx.compiler/suite.py#L129-L147):
+```python
+"org.graalvm.compiler.serviceprovider" : {
+  "subDir" : "src",
+  "sourceDirs" : ["src"],
+  "dependencies" : ["JVMCI_SERVICES"],
+  "checkstyle" : "org.graalvm.compiler.graph",
+  "javaCompliance" : "8",
+  "workingSets" : "API,Graal",
+},
+
+"org.graalvm.compiler.serviceprovider.jdk9" : {
+  "subDir" : "src",
+  "sourceDirs" : ["src"],
+  "dependencies" : ["org.graalvm.compiler.serviceprovider"],
+  "uses" : ["org.graalvm.compiler.serviceprovider.GraalServices.JMXService"],
+  "checkstyle" : "org.graalvm.compiler.graph",
+  "javaCompliance" : "9+",
+  "multiReleaseJarVersion" : "9",
+  "workingSets" : "API,Graal",
+},
+```
+
+The `javaCompliance` attribute can be a single number (e.g. `8`), the lower bound of a range (e.g. `8+`) or a fixed range (e.g. `9..11`).
+This attribute specifies the following information:
+* The maximum Java language level used by the project. This is the lower bound in a range. It is also used at the value for the `-source`
+and `-target` javac options when compiling the project.
+* The JDKs providing any internal JDK API used by the project. A project that does not use any internal JDK API should specify an
+open range (e.g. `8+`). Otherwise, a JDK matching the exact version or range is required to compile the project.
+
+Specifying JDKs to mx is done via the `--java-home` and `--extra-java-homes` options or
+via the `JAVA_HOME` and `EXTRA_JAVA_HOMES` environment variables.
+An option has precedence over the corresponding environment variable.
+Mx comes with a `select_jdk` helper function for various shells that simplifies
+switching between different values for `JAVA_HOME` and `EXTRA_JAVA_HOMES`. The
+function definition is in a shell specific file next to `mx.py` (e.g. `select_jdk.bash`[select_jdk.bash]).
+
+The `multiReleaseJarVersion` attribute is explained in the "Versioning sources for different JDK releases" section below.
+
 ### Unit testing with Junit <a name="junit"></a>
 
 The `unittest` command supports running Junit tests in `mx` suites.
@@ -60,7 +113,46 @@ public @interface AddExports {
 }
 ```
 
-### URL rewriting ###
+### Versioning sources for different JDK releases
+
+Mx includes support for multiple versions of a Java source file that is heavily inspired
+by [multi-release jars](https://docs.oracle.com/javase/10/docs/specs/jar/jar.html#multi-release-jar-files).
+A versioned Java source has a base copy and one or more versioned copies. The signature of each
+copy (i.e., methods and fields accessed from outside the source file) must be identical.
+A versioned copy must be in a project whose Java compliance is greater than the project
+containing the base copy. For example, the base copy of [GraalServices.java](https://github.com/oracle/graal/blob/1bc9e1c762c85303d1388f98149bb9bb402f0cff/compiler/src/org.graalvm.compiler.serviceprovider/src/org/graalvm/compiler/serviceprovider/GraalServices.java)
+is in a [project with Java compliance of 8](https://github.com/oracle/graal/blob/1bc9e1c762c85303d1388f98149bb9bb402f0cff/compiler/mx.compiler/suite.py#L134).
+There is a [GraalServices.java](https://github.com/oracle/graal/blob/1bc9e1c762c85303d1388f98149bb9bb402f0cff/compiler/src/org.graalvm.compiler.serviceprovider.jdk9/src/org/graalvm/compiler/serviceprovider/GraalServices.java)
+source file that _presides over_ the base copy when running on JDK 9 or later.
+The project containing the latter [uses](https://github.com/oracle/graal/blob/1bc9e1c762c85303d1388f98149bb9bb402f0cff/compiler/mx.compiler/suite.py#L144-L145)
+the `multiReleaseJarVersion` attribute to denote that it contains classes to be packaged in
+a versioned jar directory. It also specifies the earliest Java version for which it is valid.   
+
+When these projects are combined into the same mx distribution, mx will put these class files into the distribution
+jar according to the multi-release jar format:
+```
+> jar tvf mxbuild/dists/graal.jar | grep GraalServices
+  1562 Tue May 22 22:19:02 CEST 2018 org/graalvm/compiler/serviceprovider/GraalServices$JMXService.class
+  6546 Tue May 22 22:19:02 CEST 2018 org/graalvm/compiler/serviceprovider/GraalServices.class
+  1146 Tue May 22 22:19:02 CEST 2018 META-INF/versions/9/org/graalvm/compiler/serviceprovider/GraalServices$1.class
+  1435 Tue May 22 22:19:02 CEST 2018 META-INF/versions/9/org/graalvm/compiler/serviceprovider/GraalServices$1$1.class
+   932 Tue May 22 22:19:02 CEST 2018 META-INF/versions/9/org/graalvm/compiler/serviceprovider/GraalServices$JMXService.class
+  6933 Tue May 22 22:19:02 CEST 2018 META-INF/versions/9/org/graalvm/compiler/serviceprovider/GraalServices.class
+```
+Resolving classes to the right version is done by the VM at runtime as explained in the JAR specification.
+
+When a distribution also defines a module (e.g., [GRAAL](https://github.com/oracle/graal/blob/1bc9e1c762c85303d1388f98149bb9bb402f0cff/compiler/mx.compiler/suite.py#L1662)),
+the versions are _flattened_ when creating the modular jar. That is, the modular jar contains only the class
+files that would be resolved at runtime:
+```
+> jar tvf mxbuild/modules/jdk.internal.vm.compiler.jar | grep GraalServices
+  1146 Tue May 22 22:19:12 CEST 2018 org/graalvm/compiler/serviceprovider/GraalServices$1.class
+  1435 Tue May 22 22:19:12 CEST 2018 org/graalvm/compiler/serviceprovider/GraalServices$1$1.class
+   932 Tue May 22 22:19:12 CEST 2018 org/graalvm/compiler/serviceprovider/GraalServices$JMXService.class
+  6933 Tue May 22 22:19:12 CEST 2018 org/graalvm/compiler/serviceprovider/GraalServices.class
+```
+
+### URL rewriting
 
 Mx includes support for the primary suite to be able to override the source URLs of imported suites.
 The suite level `urlrewrites` attribute allows regular expression URL rewriting. For example:
@@ -85,7 +177,7 @@ Rewrite rules can also be specified by the `MX_URLREWRITES` environment variable
 The value of this variable must either be a JSON object describing a single rewrite rule, a JSON array describing a list of rewrite rules or a file containing one of these JSON values.
 Rewrites rules specified by `MX_URLREWRITES` are applied after rules specified by the primary suite.
 
-### Environment variable processing ###
+### Environment variable processing
 
 Suites might require various environment variables to be defined for
 the suite to work and mx provides `env` files to cache this
@@ -106,7 +198,7 @@ and override any value provided by the shell environment.
 
 The `-v` option to `mx` will show the loading of `env` files during suite parsing.
 
-### Multiple suites per repository ###
+### Multiple suites per repository
 
 Sometimes it might be convenient to group multiple suites inside a single repository.
 In particular, this helps ensure that all these suites are synchronized and tested together.

ci.hocon

@@ -3,11 +3,11 @@ overlay = b1fe46376e38f11fb6f9024110dc21b389063d71
 
 # These are used in the overlay
 java7 : {name : oraclejdk, version : "7",    platformspecific: true}
-java8 : {name : labsjdk, version : "8u161-jvmci-0.42", platformspecific: true}
+java8 : {name : labsjdk, version : "8u171-jvmci-0.43", platformspecific: true}
 java9 : {name : oraclejdk, version : "9.0.4+11", platformspecific: true}
 
 downloads.java : {
-    JAVA_HOME : {name : oraclejdk, version : "8u161", platformspecific: true}
+    JAVA_HOME : {name : oraclejdk, version : "8u171", platformspecific: true}
 }
 
 gate : {
@@ -30,6 +30,7 @@ gate : {
 }
 
 bench-test : {
+  downloads : ${downloads.java}
   run: [
     ["./mx", "benchmark", "--results-file", "bench-results.json", "--ignore-suite-commit-info=mx", "test"]
   ]