Skip to content
Permalink
Browse files
Merge pull request #153 from apache/UpdateDocs
Update docs
  • Loading branch information
davecromberge committed Sep 8, 2021
2 parents d002196 + b0fd160 commit 139a2583281c875367e835ffd5cec93590df73d3
Showing 9 changed files with 197 additions and 145 deletions.
@@ -122,6 +122,12 @@ jobs:
-Dmatrix.jdk.version=${{matrix.jdk}}
--toolchains .github/workflows/.toolchains.xml
# The GitTag for CI purposes is irrelevant
- name: Custom build script
run: |
./tools/scripts/compile-package-jar.sh $JAVA_HOME x.y.z .
shell: bash

- name: Test & Report
if: ${{ matrix.jdk == 8 && success() }}
run: >
@@ -29,7 +29,7 @@
This Memory component is general purpose, has no external runtime dependencies and can be used in any
application that needs to manage data structures inside or outside the Java heap.

The goal of this component of the DataSketches library is to provide high performance access API for accessing four different types of memory resources. Each of the four resource types is accessed using different API methods in the Memory component.
The goal of this component of the DataSketches library is to provide a high performance access API for accessing four different types of memory resources. Each of the four resource types is accessed using different API methods in the Memory component.

* **Heap:** Contiguous bytes on the Java Heap constructed by, e.g., *WritableMemory.writableWrap(byte[])* or using the *WritableMemory.allocate(int)* method. For purposes of this document this includes on-heap ByteBuffers constructed using *ByteBuffer.allocate(int)*.

@@ -47,7 +47,7 @@ If you are interested in making contributions to this Memory component please se
## Release 2.0.0+
Starting with release *datasketches-memory-2.0.0*, this Memory component supports Java 8 through Java 13. Providing access to the four contiguous byte resources (mentioned above) in Java 8 only requires reflection. However, **Java 9 introduced the Java Platform Module System (JPMS) where access to these internal classes requires starting up the JVM with special JPMS arguments.** The actual JVM arguments required will depend on how the user intends to use the Memory API, the Java version used to run the user's application and whether the user's application is a JPMS application or not.

Also see the [usage instructions](docs/usage-instructions.md) for more information.
Also see the [usage examples](docs/usage-examples.md) for more information.

## USE AS A LIBRARY (using jars from Maven Central)
In this environment, the user is using the Jars from Maven Central as a library dependency and not attempting to build the Memory component from the source code or run the Memory component tests.
@@ -106,24 +106,22 @@ __NOTES:__
2) This project is structured as a Maven multi-module project.
Building this project might affect plugins that require early dependency resolution, such as the
javadoc and eclipse plugins.
The build instructions below have been modified to use the `process-classes` phase (instead of `compile`)
The Maven build instructions below have been modified to use the `process-classes` phase (instead of `compile`)
for these use cases.

For more information, see this [Maven Reactor Issue](https://issues.apache.org/jira/browse/MNG-3283).

#### JDK versions required to compile
This DataSketches component is pure Java and requires the following JDKs to compile:
#### Dependencies
There are no run-time dependencies. See the pom.xml file for test dependencies.

#### Maven build instructions
The Maven build requires the following JDKs to compile:

- JDK8/Hotspot
- JDK9/Hotspot
- JDK11/Hotspot

Ensure that your local environment has been configured according to the
[Maven Toolchains Configuration](docs/maven-toolchains.md).

#### Recommended Build Tool
This DataSketches component is structured as a Maven project and Maven is the recommended Build
Tool.
Before building, first ensure that your local environment has been configured according to the [Maven Toolchains Configuration](docs/maven-toolchains.md).

There are two types of tests: normal unit tests and tests run by the strict profile.

@@ -159,18 +157,39 @@ This will create the following Jars:
* datasketches-memory-X.Y.Z-test-sources.jar The test source files
* datasketches-memory-X.Y.Z-javadoc.jar The compressed Javadocs.

#### Toolchains
#### Building for a specific java version

This project makes use of Maven toolchains to ensure that the correct Java compiler version is
used when compiling source files.
See the [Maven Toolchains Configuration](docs/maven-toolchains.md) for more details.
A build script named **compile-package-jar.sh** has been provided to package a JAR for a specific java version. This is necessary in cases where a developer is unable to install all of the required versions of the JDK that are required as part of the Maven build.

#### Dependencies
The build script performs the following steps:

1. Sets up staging directories under **target/** for the package files
2. Uses git commands to gather information about the current Git commit and branch
3. Compiles java source tree
4. Packages a JAR containing compiled sources together with the Manifest, Licence and Notice files
5. Checks and tests the assembled JAR by using the API to access four different resource types

The build script is located in the **tools/scripts/** directory and requires the following arguments:

* JDK Home Directory - The first argument is the absolute path of JDK home directory e.g. $JAVA_HOME
* Git Version Tag - The second argument is the Git Version Tag for this deployment e.g. 1.0.0-SNAPSHOT, 1.0.0-RC1, 1.0.0 etc.
* Project Directory - The third argument is the absolute path of project.basedir e.g. /src/apache-datasketches-memory

For example, if the project base directory is `/src/datasketches-memory`;

To run the script for a release version:

$ tools/scripts/compile-package-jar.sh $JAVA_HOME 2.0.0 /src/datasketches-memory

To run the script for a snapshot version:

$ tools/scripts/compile-package-jar.sh $JAVA_HOME 2.1.0-SNAPSHOT /src/datasketches-memory

To run the script for an RC version:

There are no run-time dependencies.
$ tools/scripts/compile-package-jar.sh $JAVA_HOME 2.0.0-RC1 /src/datasketches-memory

#### Testing
See the pom.xml file for test dependencies.
Note that the script does **not** use the _Git Version Tag_ to adjust the working copy to a remote tag - it is expected that the user has a pristine copy of the desired branch/tag available **before** using the script.

---

@@ -183,7 +202,7 @@ detail:
* [Maven Toolchains Configuration](docs/maven-toolchains.md)
* [Multi-Release Jar](docs/multi-release-jar.md)
* [Java Platform Module System](docs/module-system.md)
* [Usage instructions](docs/usage-instructions.md)
* [Usage examples](docs/usage-examples.md)

In order to build and contribute to this project, please read the relevant IDE documentation:

@@ -58,7 +58,7 @@ configuration:

---

## Running Datasketches-Memory in IntelljJ-IDEA
## Running Datasketches-Memory in IntelliJ-IDEA

Note that the following configuration was verified using IntelliJ IDEA 2021.1.2
(Community Edition).
@@ -27,7 +27,7 @@ system for the Java Platform. For more documentation on the implementation, see

> Reliable configuration, to replace the brittle, error-prone class-path mechanism with a means
for program components
> to declare explicit dependences upon one another;
> to declare explicit dependencies upon one another;
This prevents ClassLoader errors such as `NoClassDefFoundError` that typically occur at runtime
and make applications less reliable.
@@ -72,7 +72,7 @@ Some dependencies are encapsulated by default, and this causes compilation to fa
Java versions 9 and above.
These dependencies can be made accessible at compile time through the use of the
`add-exports` compiler argument.
This argument allows one module to access some of the unexported types of another module.
This argument allows one module to access some un-exported types of another module.
Datasketches Memory depends on several internal APIs and therefore requires special
exposition.

@@ -96,7 +96,7 @@ requires that the user add additional JVM run-time arguments (`add-opens...`, wh
Note that if the user has allocated off-heap memory using ByteBuffer.allocateDirect(...),
the DataSketches memory component can still read and write to this memory without these `add-opens...` arguments.

See the main [README](../README.md) and the [usage examples](usage-examples.md) for more details.
See the use **Use as a Library** and **Developer Usage** sections in the main [README](../README.md) for more details. In addition, examples are provided in the [usage examples](usage-examples.md) document.

### JPMS and Java 8

@@ -20,7 +20,8 @@
# Usage Examples

You may need to supply additional runtime arguments to the JVM depending on how you are using the Datasketches Memory library.
There are several applicable use cases that are considered:
For more information regarding required JPMS arguments and when they are applicable, see the [README](../README.md).
This document provides examples for the following scenarios:

1. Using the library from a Java 8 application
2. Using the library with on-heap memory only
@@ -34,10 +35,11 @@ No additional runtime arguments are required.
As an example, consider the following launch script that compiles and runs a simple Java 8 application:

```shell
export JAVA_HOME=$JAVA8_HOME
export JAVAC=$JAVA_HOME/bin/javac
export JAR=$JAVA_HOME/bin/jar
export JAVA=$JAVA_HOME/bin/java
JH=$JAVA8_HOME
JAVAC=$JH/bin/javac
JAVA=$JH/bin/java
JAR=$JH/bin/jar
patha=nomvn-jdk8
cd $patha
@@ -46,54 +48,52 @@ As an example, consider the following launch script that compiles and runs a sim
echo "--- CLEAN & COMPILE ---"
rm -rf target
mkdir target
mkdir target/classes
mkdir target/test-classes
mkdir -p target/test-classes
$JAVAC -d target/test-classes/ -cp "libs/*" $(find . -name '*.java')
$JAVAC\
-d target/test-classes/\
-cp libs/*\
$(find . -name '*.java')
echo "---- RUN ----"
echo PWD:$(pwd)
$JAVA\
-cp target/test-classes:libs/*:src/test/resources/\
org.xyz.memory.CheckJava8
-cp libs/*:target/test-classes:src/test/resources/\
org.xyz.memory.RunMain
```

### 2) Using the library with on-heap memory only, or with off-heap memory allocated via ByteBuffer by the user.
### 2) Using the library with on-heap memory only

No additional runtime arguments are required, regardless of whether the library is used from a Java 8 or Java 9+
application.
Similarly, no additional runtime arguments are required in this scenario - regardless of whether the library is used from a Java 8 or Java 9+ application.

As an example, consider the following launch script that compiles and runs a simple Java 9 application that only exclusively
uses on-heap memory:

```shell
export JAVA_HOME=$JAVA9_HOME
export JAVAC=$JAVA_HOME/bin/javac
export JAR=$JAVA_HOME/bin/jar
export JAVA=$JAVA_HOME/bin/java
JH=$JAVA9_HOME
JAVAC=$JH/bin/javac
JAVA=$JH/bin/java
JAR=$JH/bin/jar
patha=nomvn-nomod-heap-jdk9
cd $patha
echo PWD:$(pwd)
echo $JAVA_HOME
echo "--- CLEAN & COMPILE ---"
rm -rf target
mkdir target
mkdir target/classes
mkdir target/test-classes
mkdir -p target/test-classes
$JAVAC -d target/test-classes -cp "mods/*":"libs/*" -p mods $(find . -name '*.java')
$JAVAC\
-d target/test-classes/\
-cp "mods/*":"libs/*"
-p mods
$(find . -name '*.java')
echo "---- RUN ----"
echo PWD:$(pwd)
$JAVA\
-cp target/test-classes:"mods/*":"libs/*":src/test/resources\
org.xyz.memory.CheckJava9plus
org.xyz.memory.RunMain
```

### 3) Using off-heap memory in a non-modularized Java 9+ application
@@ -116,40 +116,38 @@ which runs as an UNNAMED module in a non-JPMS (non-modularized) application.
The following launch script compiles and runs a non-modularized Java 9 application:

```shell
JH=$JAVA9_HOME
JAVAC=$JH/bin/javac
JAVA=$JH/bin/java
JAR=$JH/bin/jar
export JAVA_HOME=$JAVA9_HOME
export JAVAC=$JAVA_HOME/bin/javac
export JAR=$JAVA_HOME/bin/jar
export JAVA=$JAVA_HOME/bin/java
patha=nomvn-nomod-jdk9
cd $patha
echo PWD:$(pwd)
echo $JAVA_HOME
echo "--- CLEAN & COMPILE ---"
rm -rf target
mkdir target
mkdir target/classes
mkdir target/test-classes
mkdir -p target/test-classes
$JAVAC -d target/test-classes -cp "mods/*":"libs/*" -p mods $(find . -name '*.java')
$JAVAC\
-d target/test-classes/\
-cp "mods/*":"libs/*"
-p mods
$(find . -name '*.java')
echo "---- RUN ----"
echo PWD:$(pwd)
$JAVA\
--add-exports java.base/jdk.internal.misc=ALL-UNNAMED\
--add-exports java.base/jdk.internal.ref=ALL-UNNAMED\
--add-opens java.base/java.nio=ALL-UNNAMED\
--add-opens java.base/sun.nio.ch=ALL-UNNAMED\
-cp target/test-classes:"mods/*":"libs/*":src/test/resources\
org.xyz.memory.CheckJava9plus
org.xyz.memory.RunMain
```
where the traditional classpath (`-cp`) argument contains all modules, libraries and resources.

Note: `mods` is a local directory containing external modules, and `libs` is a localy directory for external library
Note: `mods` is a local directory containing external modules, and `libs` is a local directory for external library
dependencies. No distinction is made between modules and libraries since they are both appended to the classpath.

### 4) Using off-heap memory in a modularized Java 9+ application
@@ -171,10 +169,11 @@ These arguments expose encapsulated packages in the `java.base` package to the `
The following launch script compiles and runs a modularized Java 9 application:

```shell
export JAVA_HOME=$JAVA9_HOME
export JAVAC=$JAVA_HOME/bin/javac
export JAR=$JAVA_HOME/bin/jar
export JAVA=$JAVA_HOME/bin/java
JH=$JAVA9_HOME
JAVAC=$JH/bin/javac
JAVA=$JH/bin/java
JAR=$JH/bin/jar
patha=nomvn-mod-jdk9
cd $patha
@@ -183,11 +182,13 @@ The following launch script compiles and runs a modularized Java 9 application:
echo "--- CLEAN & COMPILE ---"
rm -rf target
mkdir target
mkdir target/classes
mkdir target/test-classes
mkdir -p target/test-classes
$JAVAC -d target/test-classes -cp "libs/*" -p mods $(find . -name '*.java')
$JAVAC\
-d target/test-classes/\
-cp "mods/*":"libs/*"
-p mods
$(find . -name '*.java')
echo "---- RUN ----"
echo PWD:$(pwd)
@@ -197,7 +198,7 @@ The following launch script compiles and runs a modularized Java 9 application:
--add-opens java.base/sun.nio.ch=org.apache.datasketches.memory\
-cp "/libs/*":src/test/resources\
-p target/test-classes:mods\
-m org.xyz.memory/org.xyz.memory.CheckJava9plus
-m org.xyz.memory/org.xyz.memory.RunMain
```
where the traditional classpath (`-cp`) argument contains libraries and resources, and the module-path argument (`-p`)
references all external modules and compiled classes for the current user application, which is itself a module.
@@ -324,6 +324,7 @@ under the License.
<exclude>**/test/resources/**/*.txt</exclude>
<exclude>**/img/**/*.png</exclude>
<exclude>**/git.properties</exclude>
<exclude>**/scripts/LoremIpsum.txt</exclude>
<exclude>LICENSE</exclude>
<exclude>NOTICE</exclude>
</excludes>

0 comments on commit 139a258

Please sign in to comment.