Skip to content
Permalink
Browse files
Update compile script with instructions in README
  • Loading branch information
davecromberge committed Sep 7, 2021
1 parent d002196 commit cb28369210f0f5b393cc1e11659491c1bf696349
Show file tree
Hide file tree
Showing 6 changed files with 156 additions and 93 deletions.
@@ -29,7 +29,7 @@
This Memory component is general purpose, has no external runtime dependencies and can be used in any
application that needs to manage data structures inside or outside the Java heap.

The goal of this component of the DataSketches library is to provide high performance access API for accessing four different types of memory resources. Each of the four resource types is accessed using different API methods in the Memory component.
The goal of this component of the DataSketches library is to provide a high performance access API for accessing four different types of memory resources. Each of the four resource types is accessed using different API methods in the Memory component.

* **Heap:** Contiguous bytes on the Java Heap constructed by, e.g., *WritableMemory.writableWrap(byte[])* or using the *WritableMemory.allocate(int)* method. For purposes of this document this includes on-heap ByteBuffers constructed using *ByteBuffer.allocate(int)*.

@@ -47,7 +47,7 @@ If you are interested in making contributions to this Memory component please se
## Release 2.0.0+
Starting with release *datasketches-memory-2.0.0*, this Memory component supports Java 8 through Java 13. Providing access to the four contiguous byte resources (mentioned above) in Java 8 only requires reflection. However, **Java 9 introduced the Java Platform Module System (JPMS) where access to these internal classes requires starting up the JVM with special JPMS arguments.** The actual JVM arguments required will depend on how the user intends to use the Memory API, the Java version used to run the user's application and whether the user's application is a JPMS application or not.

Also see the [usage instructions](docs/usage-instructions.md) for more information.
Also see the [usage examples](docs/usage-examples.md) for more information.

## USE AS A LIBRARY (using jars from Maven Central)
In this environment, the user is using the Jars from Maven Central as a library dependency and not attempting to build the Memory component from the source code or run the Memory component tests.
@@ -106,24 +106,22 @@ __NOTES:__
2) This project is structured as a Maven multi-module project.
Building this project might affect plugins that require early dependency resolution, such as the
javadoc and eclipse plugins.
The build instructions below have been modified to use the `process-classes` phase (instead of `compile`)
The Maven build instructions below have been modified to use the `process-classes` phase (instead of `compile`)
for these use cases.

For more information, see this [Maven Reactor Issue](https://issues.apache.org/jira/browse/MNG-3283).

#### JDK versions required to compile
This DataSketches component is pure Java and requires the following JDKs to compile:
#### Dependencies
There are no run-time dependencies. See the pom.xml file for test dependencies.

#### Maven build instructions
The Maven build requires the following JDKs to compile:

- JDK8/Hotspot
- JDK9/Hotspot
- JDK11/Hotspot

Ensure that your local environment has been configured according to the
[Maven Toolchains Configuration](docs/maven-toolchains.md).

#### Recommended Build Tool
This DataSketches component is structured as a Maven project and Maven is the recommended Build
Tool.
Before building, first ensure that your local environment has been configured according to the [Maven Toolchains Configuration](docs/maven-toolchains.md).

There are two types of tests: normal unit tests and tests run by the strict profile.

@@ -159,18 +157,40 @@ This will create the following Jars:
* datasketches-memory-X.Y.Z-test-sources.jar The test source files
* datasketches-memory-X.Y.Z-javadoc.jar The compressed Javadocs.

#### Toolchains
#### Building for a specific java version

This project makes use of Maven toolchains to ensure that the correct Java compiler version is
used when compiling source files.
See the [Maven Toolchains Configuration](docs/maven-toolchains.md) for more details.
A build script named **compile-package-jar.sh** has been provided to package a JAR for a specific java version. This is necessary in cases where a developer is unable to install all of the required versions of the JDK that are required as part of the Maven build.

#### Dependencies
The build script performs the following steps:

1) Sets up staging directories under **target/** for the package files
2) Uses git commands to gather information about the current Git commit and branch
3) Compiles java source tree and assembles javadocs
4) Packages a JAR containing compiled sources together with the Manifest, Licence and Notice files
5) Packages a javadoc JAR containing documentation files together with the Manifest, Licence and Notice files
6) Checks and tests the assembled JAR by using the API to access four different resource types

The build script is located in the **tools/scripts/** directory and requires the following arguments:

* JDK Home Directory - The first argument is the absolute path of JDK home directory e.g. $JAVA_HOME
* Git Version Tag - The second argument is the Git Version Tag for this deployment e.g. 1.0.0-SNAPSHOT, 1.0.0-RC1, 1.0.0 etc.
* Project Directory - The third argument is the absolute path of project.basedir e.g. /src/apache-datasketches-memory

For example, if the project base directory is `/src/datasketches-memory`;

To run the script for a release version:

$ tools/scripts/compile-package-jar.sh $JAVA_HOME 2.0.0 /src/datasketches-memory

To run the script for a snapshot version:

$ tools/scripts/compile-package-jar.sh $JAVA_HOME 2.1.0-SNAPSHOT /src/datasketches-memory

To run the script for an RC version:

There are no run-time dependencies.
$ tools/scripts/compile-package-jar.sh $JAVA_HOME 2.0.0-RC1 /src/datasketches-memory

#### Testing
See the pom.xml file for test dependencies.
Note that the script does **not** use the _Git Version Tag_ to adjust the working copy to a remote tag - it is expected that the user has a pristine copy of the desired branch/tag available **before** using the script.

---

@@ -183,7 +203,7 @@ detail:
* [Maven Toolchains Configuration](docs/maven-toolchains.md)
* [Multi-Release Jar](docs/multi-release-jar.md)
* [Java Platform Module System](docs/module-system.md)
* [Usage instructions](docs/usage-instructions.md)
* [Usage examples](docs/usage-examples.md)

In order to build and contribute to this project, please read the relevant IDE documentation:

@@ -58,7 +58,7 @@ configuration:

---

## Running Datasketches-Memory in IntelljJ-IDEA
## Running Datasketches-Memory in IntelliJ-IDEA

Note that the following configuration was verified using IntelliJ IDEA 2021.1.2
(Community Edition).
@@ -27,7 +27,7 @@ system for the Java Platform. For more documentation on the implementation, see

> Reliable configuration, to replace the brittle, error-prone class-path mechanism with a means
for program components
> to declare explicit dependences upon one another;
> to declare explicit dependencies upon one another;
This prevents ClassLoader errors such as `NoClassDefFoundError` that typically occur at runtime
and make applications less reliable.
@@ -72,7 +72,7 @@ Some dependencies are encapsulated by default, and this causes compilation to fa
Java versions 9 and above.
These dependencies can be made accessible at compile time through the use of the
`add-exports` compiler argument.
This argument allows one module to access some of the unexported types of another module.
This argument allows one module to access some un-exported types of another module.
Datasketches Memory depends on several internal APIs and therefore requires special
exposition.

@@ -96,7 +96,7 @@ requires that the user add additional JVM run-time arguments (`add-opens...`, wh
Note that if the user has allocated off-heap memory using ByteBuffer.allocateDirect(...),
the DataSketches memory component can still read and write to this memory without these `add-opens...` arguments.

See the main [README](../README.md) and the [usage examples](usage-examples.md) for more details.
See the use **Use as a Library** and **Developer Usage** sections in the main [README](../README.md) for more details. In addition, examples are provided in the [usage examples](usage-examples.md) document.

### JPMS and Java 8

@@ -20,6 +20,8 @@
package org.apache.datasketches.memory.tools.scripts;

import java.io.File;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;

import org.apache.datasketches.memory.MapHandle;
import org.apache.datasketches.memory.Memory;
@@ -28,6 +30,14 @@

public class CheckMemoryJar {

public void printJDK() {
String JdkVersionString = System.getProperty("java.version");
int JdkMajorVersion = getJavaMajorVersion(JdkVersionString);
println("JDK Full Version : " + JdkVersionString);
println("JDK Major Version: " + JdkMajorVersion);
println("");
}

public void checkHeapWritableMemory() {
try {
String str = "1 - Heap WritableMemory Successful";
@@ -50,9 +60,21 @@ public void checkAllocateDirect() throws Exception {
}
}

public void checkByteBuffer() throws Exception {
try {
String str = "3 - Map ByteBuffer Successful";
ByteBuffer bb = ByteBuffer.allocateDirect(2 * str.length());
bb.order(ByteOrder.nativeOrder());
WritableMemory wmem = WritableMemory.writableWrap(bb);
writeReadAndPrintString(wmem, str);
} catch (Exception ex) {
exitOnError("Map ByteBuffer", ex);
}
}

public void checkMap(String mappedFilePath) throws Exception {
try {
String str = "3 - Memory Map Successful";
String str = "4 - Memory Map Successful";
File file = new File(mappedFilePath);
MapHandle mh = Memory.map(file);
Memory mem = mh.get();
@@ -63,6 +85,23 @@ public void checkMap(String mappedFilePath) throws Exception {
}
}

public static void main(final String[] args) throws Exception {
if (args.length < 1) {
System.out.println("Please provide the full path to the memory mapped file!");
System.exit(1);
}

String mappedFilePath = args[0];
CheckMemoryJar check = new CheckMemoryJar();
check.printJDK();
check.checkHeapWritableMemory();
check.checkAllocateDirect();
check.checkByteBuffer();
check.checkMap(mappedFilePath);
println("");
println("All checks passed.");
}

/**********************/

private static void writeReadAndPrintString(WritableMemory wmem, String str) {
@@ -80,20 +119,28 @@ private static void exitOnError(String checkName, Exception ex){
System.exit(1);
}

public static void main(final String[] args) throws Exception {
if (args.length < 1) {
System.out.println("Please provide the full path to the memory mapped file!");
System.exit(1);
}
private static int getJavaMajorVersion(final String jdkVersion) {
int[] verArr = parseJavaVersion(jdkVersion);
return (verArr[0] == 1) ? verArr[1] : verArr[0];
}

String mappedFilePath = args[0];
CheckMemoryJar check = new CheckMemoryJar();
check.checkHeapWritableMemory();
check.checkAllocateDirect();
check.checkMap(mappedFilePath);
println("");
println("All checks passed.");
/**
* Returns first two number groups of the java version string.
* @param jdkVersion the java version string from System.getProperty("java.version").
* @return first two number groups of the java version string.
*/
private static int[] parseJavaVersion(final String jdkVersion) {
final int p0, p1;
try {
String[] parts = jdkVersion.trim().split("[^0-9\\.]");//grab only number groups and "."
parts = parts[0].split("\\."); //split out the number groups
p0 = Integer.parseInt(parts[0]); //the first number group
p1 = (parts.length > 1) ? Integer.parseInt(parts[1]) : 0; //2nd number group, or 0
} catch (final NumberFormatException | ArrayIndexOutOfBoundsException e) {
throw new IllegalArgumentException("Improper Java -version string: " + jdkVersion + "\n" + e);
}
return new int[] {p0, p1};
}

static void println(Object obj) { System.out.println(obj.toString()); }
private static void println(Object obj) { System.out.println(obj.toString()); }
}
@@ -0,0 +1,3 @@
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Maecenas id fermentum sem. Donec malesuada tristique erat vitae molestie. In urna eros, scelerisque ut finibus a, commodo et felis. Sed lacinia lacus ex, sit amet imperdiet tortor rutrum dictum. Nunc turpis dolor, placerat non condimentum quis, suscipit ut libero. Morbi suscipit porta nibh, sit amet bibendum erat posuere maximus. Pellentesque egestas magna sed purus scelerisque aliquam. Vestibulum placerat odio risus, nec semper sapien imperdiet eu.

Suspendisse vitae elit urna. Donec orci dolor, ullamcorper ut felis eget, porta facilisis ante. Vivamus odio quam, porta sed sapien quis, sodales tincidunt ligula. Donec porttitor.

0 comments on commit cb28369

Please sign in to comment.