Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add CD build pipeline #1

Merged
merged 44 commits into from
Jan 12, 2019
Merged

Add CD build pipeline #1

Show file tree
Hide file tree
Changes from 40 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
2313ce1
prepare POM for publishing to maven central
Jan 7, 2019
9391b8d
add sonatype snapshots
Jan 8, 2019
9bbe315
add maven settings containing env variable key names, and codesigning…
Jan 8, 2019
9b186d3
add sonatype staging plugin
Jan 8, 2019
6e8511f
fix typo
Jan 8, 2019
22e401c
remove need for apache commons-codec and hadoop writable
Jan 8, 2019
8f7f76b
dont inhert from liveramp commons pom
Jan 8, 2019
3bb5dc1
add travis deploy instructions
Jan 8, 2019
740f0da
fix typo
Jan 8, 2019
de375a1
add necessary plugins and profiles for release
Jan 8, 2019
30a81ed
add WIP deploy script for importing secrets and deploying artifacts
Jan 8, 2019
b5471c9
make cd/deploy.sh executable
Jan 8, 2019
f6b003e
use absolute paths in deploy script
Jan 8, 2019
6892f6d
bump test error bound to 2.5%
Jan 8, 2019
85e1ad2
fix linting errors
Jan 8, 2019
7c4b24b
disable failing over lint
Jan 8, 2019
37dec58
skip tests when deploying since we already run tests during the build…
Jan 8, 2019
614e53a
fix linting issues
Jan 8, 2019
448da29
fix lint
Jan 8, 2019
0053466
add escaped secret
Jan 8, 2019
2b1e4da
remove empty <p> tags
Jan 8, 2019
50344c3
remove doclint disable profile and encrypted travis key
Jan 8, 2019
efc4a46
verbose maven output
Jan 8, 2019
2dc564d
remove whitespace
Jan 8, 2019
726aaa6
fix snapshot repository IDs
Jan 8, 2019
6a7daa9
Make publishing part of main build script.
Jan 8, 2019
31f8af0
don't print debug output when deploying
Jan 8, 2019
2041204
add comments for readability
Jan 8, 2019
1dd6844
move test dir to default maven dir
Jan 8, 2019
e4157db
package instead of install during build
Jan 8, 2019
22acde0
use env gpg cert instead of encrypted cert file
Jan 8, 2019
3abe674
remove Writable reference from README
Jan 8, 2019
49b4a12
formatting
Jan 9, 2019
d68ab58
formatting
Jan 9, 2019
00c9ffc
move tests to correct dir
Jan 9, 2019
d6eb021
formatting
Jan 9, 2019
e31ca89
add comments explaining use of shading plugin
Jan 9, 2019
3d12c2c
add echo statements for debugging
Jan 9, 2019
67704d0
delete encrypted cert
Jan 9, 2019
8c3160b
explicitly specify mvn settings dir
Jan 9, 2019
6e6c99d
Merge branch 'master' of github.com:LiveRamp/HyperMinHash-java into p…
Jan 11, 2019
34ea4e9
fix conflict
Jan 11, 2019
349a20b
fix tests
Jan 11, 2019
61ae497
only upload to sonatype if we're on master
Jan 11, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,13 @@
language: java
jdk: oraclejdk8
jdk: oraclejdk8

# Since Maven installs deps, skip the install phase since it doesn't really apply.
# This just runs the command "true" in the install phase which doesn't do much of anything.
install: true

script:
# First install any deps and runs tests
- mvn clean package test -P-build-src-and-docs -B -V
# If successful, run the deploy script to publish produced artifacts
- ./cd/deploy.sh

2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ A Java implementation of the HyperMinHash algorithm, presented in [Yu and Weber]

This library uses [Loglog-Beta](https://arxiv.org/pdf/1612.02284.pdf) for the underlying LogLog implementation. Loglog-beta is almost identical in accuracy to HyperLogLog++, except it performs better on cardinality estimations for small datasets (n <= 200k). Since we use Loglog-Beta, we refer to our implementation as BetaMinHash.

In addition to the features described above, this library adds the ability to do many-way intersections between sets, a new feature not described in the original paper (though, credit to the authors, easy to deduce from their examples). We also provide an implementation of the Hadoop Writable interface for easy use with MapReduce.
In addition to the features described above, this library adds the ability to do many-way intersections between sets, a new feature not described in the original paper (though, credit to the authors, easy to deduce from their examples).

## Demo Usage

Expand Down
31 changes: 31 additions & 0 deletions cd/deploy.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
#! /usr/bin/env bash

# TODO comment this back in
#if [ "$TRAVIS_BRANCH" = 'master' ] && [ "$TRAVIS_PULL_REQUEST" == 'false' ]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment this back in?

CERT_DIR="${TRAVIS_BUILD_DIR}/cd/codesigning.asc"
echo "Set certificate directory to ${CERT_DIR}"

# This block unencrypts GPG signing keys and imports them into the build environment so they
# can be used to sign artifacts produced by this build.

# If any of these commands fail, exit so we get quick feedback from the build.
set -e

# The cert is stored as base64 to prevent interpretation issues in bash, so decode it and store
# it to the cert directory.
echo "${GPG_CERT_BASE64}" | base64 --decode -i - > "${CERT_DIR}"
echo "Saved GPG cert to file"

# Remove the unencrypted cert file inside a trap. This way, we'll always remove the cert even if
# if the script fails.
trap "rm ${CERT_DIR}; echo 'Removed cert file'; " EXIT

# import these certs into GPG
gpg --fast-import "${CERT_DIR}"
echo "Imported certs into GPG"

# Run the deploy phase (which will sign any artifacts). The build will use the GPG certs we
# imported above
mvn deploy -P sign,build-src-and-docs -DskipTests=true --settings "${TRAVIS_BUILD_DIR}/cd/mvnsettings.xml"
#fi TODO comment back in

29 changes: 29 additions & 0 deletions cd/mvnsettings.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
https://maven.apache.org/xsd/settings-1.0.0.xsd">
<servers>
<server>
<!-- In order for these credentials to be used for the desired repository/server, the ID
specified here must match the ID of the repositories specified in the distributionManagement
or repositories tags. -->
<id>ossrh</id>
<username>${env.OSSRH_JIRA_USERNAME}</username>
<password>${env.OSSRH_JIRA_PASSWORD}</password>
</server>
</servers>

<profiles>
<profile>
<id>ossrh</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<gpg.executable>gpg</gpg.executable>
<gpg.keyname>${env.GPG_KEY_NAME}</gpg.keyname>
<gpg.passphrase>${env.GPG_PASSPHRASE}</gpg.passphrase>
</properties>
</profile>
</profiles>
</settings>
244 changes: 195 additions & 49 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,34 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://maven.apache.org/POM/4.0.0"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>com.liveramp</groupId>
<artifactId>hyperminhash</artifactId>
<version>1.0-SNAPSHOT</version>
<version>0.1-SNAPSHOT</version>
<packaging>jar</packaging>

<parent>
<groupId>com.liveramp</groupId>
<artifactId>pom-common</artifactId>
<version>1.1-SNAPSHOT</version>
</parent>
<name>${project.groupId}:${project.artifactId}</name>
<description>
HyperMinHash is a probabilistic data structure that can approximate union, intersection, and set
cardinalities as well as Jaccard indices of very large sets with high accuracy, in loglog space,
and in a streaming fashion.
</description>
<url>www.github.com/LiveRamp/HyperMinHash-java</url>

<organization>
<name>LiveRamp</name>
<url>www.liveramp.com</url>
</organization>

<licenses>
<license>
<name>Apache 2</name>
<url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
<distribution>repo</distribution>
</license>
</licenses>

<scm>
<connection>scm:git:git@github.com:LiveRamp/HyperMinHash-java.git</connection>
Expand All @@ -19,72 +37,200 @@
<tag>HEAD</tag>
</scm>

<issueManagement>
<system>Github</system>
<url>www.github.com/LiveRamp/HyperMinHash-java</url>
</issueManagement>

<developers>
<developer>
<name>Christian Hansen</name>
<email>christianhshansen@gmail.com</email>
<organization>LiveRamp</organization>
<organizationUrl>www.liveramp.com</organizationUrl>
</developer>
<developer>
<name>Harry Rackmil</name>
<email>harryrackmil@gmail.com</email>
<organization>LiveRamp</organization>
<organizationUrl>www.liveramp.com</organizationUrl>
</developer>
<developer>
<name>Shrif Nada</name>
<email>snadalive@gmail.com</email>
<organization>LiveRamp</organization>
<organizationUrl>www.liveramp.com</organizationUrl>
</developer>
</developers>

<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<skip.tests>false</skip.tests>
<hadoop.version>2.5.0</hadoop.version>
</properties>

<build>
<plugins>
<!-- Used to pin the compiler to JDK 1.8 -->
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>

<!--
Used to publish artifacts produced by this project to Sonatype (which is mirrored to
Maven Central).
-->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
<groupId>org.sonatype.plugins</groupId>
<artifactId>nexus-staging-maven-plugin</artifactId>
<version>1.6.8</version>
<extensions>true</extensions>
<configuration>
<finalName>uber-${artifactId}-${version}</finalName>
<serverId>ossrh</serverId>
<nexusUrl>https://oss.sonatype.org/</nexusUrl>
<autoReleaseAfterClose>true</autoReleaseAfterClose>
</configuration>
</plugin>
</plugins>
</build>

<profiles>
<!-- Used to sign artifacts with our GPG key, as required by Sona Type. -->
<profile>
<id>sign</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-gpg-plugin</artifactId>
<version>1.6</version>
<executions>
<execution>
<id>sign-artifacts</id>
<phase>verify</phase>
<goals>
<goal>sign</goal>
</goals>
<!-- If testing releases with hardcoded values in your settings.xml file, uncomment
this setting to make GPG read those hardcoded literals -->
<!--<configuration>-->
<!--<gpgArguments>-->
<!--<arg>&#45;&#45;pinentry-mode</arg>-->
<!--<arg>loopback</arg>-->
<!--</gpgArguments>-->
<!--</configuration>-->
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>

<!-- Used to generate a stand-alone JAR which includes all dependencies. Note that even though
we're using the shading plugin, we're not actually renaming any dependencies. We're only
using the shading plugin's capability to create an uber jar. -->
<profile>
<id>uberjar</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we just uber-jarring here, or is there actual shading needed? Can we have comment explaining this?

</goals>
</execution>
</executions>
<configuration>
<finalName>uber-${artifactId}-${version}</finalName>
</configuration>
</plugin>
</plugins>
</build>
</profile>

<!-- Used to package the Javadocs and sourcecode for this project -->
<profile>
<id>build-src-and-docs</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>

<build>

<plugins>
<!-- Used to build a JAR containing the source code from this project -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>2.4</version>
<executions>
<execution>
<id>attach-sources</id>
<goals>
<goal>jar-no-fork</goal>
</goals>
</execution>
</executions>
</plugin>

<!-- Used to generate javadocs for this project -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.10.4</version>
<executions>
<execution>
<id>attach-javadocs</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>

</build>
</profile>
</profiles>

<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
</dependency>
<!-- Use metrohash for convenient & fast 128 bit hashing functions -->
<dependency>
<groupId>com.adgear</groupId>
<artifactId>metrohash</artifactId>
<version>1.0.0</version>
</dependency>

<!-- Use Junit even though our code has no bugs ;) -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>

</dependencies>

<repositories>
<repository>
<id>conjars.org</id>
<name>Conjars</name>
<url>http://conjars.org/repo</url>
</repository>
<distributionManagement>

<snapshotRepository>
<id>ossrh</id>
<url>https://oss.sonatype.org/content/repositories/snapshots</url>
</snapshotRepository>

<repository>
<id>liveramp-repositories</id>
<name>Liveramp Repositories</name>
<url>http://repository.liveramp.com/artifactory/liveramp-repositories</url>
<snapshots>
<enabled>true</enabled>
<updatePolicy>always</updatePolicy>
</snapshots>
<id>ossrh</id>
<url>https://oss.sonatype.org/service/local/staging/deploy/maven2/</url>
</repository>
</repositories>

<pluginRepositories>
<pluginRepository>
<id>sonatype-releases</id>
<url>http://oss.sonatype.org/content/repositories/releases</url>
</pluginRepository>
<pluginRepository>
<id>clojars.org</id>
<url>http://clojars.org/repo</url>
</pluginRepository>
</pluginRepositories>

</distributionManagement>
</project>
Loading