Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Matrix build for Spark and Scala versions #447

Merged
merged 29 commits into from
Mar 15, 2022
Merged

Matrix build for Spark and Scala versions #447

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
5da47eb
Matrix build for Spark and Scala versions
Feb 18, 2022
2ac7249
Merge branch 'mainline' into dev/richard/matrix
richard-rogers Feb 18, 2022
1dd24fb
add spark 3.1.1
Feb 21, 2022
dd4566b
Merge branch 'dev/richard/matrix' of github.com:whylabs/whylogs into …
Feb 21, 2022
6f86696
exclude scala 2.11 & spark 3.1.1
Feb 21, 2022
442facf
publish matrix builds
Feb 21, 2022
95d228b
include Spark 3.2 & Scala 2.13
Feb 21, 2022
412eb26
need os in included matrix entry
Feb 21, 2022
6c5ee0a
experiment
Feb 22, 2022
d8d08d9
experiment
Feb 22, 2022
9a7d284
experiment failed :(
Feb 22, 2022
72f3d3b
syntax fix
Feb 22, 2022
f70954f
typo
Feb 22, 2022
d070065
Scala 2.13 will need code changes
Feb 22, 2022
dfe341b
skip problematic Spark/Scala combinations
Feb 22, 2022
b7b0153
Merge branch 'mainline' into dev/richard/matrix
richard-rogers Feb 22, 2022
166f9eb
match publish matrix to build matrix
Feb 22, 2022
24ee948
Merge branch 'dev/richard/matrix' of github.com:whylabs/whylogs into …
Feb 22, 2022
6ec998e
Merge branch 'mainline' into dev/richard/matrix
richard-rogers Feb 22, 2022
224cb2a
refer to clickup ticket
Feb 22, 2022
b112941
git Merge branch 'dev/richard/matrix' of github.com:whylabs/whylogs i…
Feb 23, 2022
b1088b0
spark/scala default versions
Mar 8, 2022
b0724c2
drop combinations that require code change
Mar 8, 2022
08360ca
Merge branch 'mainline' into dev/richard/matrix
richard-rogers Mar 8, 2022
26e027b
Merge branch 'mainline' into dev/richard/matrix
jamie256 Mar 8, 2022
e635e6b
added spark/scala versions
Mar 9, 2022
d8ceb8c
Merge branch 'dev/richard/matrix' of github.com:whylabs/whylogs into …
Mar 9, 2022
78f34f3
Merge branch 'mainline' into dev/richard/matrix
richard-rogers Mar 9, 2022
7b19ce5
Merge branch 'mainline' into dev/richard/matrix
jamie256 Mar 15, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .github/workflows/java-continuous-integration.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,27 @@ jobs:
strategy:
matrix:
os: [ubuntu-latest]
scala-version: [2.11, 2.12]
spark-version: [2.4.7, 2.4.8]
include:
- os: ubuntu-latest
scala-version: 2.11
spark-version: 2.4.5
- os: ubuntu-latest
scala-version: 2.12
spark-version: 3.0.1
- os: ubuntu-latest
scala-version: 2.12
spark-version: 3.0.3
- os: ubuntu-latest
scala-version: 2.12
spark-version: 3.1.1
- os: ubuntu-latest
scala-version: 2.12
spark-version: 3.1.2
- os: ubuntu-latest
scala-version: 2.12
spark-version: 3.1.3
runs-on: ${{ matrix.os }}
defaults:
run:
Expand Down Expand Up @@ -37,6 +58,9 @@ jobs:
with:
build-root-directory: java
arguments: build
env:
SCALA_VERSION : ${{ matrix.scala-version }}
SPARK_VERSION : ${{ matrix.spark-version }}
- name: Smoke test jar in client environment
uses: gradle/gradle-build-action@v2
with:
Expand Down
29 changes: 26 additions & 3 deletions .github/workflows/push_release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,34 @@ name: Upload Whylogs Packages

on:
release:
types: [ released, prereleased]
types: [released, prereleased]

jobs:
deploy:

strategy:
matrix:
scala-version: [2.11, 2.12]
spark-version: [2.4.7, 2.4.8]
include:
- os: ubuntu-latest
scala-version: 2.11
spark-version: 2.4.5
- os: ubuntu-latest
scala-version: 2.12
spark-version: 3.0.1
- os: ubuntu-latest
scala-version: 2.12
spark-version: 3.0.3
- os: ubuntu-latest
scala-version: 2.12
spark-version: 3.1.1
- os: ubuntu-latest
scala-version: 2.12
spark-version: 3.1.2
- os: ubuntu-latest
scala-version: 2.12
spark-version: 3.1.3
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
- name: Set up Python
Expand Down Expand Up @@ -69,6 +90,8 @@ jobs:
env:
ORG_GRADLE_PROJECT_signingKey: ${{ secrets.SIGNING_KEY_ASCII }}
ORG_GRADLE_PROJECT_signingPassword: ${{ secrets.SIGNING_KEY_PASSWORD }}
SCALA_VERSION : ${{ matrix.scala-version }}
SPARK_VERSION : ${{ matrix.spark-version }}
run: ./gradlew publish -PossrhUsername=${{ secrets.NEXUS_TOKEN_USERNAME }} -PossrhPassword=${{ secrets.NEXUS_TOKEN_PASSWORD }}
working-directory: java

Expand Down
3 changes: 1 addition & 2 deletions .github/workflows/python-continuous-integration.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,7 @@ jobs:
max-parallel: 6
matrix:
os: [ubuntu-latest, macOS-latest]
python-version: [ 3.6, 3.7, 3.8, 3.9]

python-version: [3.6, 3.7, 3.8, 3.9]
steps:
- uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
Expand Down
4 changes: 2 additions & 2 deletions java/spark-bundle/build.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ plugins {
id("com.github.johnrengelman.shadow") version ("5.2.0")
}

val scalaVersion = project.properties.getOrDefault("scalaVersion", "2.12")
val sparkVersion = project.properties.getOrDefault("sparkVersion", "3.1.1") as String
richard-rogers marked this conversation as resolved.
Show resolved Hide resolved
val scalaVersion = System.getenv("SCALA_VERSION") ?: "2.12"
val sparkVersion = System.getenv("SPARK_VERSION") ?: "3.1.1"
val artifactBaseName = "whylogs-spark-bundle_${sparkVersion}-scala_$scalaVersion"
val versionString = rootProject.version

Expand Down
6 changes: 3 additions & 3 deletions java/spark/build.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ spotless {
}
}

val scalaVersion = project.properties.getOrDefault("scalaVersion", "2.12")
val sparkVersion = project.properties.getOrDefault("sparkVersion", "3.1.1") as String
val scalaVersion = System.getenv("SCALA_VERSION") ?: "2.12"
val sparkVersion = System.getenv("SPARK_VERSION") ?: "3.1.1"
val artifactBaseName = "${rootProject.name}-spark_$sparkVersion-scala_$scalaVersion"

tasks.jar {
Expand All @@ -32,7 +32,7 @@ tasks.withType<Jar> {
}

fun scalaPackage(groupId: String, name: String, version: String) =
"$groupId:${name}_$scalaVersion:$version"
"$groupId:${name}_${scalaVersion}:${version}"

sourceSets {
main {
Expand Down