chore: Split long running integration tests and fast unit tests (DEV-…

…1537) (#2315)

Split tests into different source sets, so that long running integration tests are separated from faster unit tests.

Make sbt it run all tests from all sbt sub projects from the IntegrationTest configuration in sequence.

Make stb test run all tests from all sbt sub projects from the Test configuration in parallel.

Run integration and unit tests in parallel Github Action build jobs.

Move most tests to it source set and keep some zio-test unit tests in test source set.

Fix schema-core to run tests by adding zio-test-sbt to its dependencies.

Add documentation: README.md and internal docs.

Issue Number: DEV-1537

Co-authored-by: irinaschubert <irina.schubert@dasch.swiss>

.github/workflows/build-and-test.yml

@@ -14,7 +14,25 @@ jobs:
       - name: Run preparatory steps
         uses: dasch-swiss/dsp-api/.github/actions/preparation@main
       - name: Run all tests
-        run: make test
+        run: sbt -v coverage test coverageAggregate
+      - name: Upload coverage data to codacy
+        uses: codacy/codacy-coverage-reporter-action@v1
+        with:
+          project-token: ${{ secrets.CODACY_PROJECT_TOKEN }}
+          coverage-reports: ./target/scala-2.13/coverage-report/cobertura.xml
+      - name: Upload coverage data to codecov
+        uses: codecov/codecov-action@v3
+        with:
+          files: ./target/scala-2.13/coverage-report/cobertura.xml
+
+  integration-test:
+    name: Build and integration-test
+    runs-on: ubuntu-latest
+    steps:
+      - name: Run preparatory steps
+        uses: dasch-swiss/dsp-api/.github/actions/preparation@main
+      - name: Run all integration tests
+        run: make integration-test
       - name: Upload coverage data to codacy
         uses: codacy/codacy-coverage-reporter-action@v1
         with:

Makefile

@@ -185,7 +185,7 @@ stack-db-only: env-file  ## starts only fuseki.
 client-test-data: export KNORA_WEBAPI_COLLECT_CLIENT_TEST_DATA := true
 client-test-data: build ## runs the dsp-api e2e and r2r tests and generates client-test-data.
 	$(CURRENT_DIR)/webapi/scripts/zap-client-test-data.sh
-	sbt -v "webapi/testOnly *E2ESpec *R2RSpec"
+	sbt -v "webapi/IntegrationTest/testOnly *E2ESpec *R2RSpec"
 	$(CURRENT_DIR)/webapi/scripts/zip-client-test-data.sh
 
 .PHONY: test-repository-upgrade
@@ -201,12 +201,17 @@ test-repository-upgrade: build init-db-test-minimal ## runs DB upgrade integrati
 	# after a certain time. at startup, data should be upgraded.
 	@$(MAKE) -f $(THIS_FILE) stack-up
 
+.PHONY: test-all
+test-all: test integration-test
+
 .PHONY: test
-test: build test-shared test-user-slice test-role-slice test-project-slice ## runs all tests
-	sbt -v coverage "webapi/test"
-	sbt -v coverage "schemaCore/test"
-	sbt coverageAggregate
-
+test: ## runs all unit tests
+	sbt -v coverage test coverageAggregate
+
+.PHONY: integration-test
+integration-test: docker-build-sipi-image ## runs all integration tests
+	sbt -v coverage "IntegrationTest/test" coverageAggregate
+
 .PHONY: test-shared
 test-shared: ## tests the shared projects (build is not called from this target)
 	sbt -v coverage "shared/test"

README.md

@@ -94,10 +94,25 @@ make stack-down
 
 ### Run the automated tests
 
+Automated tests are split into different source sets into slow running integration tests (i.e. tests which do IO or are
+using [Testcontainers](https://www.testcontainers.org/)) and fast running unit tests.
+
+Run unit tests:
+
+```shell
+sbt test
+```
+
+Run integration tests:
+
+```shell
+make integration-test
+```
+
 Run all tests:
 
 ```shell
-make test
+make test-all
 ```
 
 ## Release Versioning Convention

build.sbt

@@ -2,13 +2,10 @@ import com.typesafe.sbt.SbtNativePackager.autoImport.NativePackagerHelper._
 import com.typesafe.sbt.packager.docker.DockerPlugin.autoImport.{Docker, dockerRepository}
 import com.typesafe.sbt.packager.docker.{Cmd, ExecCmd}
 import org.knora.Dependencies
-
-import sbt._
 import sbt.Keys.version
-import sbt.librarymanagement.Resolver
+import sbt._
 
 import scala.language.postfixOps
-import scala.sys.process.Process
 
 //////////////////////////////////////
 // GLOBAL SETTINGS
@@ -76,6 +73,7 @@ lazy val root: Project = Project(id = "root", file("."))
 
 addCommandAlias("fmt", "; all root/scalafmtSbt root/scalafmtAll; root/scalafixAll")
 addCommandAlias("check", "; all root/scalafmtSbtCheck root/scalafmtCheckAll; root/scalafixAll --check")
+addCommandAlias("it", "IntegrationTest/test")
 
 lazy val customScalacOptions = Seq(
   "-feature",
@@ -103,8 +101,7 @@ lazy val sipi: Project = Project(id = "sipi", base = file("sipi"))
     Docker / dockerExposedPorts ++= Seq(1024),
     Docker / defaultLinuxInstallLocation := "/sipi",
     Universal / mappings ++= {
-      // copy the sipi/scripts folder
-      directory("sipi/scripts"),
+      directory("sipi/scripts")
     },
     // use filterNot to return all items that do NOT meet the criteria
     dockerCommands := dockerCommands.value.filterNot {
@@ -132,20 +129,31 @@ run / connectInput := true
 lazy val webApiCommonSettings = Seq(
   name := "webapi"
 )
+testFrameworks := Seq(new TestFramework("zio.test.sbt.ZTestFramework"))
 
 lazy val webapi: Project = Project(id = "webapi", base = file("webapi"))
   .settings(buildSettings)
+  .settings(
+    inConfig(Test) {
+      Defaults.testSettings
+    },
+    Test / testFrameworks     := Seq(new TestFramework("zio.test.sbt.ZTestFramework")),
+    Test / fork               := true, // run tests in a forked JVM
+    Test / testForkedParallel := true, // run tests in parallel
+    Test / parallelExecution  := true, // run tests in parallel
+    libraryDependencies ++= Dependencies.webapiDependencies ++ Dependencies.webapiTestDependencies
+  )
   .enablePlugins(SbtTwirl, JavaAppPackaging, DockerPlugin, GatlingPlugin, JavaAgent, BuildInfoPlugin)
   .settings(
     name := "webapi",
     resolvers ++= Seq(
-      Resolver.bintrayRepo("hseeberger", "maven"),
-      "Sonatype" at "https://oss.sonatype.org/content/repositories/snapshots"
+      "Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots"
     ),
-    libraryDependencies ++= Dependencies.webapiLibraryDependencies
+    libraryDependencies ++= Dependencies.webapiDependencies ++ Dependencies.webapiTestDependencies ++ Dependencies.webapiIntegrationTestDependencies
   )
   .settings(
-    inConfig(Test)(Defaults.testTasks ++ baseAssemblySettings)
+    inConfig(IntegrationTest)(Defaults.itSettings ++ Defaults.testTasks ++ baseAssemblySettings),
+    libraryDependencies ++= Dependencies.webapiDependencies ++ Dependencies.webapiIntegrationTestDependencies
   )
   .settings(
     // add needed files to production jar
@@ -160,12 +168,12 @@ lazy val webapi: Project = Project(id = "webapi", base = file("webapi"))
     // use packaged jars (through packageBin) on classpaths instead of class directories for production
     Compile / exportJars := true,
     // add needed files to test jar
-    Test / packageBin / mappings ++= Seq(
+    IntegrationTest / packageBin / mappings ++= Seq(
       (rootBaseDir.value / "webapi" / "scripts" / "fuseki-repository-config.ttl.template") -> "webapi/scripts/fuseki-repository-config.ttl.template", // needed for initialization of triplestore
       (rootBaseDir.value / "sipi" / "config" / "sipi.docker-config.lua")                   -> "sipi/config/sipi.docker-config.lua"
     ),
     // use packaged jars (through packageBin) on classpaths instead of class directories for test
-    Test / exportJars := true
+    IntegrationTest / exportJars := true
   )
   .settings(
     scalacOptions ++= Seq(
@@ -178,19 +186,18 @@ lazy val webapi: Project = Project(id = "webapi", base = file("webapi"))
       "-Wconf:src=target/.*:s",
       "-Wunused:imports"
     ),
-    logLevel          := Level.Info,
-    run / javaOptions := webapiJavaRunOptions,
+    logLevel := Level.Info,
     javaAgents += Dependencies.aspectjweaver,
-    Test / fork               := true,  // run tests in a forked JVM
-    Test / testForkedParallel := false, // not run forked tests in parallel
-    Test / parallelExecution  := false, // not run non-forked tests in parallel
+    IntegrationTest / fork               := true,  // run tests in a forked JVM
+    IntegrationTest / testForkedParallel := false, // not run forked tests in parallel
+    IntegrationTest / parallelExecution  := false, // not run non-forked tests in parallel
     // Global / concurrentRestrictions += Tags.limit(Tags.Test, 1), // restrict the number of concurrently executing tests in all projects
-    Test / javaOptions ++= Seq("-Dconfig.resource=fuseki.conf") ++ webapiJavaTestOptions,
+    IntegrationTest / javaOptions += "-Dconfig.resource=fuseki.conf",
     // Test / javaOptions ++= Seq("-Dakka.log-config-on-start=on"), // prints out akka config
     // Test / javaOptions ++= Seq("-Dconfig.trace=loads"), // prints out config locations
-    Test / testOptions += Tests.Argument("-oDF"), // show full stack traces and test case durations
+    IntegrationTest / testOptions += Tests.Argument("-oDF"), // show full stack traces and test case durations
     // add test framework for running zio-tests
-    Test / testFrameworks ++= Seq(new TestFramework("zio.test.sbt.ZTestFramework"))
+    IntegrationTest / testFrameworks ++= Seq(new TestFramework("zio.test.sbt.ZTestFramework"))
   )
   .settings(
     // prepare for publishing
@@ -241,22 +248,6 @@ lazy val webapi: Project = Project(id = "webapi", base = file("webapi"))
   )
   .dependsOn(shared, schemaCore)
 
-lazy val webapiJavaRunOptions = Seq(
-  "-Xms1G",
-  "-Xmx1G",
-  "-Xss6M"
-)
-
-lazy val webapiJavaTestOptions = Seq(
-  // "-showversion",
-  "-Xms1G",
-  "-Xmx1G"
-  // "-verbose:gc",
-  // "-XX:+UseG1GC",
-  // "-XX:MaxGCPauseMillis=500",
-  // "-XX:MaxMetaspaceSize=4096m"
-)
-
 //////////////////////////////////////
 // DSP's new codebase
 //////////////////////////////////////

docs/05-internals/development/building-and-running.md

@@ -114,7 +114,7 @@ The docker plugin also allows for a number of other useful features, like inspec
 To run all test targets, use the following in the command line:
 
 ```
-$ make test
+$ make test-all
 ```
 
 To run a single test from the command line, for example `SearchV1R2RSpec`, 

docs/05-internals/development/testing.md

@@ -5,30 +5,46 @@
 
 # Testing
 
-## How to Write Unit Tests
+## How to Write and Run Unit Tests
 
-1)  Inside a test, at the beginning, add the following (change the paths
-    to the test data as needed):
+A test is not a [unit test](https://www.martinfowler.com/bliki/UnitTest.html) if:
 
-```scala
-val rdfDataObjects = List (
-       RdfDataObject(path = "test_data/responders.v1.ValuesResponderV1Spec/incunabula-data.ttl", name = "http://www.knora.org/data/incunabula")
-)
-```
-The data will be automatically loaded before any tests are executed. These tests should be stored inside
-the `src/test` folder hierarchy.
+* It talks to the database
+* It communicates across the network
+* It touches the file system
+* It can’t run at the same time as any of your other unit tests
+* You have to do special things to your environment (such as editing config files) to run it
 
-2)  Call the test from terminal:
+Unit tests live in the `src/test` folder of our sbt projects.
 
+Run all unit tests from terminal:
+
+```shell
+sbt test
 ```
-$ make test-unit
-$ make test-e2e
-```
 
-## How to Write Integration Tests
+## How to Write and Run Integration Tests
+
+[Mostly you should consider writing unit tests](https://www.youtube.com/watch?v=VDfX44fZoMc). These can be executed fast and help developers more in their daily work.
+
+You might need to create an integration test because:
 
-The only difference between Integration and Unit tests is the location
-where they are stored and the way how they are called:
+* The test needs to talk to a database
+* It requires network
+* It is slow and cannot run in parallel with other tests
+* You have to do special things to the environment in order to run it
+
+In this case create it in the `src/it`  source set of our projects.
+
+.NOTE
+_Currently only the `webapi` project supports integration tests_
+
+Run all integration tests from the terminal.
+
+```shell
+make integration-test
+```
 
-1)  Store tests inside the `src/it` folder hierarchy.
-2)  Call the tests from the terminal: `make test-it`
+.NOTE
+_The integration tests currently depend on a locally published Sipi container. That is why we need to start
+the `make` command and not `sbt it`._