Skip to content
Permalink
Browse files
refactor(cacheservice): use ZIO (DEV-546) (#2022)
* refactor(cache): remove service from name

* refactor(cacheservice): inmem (ongoing)

* refactor(cacheservice): inmem (ongoing)

* chore(docs): add structurizr (c4)

* refactor(cache): inmem get user

* refactor(cacheservice): in-mem version

* refactor(cacheservice): redis version (ongoing)

* refactor(cacheservice): add zio-config (ongoing)

* refactor(cacheservice): redis implementation (ongoing)

* refactor(cacheservice): redis implementation (ongoing)

* refactor(cacheservice): redis implementation (ongoing)

* refactor(cacheservice): add redis implementation

* refactor(cacheservice): add redis serialization implementation (ongoing)

* refactor: after spartan session (ongoing)

* refactor(cacheservice): remove actor

* refactor(cacheservice): fix tests (ongoing)

* build: fix after merge

* refactor(cacheservice): fix tests

* refactor(cacheservice): add Redis Testcontainers layer

* refactor(cacheservice): moving runtime initialization to main

* refactor(cacheservice): moving runtime initialisation back where it was

* docs: add ADR record

* chore(deps): bump zio libraries

* chore(deps): bump zio libraries

* chore(deps): bump zio libraries

* chore: remove unneeded file

* refactor(cacheservice): ziofy redis testcontainers

* cleanup

* fix redis test container

* fix redis test container

* remove metrics logging

* Update webapi/src/main/scala/org/knora/webapi/store/cacheservice/CacheServiceManager.scala

Co-authored-by: irinaschubert <irina.schubert@dasch.swiss>

* Update webapi/src/main/scala/org/knora/webapi/store/cacheservice/CacheServiceManager.scala

Co-authored-by: irinaschubert <irina.schubert@dasch.swiss>

* Update webapi/src/main/scala/org/knora/webapi/store/cacheservice/CacheServiceManager.scala

Co-authored-by: irinaschubert <irina.schubert@dasch.swiss>

* Update docs/architecture/decisions/0002-change-cache-service-manager-from-akka-actor-to-zlayer.md

Co-authored-by: irinaschubert <irina.schubert@dasch.swiss>

* cleanup

* bumping dependencies

* cleanup

Co-authored-by: irinaschubert <irina.schubert@dasch.swiss>
  • Loading branch information
subotic and irinaschubert committed Apr 12, 2022
1 parent 5c27eef commit 521150fbf3dac5ce4c075e6b3ef3c537004a750d
Showing with 2,095 additions and 1,816 deletions.
  1. +1 −0 .adr-dir
  2. +3 −0 .gitignore
  3. +1 −1 .scalafmt.conf
  4. +5 −0 Makefile
  5. +17 −11 build.sbt
  6. +15 −0 docs/architecture/README.md
  7. +19 −0 docs/architecture/decisions/0001-record-architecture-decisions.md
  8. +19 −0 docs/architecture/decisions/0002-change-cache-service-manager-from-akka-actor-to-zlayer.md
  9. +21 −0 docs/architecture/workspace.dsl
  10. +2 −2 dsp-schema/repo/src/main/scala/dsp/schema/repo/SchemaRepoLive.scala
  11. +4 −4 dsp-schema/repo/src/main/scala/dsp/schema/repo/SchemaRepoTest.scala
  12. +70 −55 project/Dependencies.scala
  13. +63 −27 webapi/src/main/scala/org/knora/webapi/app/ApplicationActor.scala
  14. +36 −50 webapi/src/main/scala/org/knora/webapi/app/Main.scala
  15. +15 −0 webapi/src/main/scala/org/knora/webapi/config/AppConfig.scala
  16. +28 −0 webapi/src/main/scala/org/knora/webapi/core/Logging.scala
  17. +1 −1 webapi/src/main/scala/org/knora/webapi/exceptions/Exceptions.scala
  18. +6 −8 ...c/main/scala/org/knora/webapi/messages/admin/responder/projectsmessages/ProjectsMessagesADM.scala
  19. +16 −16 webapi/src/main/scala/org/knora/webapi/messages/admin/responder/usersmessages/UsersMessagesADM.scala
  20. +1 −6 ...pi/src/main/scala/org/knora/webapi/messages/store/cacheservicemessages/CacheServiceMessages.scala
  21. +2 −2 webapi/src/main/scala/org/knora/webapi/responders/admin/ProjectsResponderADM.scala
  22. +3 −3 webapi/src/main/scala/org/knora/webapi/responders/admin/StoresResponderADM.scala
  23. +680 −678 webapi/src/main/scala/org/knora/webapi/responders/admin/UsersResponderADM.scala
  24. +3 −3 webapi/src/main/scala/org/knora/webapi/settings/package.scala
  25. +6 −11 webapi/src/main/scala/org/knora/webapi/store/StoreManager.scala
  26. +0 −27 webapi/src/main/scala/org/knora/webapi/store/cacheservice/CacheService.scala
  27. +81 −69 webapi/src/main/scala/org/knora/webapi/store/cacheservice/CacheServiceManager.scala
  28. +47 −0 webapi/src/main/scala/org/knora/webapi/store/cacheservice/api/CacheService.scala
  29. +1 −1 webapi/src/main/scala/org/knora/webapi/store/cacheservice/{ → api}/CacheServiceExceptions.scala
  30. +7 −0 webapi/src/main/scala/org/knora/webapi/store/cacheservice/config/CacheServiceConfig.scala
  31. +220 −0 webapi/src/main/scala/org/knora/webapi/store/cacheservice/impl/CacheServiceInMemImpl.scala
  32. +293 −0 webapi/src/main/scala/org/knora/webapi/store/cacheservice/impl/CacheServiceRedisImpl.scala
  33. +0 −175 webapi/src/main/scala/org/knora/webapi/store/cacheservice/inmem/CacheServiceInMemImpl.scala
  34. +0 −397 webapi/src/main/scala/org/knora/webapi/store/cacheservice/redis/CacheServiceRedisImpl.scala
  35. +23 −32 webapi/src/main/scala/org/knora/webapi/store/cacheservice/serialization/CacheSerialization.scala
  36. +57 −12 webapi/src/main/scala/org/knora/webapi/util/ActorUtil.scala
  37. +3 −3 webapi/src/test/scala/org/knora/webapi/CoreSpec.scala
  38. +9 −23 webapi/src/test/scala/org/knora/webapi/IntegrationSpec.scala
  39. +12 −2 webapi/src/test/scala/org/knora/webapi/ManagersWithMockedSipi.scala
  40. +5 −5 webapi/src/test/scala/org/knora/webapi/TestContainerRedis.scala
  41. +2 −2 webapi/src/test/scala/org/knora/webapi/responders/IriLockerSpec.scala
  42. +8 −5 webapi/src/test/scala/org/knora/webapi/responders/admin/UsersResponderADMSpec.scala
  43. +5 −3 webapi/src/test/scala/org/knora/webapi/store/MockableStoreManager.scala
  44. +8 −10 webapi/src/test/scala/org/knora/webapi/store/cacheservice/CacheServiceManagerSpec.scala
  45. +16 −0 webapi/src/test/scala/org/knora/webapi/store/cacheservice/config/RedisTestConfig.scala
  46. +116 −0 webapi/src/test/scala/org/knora/webapi/store/cacheservice/impl/CacheInMemImplSpec.scala
  47. +92 −0 webapi/src/test/scala/org/knora/webapi/store/cacheservice/impl/CacheRedisImplSpec.scala
  48. +0 −69 webapi/src/test/scala/org/knora/webapi/store/cacheservice/inmem/CacheServiceInMemImplSpec.scala
  49. +0 −76 webapi/src/test/scala/org/knora/webapi/store/cacheservice/redis/CacheServiceRedisImplSpec.scala
  50. +20 −27 webapi/src/test/scala/org/knora/webapi/store/cacheservice/serialization/CacheSerializationSpec.scala
  51. +33 −0 webapi/src/test/scala/org/knora/webapi/testcontainers/RedisTestContainer.scala
@@ -0,0 +1 @@
docs/architecture/decisions
@@ -18,6 +18,8 @@ webapi-it
webapi-test
upgrade/project
sipi/test
/docs/architecture/.structurizr
/docs/architecture/workspace.json

**/target/
*.aux
@@ -31,6 +33,7 @@ sipi/test
*.tdo
*.toc
*.bak
*.rdb
.sbtrc

dependencies.txt
@@ -1,6 +1,6 @@
version = "2.7.5"
maxColumn = 120
align.preset = some
align.preset = most
align.multiline = false
continuationIndent.defnSite = 2
assumeStandardLibraryStripMargin = true
@@ -33,6 +33,11 @@ docs-install-requirements: ## install requirements
docs-clean: ## cleans the project directory
@rm -rf site/

.PHONY: structurizer
structurizer: ## starts the structurizer and serves c4 architecture docs
@docker pull structurizr/lite
@docker run -it --rm -p 8080:8080 -v $(CURRENT_DIR)/docs/architecture:/usr/local/structurizr structurizr/lite

#################################
# Docker targets
#################################
@@ -100,7 +100,7 @@ lazy val webapi: Project = Project(id = "webapi", base = file("webapi"))
.settings(buildSettings)
.enablePlugins(SbtTwirl, JavaAppPackaging, DockerPlugin, GatlingPlugin, JavaAgent, RevolverPlugin, BuildInfoPlugin)
.settings(
webApiCommonSettings,
name := "webapi",
resolvers ++= Seq(
Resolver.bintrayRepo("hseeberger", "maven")
),
@@ -112,19 +112,19 @@ lazy val webapi: Project = Project(id = "webapi", base = file("webapi"))
.settings(
// add needed files to production jar
Compile / packageBin / mappings ++= Seq(
(rootBaseDir.value / "knora-ontologies" / "knora-admin.ttl") -> "knora-ontologies/knora-admin.ttl",
(rootBaseDir.value / "knora-ontologies" / "knora-base.ttl") -> "knora-ontologies/knora-base.ttl",
(rootBaseDir.value / "knora-ontologies" / "salsah-gui.ttl") -> "knora-ontologies/salsah-gui.ttl",
(rootBaseDir.value / "knora-ontologies" / "standoff-data.ttl") -> "knora-ontologies/standoff-data.ttl",
(rootBaseDir.value / "knora-ontologies" / "standoff-onto.ttl") -> "knora-ontologies/standoff-onto.ttl",
(rootBaseDir.value / "knora-ontologies" / "knora-admin.ttl") -> "knora-ontologies/knora-admin.ttl",
(rootBaseDir.value / "knora-ontologies" / "knora-base.ttl") -> "knora-ontologies/knora-base.ttl",
(rootBaseDir.value / "knora-ontologies" / "salsah-gui.ttl") -> "knora-ontologies/salsah-gui.ttl",
(rootBaseDir.value / "knora-ontologies" / "standoff-data.ttl") -> "knora-ontologies/standoff-data.ttl",
(rootBaseDir.value / "knora-ontologies" / "standoff-onto.ttl") -> "knora-ontologies/standoff-onto.ttl",
(rootBaseDir.value / "webapi" / "scripts" / "fuseki-repository-config.ttl.template") -> "webapi/scripts/fuseki-repository-config.ttl.template" // needed for initialization of triplestore
),
// use packaged jars (through packageBin) on classpaths instead of class directories for production
Compile / exportJars := true,
// add needed files to test jar
Test / packageBin / mappings ++= Seq(
(rootBaseDir.value / "webapi" / "scripts" / "fuseki-repository-config.ttl.template") -> "webapi/scripts/fuseki-repository-config.ttl.template", // needed for initialization of triplestore
(rootBaseDir.value / "sipi" / "config" / "sipi.docker-config.lua") -> "sipi/config/sipi.docker-config.lua"
(rootBaseDir.value / "sipi" / "config" / "sipi.docker-config.lua") -> "sipi/config/sipi.docker-config.lua"
),
// use packaged jars (through packageBin) on classpaths instead of class directories for test
Test / exportJars := true
@@ -140,12 +140,14 @@ lazy val webapi: Project = Project(id = "webapi", base = file("webapi"))
javaAgents += Dependencies.aspectjweaver,
fork := true, // run tests in a forked JVM
Test / testForkedParallel := false, // run forked tests in parallel
Test / parallelExecution := false, // run non-forked tests in parallel
Test / parallelExecution := false, // run non-forked tests in parallel
// Global / concurrentRestrictions += Tags.limit(Tags.Test, 1), // restrict the number of concurrently executing tests in all projects
Test / javaOptions ++= Seq("-Dconfig.resource=fuseki.conf") ++ webapiJavaTestOptions,
// Test / javaOptions ++= Seq("-Dakka.log-config-on-start=on"), // prints out akka config
// Test / javaOptions ++= Seq("-Dconfig.trace=loads"), // prints out config locations
Test / testOptions += Tests.Argument("-oDF") // show full stack traces and test case durations
Test / testOptions += Tests.Argument("-oDF"), // show full stack traces and test case durations
// add test framework for running zio-tests
Test / testFrameworks ++= Seq(new TestFramework("zio.test.sbt.ZTestFramework"))
)
.settings(
// prepare for publishing
@@ -189,8 +191,8 @@ lazy val webapi: Project = Project(id = "webapi", base = file("webapi"))
name,
version,
"akkaHttp" -> Dependencies.akkaHttpVersion,
"sipi" -> Dependencies.sipiImage,
"fuseki" -> Dependencies.fusekiImage
"sipi" -> Dependencies.sipiImage,
"fuseki" -> Dependencies.fusekiImage
),
buildInfoPackage := "org.knora.webapi.http.version"
)
@@ -219,6 +221,10 @@ lazy val webapiJavaTestOptions = Seq(
//"-XX:MaxMetaspaceSize=4096m"
)

//////////////////////////////////////
// DSP's new codebase
//////////////////////////////////////

lazy val apiMain = project
.in(file("dsp-api-main"))
.settings(
@@ -0,0 +1,15 @@
# C4 Model and ADRs

## Installation

```bash
$ brew install adr-tools
```

## Usage

Run the following command from the root directory to start the C4 model browser:

```bash
$ make structurizer
```
@@ -0,0 +1,19 @@
# 1. Record architecture decisions

Date: 2022-03-14

## Status

Accepted

## Context

We need to record the architectural decisions made on this project.

## Decision

We will use Architecture Decision Records, as [described by Michael Nygard](http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions).

## Consequences

See Michael Nygard's article, linked above. For a lightweight ADR toolset, see Nat Pryce's [adr-tools](https://github.com/npryce/adr-tools).
@@ -0,0 +1,19 @@
# 2. Change Cache Service Manager from Akka-Actor to ZLayer

Date: 2022-04-06

## Status

Accepted

## Context

The `org.knora.webapi.store.cacheservice.CacheServiceManager` was implemented as an `Akka-Actor`.

## Decision

As part of the move from `Akka` to `ZIO`, it was decided that the `CacheServiceManager` and the whole implementation of the in-memory and Redis backed cache is refactored using ZIO.

## Consequences

The usage from other actors stays the same. The actor messages and responses don't change.
@@ -0,0 +1,21 @@
workspace {

model {
user = person "User"
softwareSystem = softwareSystem "Software System"

user -> softwareSystem "Uses"
}

views {
systemContext softwareSystem "Diagram1" {
include *
autoLayout
}

theme default
}

!adrs decisions

}
@@ -14,6 +14,6 @@ case class SchemaRepoLive() extends SchemaRepo {

}

object SchemaRepoLive extends (() => SchemaRepo) {
val layer: URLayer[Any, SchemaRepo] = (SchemaRepoLive.apply _).toLayer
object SchemaRepoLive {
val layer: URLayer[Any, SchemaRepo] = ZLayer.succeed(SchemaRepoLive())
}
@@ -7,18 +7,18 @@ case class SchemaRepoTest() extends SchemaRepo {
private var map: Map[UserID, UserProfile] = Map()

def setTestData(map0: Map[UserID, UserProfile]): Task[Unit] =
Task { map = map0 }
Task.succeed { map = map0 }

def getTestData: Task[Map[UserID, UserProfile]] =
Task(map)
Task.succeed(map)

def lookup(id: UserID): Task[UserProfile] =
Task(map(id))
Task.succeed(map(id))

def update(id: UserID, profile: UserProfile): Task[Unit] =
Task.attempt { map = map + (id -> profile) }
}

object SchemaRepoTest extends (() => SchemaRepo) {
val layer: URLayer[Any, SchemaRepo] = (SchemaRepoTest.apply _).toLayer
val layer: URLayer[Any, SchemaRepo] = ZLayer.succeed(SchemaRepoTest())
}

0 comments on commit 521150f

Please sign in to comment.