Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

use circe as serde lib for json: 200 #214

Open
wants to merge 14 commits into
base: master
Choose a base branch
from
Open

use circe as serde lib for json: 200 #214

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
f5e6795
Addid circe dependency and using it
TebaleloS Jun 14, 2024
8e8e746
merge master
TebaleloS Jun 14, 2024
ac908df
modifying dependencies
TebaleloS Jun 17, 2024
1dee384
merge master and use circe syntax in function classes
TebaleloS Jun 18, 2024
852d4fd
using circe syntax in flow classes
TebaleloS Jun 18, 2024
37b5434
using circe in api http module
TebaleloS Jun 18, 2024
eb77ba5
fix implicit import in ServerOptions
TebaleloS Jun 18, 2024
f6d2ece
merge master, fix dependencies and conclude server implicit implement…
TebaleloS Jun 21, 2024
3c707e6
removing play Json dependencies, and PlayJsonImplicits class
TebaleloS Jun 24, 2024
935cde0
removing play Json dependencies, and PlayJsonImplicits class
TebaleloS Jun 24, 2024
3c2ca00
fixing bugs and using circe instead of Json4s dependency
TebaleloS Jun 25, 2024
0f2b220
revised arrayPutUsing string implicit conversion
TebaleloS Jun 26, 2024
de28a5a
merge master
TebaleloS Jun 27, 2024
ad59999
removed json4s and replaced its usage
TebaleloS Jun 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions agent/src/main/scala/za/co/absa/atum/agent/dispatcher/HttpDispatcher.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import sttp.model.Uri
import za.co.absa.atum.agent.exception.AtumAgentException.HttpException
import za.co.absa.atum.model.dto.{AdditionalDataSubmitDTO, AtumContextDTO, CheckpointDTO, PartitioningSubmitDTO}
import za.co.absa.atum.model.utils.SerializationUtils
import io.circe.generic.auto._

class HttpDispatcher(config: Config) extends Dispatcher(config: Config) with Logging {
import HttpDispatcher._
Expand Down
82 changes: 46 additions & 36 deletions model/src/main/scala/za/co/absa/atum/model/utils/SerializationUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,70 +16,80 @@

package za.co.absa.atum.model.utils

import org.json4s.JsonAST.JString
import org.json4s.jackson.Serialization
import org.json4s.jackson.Serialization.{write, writePretty}
import org.json4s.{CustomSerializer, Formats, JNull, NoTypeHints, ext}
import za.co.absa.atum.model.dto.MeasureResultDTO.ResultValueType
import za.co.absa.atum.model.dto.MeasureResultDTO.ResultValueType._

import io.circe.{Decoder, Encoder}
import io.circe.syntax._
import io.circe.parser._
import java.time.ZonedDateTime
import java.time.format.DateTimeFormatter
import java.util.UUID

object SerializationUtils {
Copy link
Collaborator

@salamonpavel salamonpavel Jun 28, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The whole object SerializationUtils should be removed (or kept only for third party types serde implicits). It's customary to define serde implicits in companion objects - for default de/serialization. If needed one could then provide alternative implementation(s) either in direct scope where the de/serialization is performed or create yet another object for non-default implementations inside the companion object and in import into direct scope only the desired implementation (the preffered approach in my opinion).

A simple example would be:

import io.circe.{Decoder, Encoder, HCursor, Json}
import io.circe.generic.semiauto.{deriveDecoder, deriveEncoder}

case class Person(name: String, age: Int)

object Person {
  implicit val defaultPersonEncoder: Encoder[Person] = deriveEncoder[Person]
  implicit val defaultPersonDecoder: Decoder[Person] = deriveDecoder[Person]

  object JsonImplicits {
    implicit val nonDefaultPersonEncoder: Encoder[Person] = new Encoder[Person] {
      final def apply(a: Person): Json = Json.obj(
        ("name", Json.fromString(a.name)),
        ("isAdult", Json.fromBoolean(a.age >= 18))
      )
    }

    implicit val nonDefaultPersonDecoder: Decoder[Person] = new Decoder[Person] {
      final def apply(c: HCursor): Decoder.Result[Person] =
        for {
          name <- c.downField("name").as[String]
          isAdult <- c.downField("isAdult").as[Boolean]
        } yield {
          val age = if (isAdult) 18 else 17
          Person(name, age)
        }
    }
  }
}

Usage would look like this:

import io.circe.syntax._
import io.circe.parser._

val person = Person("John Doe", 20)

// Using default serde from companion (always available in direct scope without the need to import it)
val defaultJson = person.asJson // implicitly looked up from companion Person.defaultPersonEncoder

// Using non-default serde, passed implicitly
import Person.JsonImplicits.nonDefaultPersonEncoder
val nonDefaultJson = person.asJson

// Using non-default serde, passed explicitly
val nonDefaultJson = person.asJson(Person.JsonImplicits.nonDefaultPersonEncoder)


implicit private val formatsJson: Formats =
Serialization.formats(NoTypeHints).withBigDecimal +
ext.UUIDSerializer +
ZonedDateTimeSerializer +
ResultValueTypeSerializer

// TODO "yyyy-MM-dd'T'hh:mm:ss.SSS'Z'" OR TODO "yyyy-MM-dd HH:mm:ss.SSSSSSX"
val timestampFormat: DateTimeFormatter = DateTimeFormatter.ISO_ZONED_DATE_TIME

implicit val encodeZonedDateTime: Encoder[ZonedDateTime] = Encoder.encodeString.contramap[ZonedDateTime](_.format(timestampFormat))
implicit val decodeZonedDateTime: Decoder[ZonedDateTime] = Decoder.decodeString.emap { str =>
Right(ZonedDateTime.parse(str, timestampFormat))
}

implicit val encodeUUID: Encoder[UUID] = Encoder.encodeString.contramap[UUID](_.toString)
implicit val decodeUUID: Decoder[UUID] = Decoder.decodeString.emap { str =>
Right(UUID.fromString(str))
}

/**
* The method returns arbitrary object as a Json string.
*
* @return A string representing the object in Json format
*/
def asJson[T <: AnyRef](obj: T): String = {
write[T](obj)
def asJson[T: Encoder](obj: T): String = {
obj.asJson.noSpaces
}

/**
* The method returns arbitrary object as a pretty Json string.
*
* @return A string representing the object in Json format
*/
def asJsonPretty[T <: AnyRef](obj: T): String = {
writePretty[T](obj)
def asJsonPretty[T: Encoder](obj: T): String = {
obj.asJson.spaces2
}

/**
* The method returns arbitrary object parsed from Json string.
*
* @return An object deserialized from the Json string
*/
def fromJson[T <: AnyRef](jsonStr: String)(implicit m: Manifest[T]): T = {
Serialization.read[T](jsonStr)
def fromJson[T: Decoder](jsonStr: String): T = {
decode[T](jsonStr) match {
case Right(value) => value
case Left(error) => throw new RuntimeException(s"Failed to decode JSON: $error")
}
}

private case object ResultValueTypeSerializer extends CustomSerializer[ResultValueType](format => (
{
case JString(resultValType) => resultValType match {
case "String" => String
case "Long" => Long
case "BigDecimal" => BigDecimal
case "Double" => Double
}
case JNull => null
},
{
case resultValType: ResultValueType => resultValType match {
case String => JString("String")
case Long => JString("Long")
case BigDecimal => JString("BigDecimal")
case Double => JString("Double")
}
}))
sealed trait ResultValueType
object ResultValueType {
case object String extends ResultValueType
case object Long extends ResultValueType
case object BigDecimal extends ResultValueType
case object Double extends ResultValueType

implicit val encodeResultValueType: Encoder[ResultValueType] = Encoder.encodeString.contramap {
case ResultValueType.String => "String"
case ResultValueType.Long => "Long"
case ResultValueType.BigDecimal => "BigDecimal"
case ResultValueType.Double => "Double"
}

implicit val decodeResultValueType: Decoder[ResultValueType] = Decoder.decodeString.emap {
case "String" => Right(ResultValueType.String)
case "Long" => Right(ResultValueType.Long)
case "BigDecimal" => Right(ResultValueType.BigDecimal)
case "Double" => Right(ResultValueType.Double)
case other => Left(s"Cannot decode $other as ResultValueType")
}
}

}
24 changes: 13 additions & 11 deletions model/src/main/scala/za/co/absa/atum/model/utils/ZonedDateTimeSerializer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,17 +16,19 @@

package za.co.absa.atum.model.utils

import org.json4s.JsonAST.JString
import org.json4s.{CustomSerializer, JNull}

import io.circe.{Decoder, Encoder}
import java.time.ZonedDateTime
import java.time.format.DateTimeFormatter

object ZonedDateTimeSerializer {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not used anywhere anymore and could be removed.

implicit val encodeZonedDateTime: Encoder[ZonedDateTime] = Encoder.encodeString.contramap(
_.format(DateTimeFormatter.ISO_ZONED_DATE_TIME))

case object ZonedDateTimeSerializer extends CustomSerializer[ZonedDateTime](_ => (
{
case JString(s) => ZonedDateTime.parse(s, SerializationUtils.timestampFormat)
case JNull => null
},
{
case d: ZonedDateTime => JString(SerializationUtils.timestampFormat.format(d))
implicit val decodeZonedDateTime: Decoder[ZonedDateTime] = Decoder.decodeString.emap { str =>
try {
Right(ZonedDateTime.parse(str, DateTimeFormatter.ISO_ZONED_DATE_TIME))
} catch {
case e: Throwable => Left(e.getMessage)
}
}
))
}
5 changes: 3 additions & 2 deletions model/src/test/scala/za/co/absa/atum/model/utils/SerializationUtilsUnitTests.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@

package za.co.absa.atum.model.utils

import io.circe.generic.auto._
import org.scalatest.flatspec.AnyFlatSpecLike
import za.co.absa.atum.model.dto.MeasureResultDTO.{ResultValueType, TypedValue}
import za.co.absa.atum.model.dto._
Expand All @@ -41,7 +42,7 @@ class SerializationUtilsUnitTests extends AnyFlatSpecLike {
val expectedAdditionalDataJson =
"""
|{"partitioning":[{"key":"key","value":"val"}],
|"additionalData":{"key1":"val1","key2":"val2"},
|"additionalData":{"key1":"val1","key2":"val2","key3":null},
|"author":"testAuthor"}
|""".linearize
val actualAdditionalDataJson = SerializationUtils.asJson(additionalDataDTO)
Expand Down Expand Up @@ -343,7 +344,7 @@ class SerializationUtilsUnitTests extends AnyFlatSpecLike {
authorIfNew = "authorTest"
)

val expectedPartitioningDTOJson = """{"partitioning":[{"key":"key","value":"val"}],"authorIfNew":"authorTest"}"""
val expectedPartitioningDTOJson = """{"partitioning":[{"key":"key","value":"val"}],"parentPartitioning":null,"authorIfNew":"authorTest"}"""
val actualPartitioningDTOJson = SerializationUtils.asJson(partitioningDTO)

assert(actualPartitioningDTOJson == expectedPartitioningDTOJson)
Expand Down
57 changes: 21 additions & 36 deletions project/Dependencies.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ object Dependencies {
val balta = "0.1.0"

val jacksonModuleScala = "2.14.2"
val circeVersion = "0.14.5"

val specs2 = "4.10.0"
val typesafeConfig = "1.4.2"
Expand All @@ -47,22 +46,6 @@ object Dependencies {

val fadb = "0.3.0"

val json4s_spark2 = "3.5.3"
val json4s_spark3 = "3.7.0-M11"
def json4s(scalaVersion: Version): String = {
// TODO done this impractical way until https://github.com/AbsaOSS/commons/issues/134
val maj2 = Component("2")
val min11 = Component("11")
val min12 = Component("12")
val min13 = Component("13")
scalaVersion.components match {
case Seq(`maj2`, `min11`, _) => json4s_spark2
case Seq(`maj2`, `min12`, _) => json4s_spark3
case Seq(`maj2`, `min13`, _) => json4s_spark3
case _ => throw new IllegalArgumentException("Only Scala 2.11, 2.12, and 2.13 are currently supported.")
}
}

val logback = "1.2.3"

val zio = "2.0.19"
Expand All @@ -74,8 +57,8 @@ object Dependencies {
val tapir = "1.9.6"
val http4sBlazeBackend = "0.23.15"
val http4sPrometheus = "0.23.6"
val playJson = "3.0.1"
val sttpPlayJson = "3.9.3"
val circeJson = "0.14.7"
val sttpCirceJson = "3.9.7"

val awssdk = "2.23.15"

Expand Down Expand Up @@ -107,26 +90,19 @@ object Dependencies {
}

private def jsonSerdeDependencies(scalaVersion: Version): Seq[ModuleID] = {
val json4sVersion = Versions.json4s(scalaVersion)

lazy val jacksonModuleScala = "com.fasterxml.jackson.module" %% "jackson-module-scala" % Versions.jacksonModuleScala
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we still need this?


lazy val json4sExt = "org.json4s" %% "json4s-ext" % json4sVersion
lazy val json4sCore = "org.json4s" %% "json4s-core" % json4sVersion
lazy val json4sJackson = "org.json4s" %% "json4s-jackson" % json4sVersion
lazy val json4sNative = "org.json4s" %% "json4s-native" % json4sVersion % Provided

lazy val circeCore = "io.circe" %% "circe-core" % Versions.circeVersion
lazy val circeParser = "io.circe" %% "circe-parser" % Versions.circeVersion
// Circe dependencies
lazy val circeCore = "io.circe" %% "circe-core" % Versions.circeJson
lazy val circeParser = "io.circe" %% "circe-parser" % Versions.circeJson
lazy val circeGeneric = "io.circe" %% "circe-generic" % Versions.circeJson

Seq(
jacksonModuleScala,
json4sExt,
json4sCore,
json4sJackson,
json4sNative,
circeCore,
circeParser,
circeGeneric,
)
}

Expand Down Expand Up @@ -163,9 +139,17 @@ object Dependencies {
lazy val tapirPrometheus = tapirOrg %% "tapir-prometheus-metrics" % Versions.tapir
lazy val tapirStubServer = tapirOrg %% "tapir-sttp-stub-server" % Versions.tapir % Test

// json
lazy val playJson = playOrg %% "play-json" % Versions.playJson
lazy val sttpPlayJson = sttpClient3Org %% "play-json" % Versions.sttpPlayJson % Test
lazy val tapirCirce = tapirOrg %% "tapir-json-circe" % Versions.tapir
lazy val tapirOpenApiDocs = tapirOrg %% "tapir-openapi-docs" % Versions.tapir
lazy val tapirOpenApiCirceYaml = tapirOrg %% "tapir-openapi-circe-yaml" % Versions.tapir
lazy val tapirHttp4sServer = tapirOrg %% "tapir-http4s-server" % Versions.tapir
lazy val tapirCore = tapirOrg %% "tapir-core" % Versions.tapir
lazy val tapirSwaggerUi = tapirOrg %% "tapir-swagger-ui-http4s" % Versions.tapir

// STTP core and Circe integration
lazy val sttpCirce = sttpClient3Org %% "circe" % Versions.sttpCirceJson % Test
lazy val sttpCore = sttpClient3Org %% "core" % Versions.sttpCirceJson
lazy val clientBackend = sttpClient3Org %% "async-http-client-backend-zio" % Versions.sttpCirceJson

// Fa-db
lazy val faDbDoobie = faDbOrg %% "doobie" % Versions.fadb
Expand Down Expand Up @@ -197,10 +181,11 @@ object Dependencies {
tapirHttp4sZio,
tapirSwagger,
tapirPlayJson,
tapirCirce,
tapirPrometheus,
tapirStubServer,
playJson,
sttpPlayJson,
sttpCirce,
sttpCore,
awsSecretsManagerSdk,
zioTest,
zioTestSbt,
Expand Down
2 changes: 1 addition & 1 deletion server/src/main/scala/za/co/absa/atum/server/api/controller/BaseController.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
package za.co.absa.atum.server.api.controller

import za.co.absa.atum.server.api.exception.ServiceError
import za.co.absa.atum.server.model.ErrorResponse.{ErrorResponse, GeneralErrorResponse, InternalServerErrorResponse}
import za.co.absa.atum.server.model.{ErrorResponse, GeneralErrorResponse, InternalServerErrorResponse}
import za.co.absa.atum.server.model.SuccessResponse.{MultiSuccessResponse, SingleSuccessResponse}
import za.co.absa.fadb.exceptions.StatusException
import zio._
Expand Down
2 changes: 1 addition & 1 deletion server/src/main/scala/za/co/absa/atum/server/api/controller/CheckpointController.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
package za.co.absa.atum.server.api.controller

import za.co.absa.atum.model.dto.CheckpointDTO
import za.co.absa.atum.server.model.ErrorResponse.ErrorResponse
import za.co.absa.atum.server.model.ErrorResponse
import za.co.absa.atum.server.model.SuccessResponse.SingleSuccessResponse
import zio.IO
import zio.macros.accessible
Expand Down
2 changes: 1 addition & 1 deletion server/src/main/scala/za/co/absa/atum/server/api/controller/CheckpointControllerImpl.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ package za.co.absa.atum.server.api.controller

import za.co.absa.atum.model.dto.CheckpointDTO
import za.co.absa.atum.server.api.service.CheckpointService
import za.co.absa.atum.server.model.ErrorResponse.ErrorResponse
import za.co.absa.atum.server.model.ErrorResponse
import za.co.absa.atum.server.model.SuccessResponse.SingleSuccessResponse
import zio._

Expand Down
2 changes: 1 addition & 1 deletion server/src/main/scala/za/co/absa/atum/server/api/controller/FlowController.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
package za.co.absa.atum.server.api.controller

import za.co.absa.atum.model.dto.{CheckpointDTO, CheckpointQueryDTO}
import za.co.absa.atum.server.model.ErrorResponse.ErrorResponse
import za.co.absa.atum.server.model.ErrorResponse
import za.co.absa.atum.server.model.SuccessResponse.MultiSuccessResponse
import zio.IO
import zio.macros.accessible
Expand Down
2 changes: 1 addition & 1 deletion server/src/main/scala/za/co/absa/atum/server/api/controller/FlowControllerImpl.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ package za.co.absa.atum.server.api.controller

import za.co.absa.atum.model.dto.{CheckpointDTO, CheckpointQueryDTO}
import za.co.absa.atum.server.api.service.FlowService
import za.co.absa.atum.server.model.ErrorResponse.ErrorResponse
import za.co.absa.atum.server.model.ErrorResponse
import za.co.absa.atum.server.model.SuccessResponse.MultiSuccessResponse
import zio._

Expand Down
2 changes: 1 addition & 1 deletion server/src/main/scala/za/co/absa/atum/server/api/controller/PartitioningController.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import za.co.absa.atum.model.dto.{
CheckpointQueryDTO,
PartitioningSubmitDTO
}
import za.co.absa.atum.server.model.ErrorResponse.ErrorResponse
import za.co.absa.atum.server.model.ErrorResponse
import za.co.absa.atum.server.model.SuccessResponse.{MultiSuccessResponse, SingleSuccessResponse}
import zio.IO
import zio.macros.accessible
Expand Down
2 changes: 1 addition & 1 deletion server/src/main/scala/za/co/absa/atum/server/api/controller/PartitioningControllerImpl.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ package za.co.absa.atum.server.api.controller
import za.co.absa.atum.model.dto._
import za.co.absa.atum.server.api.exception.ServiceError
import za.co.absa.atum.server.api.service.PartitioningService
import za.co.absa.atum.server.model.ErrorResponse.{ErrorResponse, InternalServerErrorResponse}
import za.co.absa.atum.server.model.{ErrorResponse, InternalServerErrorResponse}
import za.co.absa.atum.server.model.SuccessResponse.{MultiSuccessResponse, SingleSuccessResponse}
import zio._

Expand Down
Loading
Loading