Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Poc #3

Merged
merged 27 commits into from
Apr 28, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
0daff4c
init
HubertBalcerzak Apr 6, 2021
659d734
Added dependencies
MarconZet Apr 6, 2021
53d2573
Copied code from article
MarconZet Apr 6, 2021
4dd2c65
compilation fixes
HubertBalcerzak Apr 6, 2021
a709d1b
Application confi
MarconZet Apr 6, 2021
fd36c4e
add basic tests
HubertBalcerzak Apr 6, 2021
732d642
fix tests
HubertBalcerzak Apr 6, 2021
1b6a6f3
Added test with enumeration field type
MarconZet Apr 7, 2021
5751495
Added line that causes compilation to fail
MarconZet Apr 12, 2021
cdcb9da
reorganize dependencies
HubertBalcerzak Apr 17, 2021
826435e
use AkkaTestKit in tests, reorganize code
HubertBalcerzak Apr 17, 2021
9392fd4
add scalafmt
HubertBalcerzak Apr 19, 2021
b964abf
change package, add scalac options
HubertBalcerzak Apr 24, 2021
66599b6
add circleci config
HubertBalcerzak Apr 24, 2021
ac28923
bump akka version
HubertBalcerzak Apr 24, 2021
2b71a41
use AtomicReference
HubertBalcerzak Apr 24, 2021
8f141c9
fix formatting
HubertBalcerzak Apr 24, 2021
39d9e64
Adder runtime codex class checks
MarconZet Apr 24, 2021
22374a0
Merge remote-tracking branch 'origin/poc' into poc
MarconZet Apr 24, 2021
a3a5604
Changed Codecs from trait to object
MarconZet Apr 24, 2021
f7fdc0e
Created a Borer codec for OffsetDateTime
MarconZet Apr 26, 2021
2347a2f
Created a Borer codec for SourceRef and SinkRef
MarconZet Apr 27, 2021
2a5511c
Minor refactor
MarconZet Apr 27, 2021
703b6ad
Made akka dependencies provided
MarconZet Apr 28, 2021
61cf71a
Updated config for circleci
MarconZet Apr 28, 2021
108e066
Changed wording of RuntimeException when trying to serialise object t…
MarconZet Apr 28, 2021
9b526f8
Fixed errors on tests
MarconZet Apr 28, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
version: 2.1
jobs:
build:
machine:
# https://circleci.com/docs/2.0/configuration-reference/#available-machine-images
image: ubuntu-2004:202101-01

working_directory: ~/akka-safer-serializer

steps:
- checkout

# -batch is necessary to make sure sbt doesn't stall waiting for user input in case of certain failures
- run: echo '-batch' >> .sbtopts
- run: sbt update
- run: sbt scalafmtCheckAll
- run: sbt compile
- run: sbt test:compile
- run: sbt test
13 changes: 13 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,15 @@
*.class
*.log

.bsp/
.idea/

dist/*
target/
lib_managed/
src_managed/
project/boot/
project/plugins/project/
.history
.cache
.lib/
53 changes: 53 additions & 0 deletions .scalafmt.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
version = 2.6.4

style = defaultWithAlign

docstrings = JavaDoc
indentOperator.preset = spray
maxColumn = 120
rewrite.rules = [RedundantParens, AvoidInfix]
unindentTopLevelOperators = true
align.tokens = [{code = "=>", owner = "Case"}]
align.openParenDefnSite = false
align.openParenCallSite = false
optIn.breakChainOnFirstMethodDot = false
optIn.configStyleArguments = false
danglingParentheses.defnSite = false
danglingParentheses.callSite = false
rewrite.neverInfix.excludeFilters = [
and
min
max
until
to
by
eq
ne
"should.*"
"contain.*"
"must.*"
in
ignore
be
taggedAs
thrownBy
synchronized
have
when
size
only
noneOf
oneElementOf
noElementsOf
atLeastOneElementOf
atMostOneElementOf
allElementsOf
inOrderElementsOf
theSameElementsAs
message
]
rewriteTokens = {
"⇒": "=>"
"→": "->"
"←": "<-"
}
31 changes: 31 additions & 0 deletions build.sbt
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
import Dependencies._
import org.scalafmt.sbt.ScalafmtPlugin.autoImport.scalafmtOnCompile

ThisBuild / scalaVersion := "2.12.13"
ThisBuild / version := "0.1.0-SNAPSHOT"
ThisBuild / organization := "org.virtuslab"
ThisBuild / organizationName := "VirtusLab"
Global / onChangedBuildSource := ReloadOnSourceChanges

HubertBalcerzak marked this conversation as resolved.
Show resolved Hide resolved
lazy val root = (project in file("."))
.settings(
name := "akka-safer-serializer",
scalafmtOnCompile := true,
HubertBalcerzak marked this conversation as resolved.
Show resolved Hide resolved
scalacOptions ++= Seq(
"-deprecation",
"-encoding",
"UTF-8",
"-feature",
"-language:_",
"-Xfatal-warnings",
"-Xlog-reflective-calls",
"-Xlint:_",
"-Ybackend-parallelism",
"8",
"-Ywarn-dead-code",
"-Ywarn-unused:-imports,_",
"-unchecked"),
libraryDependencies ++= deps
)

// See https://www.scala-sbt.org/1.x/docs/Using-Sonatype.html for instructions on how to publish to Sonatype.
33 changes: 33 additions & 0 deletions project/Dependencies.scala
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
import sbt._

object Dependencies {
val borerVersion = "1.6.3"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's already Borer 1.7.0 AFAICS

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or now 1.7.1 with #16 fixed ;)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoops looks that 1.7+ is only for Scala 2.13... lemme discuss with the Borer author...

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or maybe I'll manage to migrate Hydra to 2.13 right now, lemme see :trollface:

val akkaVersion = "2.6.13"

val scalaTest = "org.scalatest" %% "scalatest" % "3.2.2"
val akkaTyped = "com.typesafe.akka" %% "akka-actor-typed" % akkaVersion
MarconZet marked this conversation as resolved.
Show resolved Hide resolved
val akkaTestKit = "com.typesafe.akka" %% "akka-actor-testkit-typed" % akkaVersion
val akkaStream = "com.typesafe.akka" %% "akka-stream" % akkaVersion
val akkaStreamTestKit = "com.typesafe.akka" %% "akka-stream-testkit" % akkaVersion

val enumeratum = "com.beachape" %% "enumeratum" % "1.6.1"

val borerCore = "io.bullet" %% "borer-core" % borerVersion
val borerDerivation = "io.bullet" %% "borer-derivation" % borerVersion
val borerAkka = "io.bullet" %% "borer-compat-akka" % borerVersion

val reflections = "net.oneandone.reflections8" % "reflections8" % "0.11.7"

val deps = Seq(
scalaTest % Test,
akkaTestKit % Test,
akkaStreamTestKit % Test,
akkaTyped % Provided,
akkaStream % Provided,
enumeratum % Test,
borerCore,
borerDerivation,
borerAkka,
reflections
)
}
1 change: 1 addition & 0 deletions project/build.properties
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
sbt.version=1.4.9
PawelLipski marked this conversation as resolved.
Show resolved Hide resolved
1 change: 1 addition & 0 deletions project/plugins.sbt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
addSbtPlugin("org.scalameta" % "sbt-scalafmt" % "2.4.2")
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
package org.virtuslab.akkasaferserializer

import akka.serialization.Serializer
import io.bullet.borer.{Cbor, Codec, Decoder, Encoder}

import java.util.concurrent.atomic._
import scala.reflect.ClassTag

trait BorerAkkaSerializer[Ser] extends Serializer {

private val registrations = new AtomicReference[List[(Class[_], Codec[_])]](List.empty)

//noinspection UnitMethodIsParameterless
PawelLipski marked this conversation as resolved.
Show resolved Hide resolved
protected def register[T <: Ser: Encoder: Decoder: ClassTag]: Unit = {
registrations.getAndAccumulate(List(scala.reflect.classTag[T].runtimeClass -> Codec.of[T]), _ ++ _)
}

override def includeManifest: Boolean = true

override def toBinary(o: AnyRef): Array[Byte] = {
val codec = getCodec(o.getClass, "encoder")
val encoder = codec.encoder.asInstanceOf[Encoder[AnyRef]]
Cbor.encode(o)(encoder).toByteArray
}

override def fromBinary(bytes: Array[Byte], manifest: Option[Class[_]]): AnyRef = {
val codec = getCodec(manifest.get, "decoder")
val decoder = codec.decoder.asInstanceOf[Decoder[AnyRef]]
Cbor.decode(bytes).to[AnyRef](decoder).value
}

private def getCodec(clazzToFind: Class[_], item: String): Codec[_] = {
registrations
.get()
.collectFirst {
case (clazz, codec) if clazz.isAssignableFrom(clazzToFind) => codec
}
.getOrElse {
throw new RuntimeException(s"$item for $clazzToFind is not registered")
}
}

protected def runtimeChecks(cl: Class[_]): Unit = {
import org.reflections8.Reflections
import scala.collection.convert.ImplicitConversions._

val reflections = new Reflections()

def findAllObjects[T](cl: Class[T]): Seq[Class[_ <: T]] = reflections.getSubTypesOf(cl).toSeq

val found = findAllObjects(cl)

val foundClasses = found.filterNot(_.isInterface)

foundClasses.foreach(getCodec(_, "codec"))
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
package org.virtuslab.akkasaferserializer

import akka.actor
import akka.actor.typed.{ActorRef, ActorRefResolver, ActorSystem}
import akka.serialization.Serialization
import akka.stream.{SinkRef, SourceRef, StreamRefResolver}
import io.bullet.borer.Codec

import java.io.{ByteArrayInputStream, ByteArrayOutputStream, ObjectInputStream, ObjectOutputStream}
import java.time.OffsetDateTime

object StandardCodecs {

implicit def sinkRefCodec[T](implicit system: actor.ActorSystem = serializationSystem): Codec[SinkRef[T]] = {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gr8, lemme quickly check on my commercial project (let's codename it Hydra from now on...)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoops I'm blocked on this problem with type aliases (since borer 1.7.1 is not yet available for Scala 2.12, only 2.13)... let's leave as is for now

val resolver = StreamRefResolver(ActorSystem.wrap(system))

Codec.bimap[String, SinkRef[T]](resolver.toSerializationFormat(_: SinkRef[T]), resolver.resolveSinkRef)
}

implicit def sourceRefCodec[T](implicit system: actor.ActorSystem = serializationSystem): Codec[SourceRef[T]] = {
val resolver = StreamRefResolver(ActorSystem.wrap(system))

Codec.bimap[String, SourceRef[T]](resolver.toSerializationFormat(_: SourceRef[T]), resolver.resolveSourceRef)
}

private def serializationSystem: actor.ActorSystem = Serialization.getCurrentTransportInformation().system

implicit val offsetDateTimeCodec: Codec[OffsetDateTime] =
Codec.bimap[Array[Byte], OffsetDateTime](
serializeSerializable[OffsetDateTime],
deserializeSerializable[OffsetDateTime])

private def serializeSerializable[T <: java.io.Serializable](ser: T): Array[Byte] = {
val os = new ByteArrayOutputStream()
val oos = new ObjectOutputStream(os)
oos.writeObject(ser)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm this uses Java serialization under the hood right? This is what we'd like to avoid, mostly due to performance & security concerns: https://doc.akka.io/docs/akka/current/serialization.html#java-serialization

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's okay just for now for prototyping, but this needs to be replaced... pls take a look at https://github.com/plokhotnyuk/jsoniter-scala, this might be useful both OffsetDateTime and for #7

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(so: pls leave as is on this PR + open an issue for better implementation)

oos.close()
os.toByteArray
}

private def deserializeSerializable[T <: java.io.Serializable](buffer: Array[Byte]): T = {
val ois = new ObjectInputStream(new ByteArrayInputStream(buffer))
val ser = ois.readObject.asInstanceOf[T]
ois.close()
ser
}
}
10 changes: 10 additions & 0 deletions src/test/resources/application.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
akka.actor {
serializers {
borer-cbor = "org.virtuslab.akkasaferserializer.serializer.TestBorerAkkaSerializer"
}
serialization-bindings {
"org.virtuslab.akkasaferserializer.data.BorerSerializable" = borer-cbor
}
enable-additional-serialization-bindings = on
allow-java-serialization = off
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
package org.virtuslab.akkasaferserializer

import akka.{Done, NotUsed}
import akka.actor.testkit.typed.scaladsl.{ActorTestKit, SerializationTestKit}
import akka.actor.typed.ActorSystem
import akka.pattern.pipe
import akka.stream.{Materializer, SinkRef, SourceRef}
import akka.stream.scaladsl.{Sink, Source, StreamRefs}
import com.typesafe.config.ConfigFactory
import org.scalatest.matchers.should.Matchers
import org.scalatest.wordspec.AnyWordSpecLike
import org.virtuslab.akkasaferserializer.data.CodecsData.{SinkRefClass, SourceRefClass}

import scala.concurrent.Future

class AkkaStreamSerializerSpec extends AnyWordSpecLike with Matchers {

"BorerAkkaSerializer" should {

val config = ConfigFactory.load()
val testKit: ActorTestKit = ActorTestKit(config)
val system: ActorSystem[Nothing] = testKit.system
val serializationTestKit = new SerializationTestKit(system)

implicit val materializer: Materializer = Materializer.createMaterializer(system)

"serialize class with SourceRef" in {
val source: Source[Int, NotUsed] = Source(1 to 100)
val ref: SourceRef[Int] = source.runWith(StreamRefs.sourceRef())
serializationTestKit.verifySerialization(SourceRefClass(ref))
}

"serialize class with SinkRef" in {
val sink: Sink[Int, Future[Int]] = Sink.fold[Int, Int](0)(_ + _)
val ref: SinkRef[Int] = StreamRefs.sinkRef[Int]().to(sink).run()
serializationTestKit.verifySerialization(SinkRefClass(ref))
}

}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
package org.virtuslab.akkasaferserializer

import akka.actor.testkit.typed.scaladsl.{ActorTestKit, SerializationTestKit}
import akka.actor.typed.ActorSystem
import com.typesafe.config.ConfigFactory
import org.virtuslab.akkasaferserializer.data.Animal.{Lion, Tiger}
import org.virtuslab.akkasaferserializer.data.Zoo.{GreetingZoo, NorthZoo}
import org.scalatest.matchers.should.Matchers
import org.scalatest.wordspec.AnyWordSpecLike
import org.virtuslab.akkasaferserializer.data.CodecsData.DateTimeClass
import org.virtuslab.akkasaferserializer.data.Greeting

import java.time.OffsetDateTime

class TestBorerAkkaSerializerSpec extends AnyWordSpecLike with Matchers {

"BorerAkkaSerializer" should {

val config = ConfigFactory.load()
val testKit: ActorTestKit = ActorTestKit(config)
val system: ActorSystem[Nothing] = testKit.system
val serializationTestKit = new SerializationTestKit(system)

"serialize singleton" in {
serializationTestKit.verifySerialization(Tiger)
}

"serialize final case class" in {
serializationTestKit.verifySerialization(Lion("lion"))
}

"serialize nested case class" in {
serializationTestKit.verifySerialization(NorthZoo(Lion("lion")))
}

"serialize case class with enumeration" in {
serializationTestKit.verifySerialization(GreetingZoo(Lion("lion"), Greeting.Hello))
}

"serialize case class with additional codecs form StandardCodecs" in {
serializationTestKit.verifySerialization(DateTimeClass(OffsetDateTime.now()))
}
}

}
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
package org.virtuslab.akkasaferserializer.data

trait BorerSerializable
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
package org.virtuslab.akkasaferserializer.data

import akka.stream.{SinkRef, SourceRef}

import java.time.OffsetDateTime

sealed trait CodecsData extends BorerSerializable

object CodecsData {
case class DateTimeClass(offsetDateTime: OffsetDateTime) extends CodecsData

case class SourceRefClass(ref: SourceRef[Int]) extends CodecsData

case class SinkRefClass(ref: SinkRef[Int]) extends CodecsData
}
Loading