Generic DecoderAndEncorder Example / Issue, Generic[T] Decoder[T], Generic[T] Encoder[T]

[aukgit]

Hello @travisbrown,

How are you? Greetings for the day.

I have things set up correctly. Data comes properly. However, I am having issues with generic types. As I am completely new to scala, probably I am missing something. However, I have read in the StackOverflow articles that circle doesn't support conversion of Any, Object (which makes sense). However, I want to make a generic one where I have a [T] and that [T] transforms into something that can be convertible to JSON. Probably there is a boundary (I am unaware of) that can be a help.

Main Goal

Make circle JSON encode/decode on the abstract class level. Just like NewtonJson (https://www.newtonsoft.com/json)

Versions (Scala 2.13.1, Circle: 0.12.3)

lazy val scalaVersionF = "2.13.1"
lazy val circeVersion = "0.12.3"

  "io.circe" %% "circe-core" % circeVersion,
  "io.circe" %% "circe-generic" % circeVersion,
  "io.circe" %% "circe-parser" % circeVersion,
  "de.heikoseeberger" %% "akka-http-circe" % "1.31.0",

Issue Summary

Things work fine when the actual type is given, however, it doesn't work when generics are used.

AnyVal JSON Add

Reference: https://www.programcreek.com/scala/io.circe.Decoder

import io.circe.{Decoder, Encoder}
import shapeless.Unwrapped

trait AnyValCirceEncoding {
  implicit def anyValEncoder[V, U](implicit ev: V <:< AnyVal,
    V: Unwrapped.Aux[V, U],
    encoder: Encoder[U]): Encoder[V] = {
    val _ = ev
    encoder.contramap(V.unwrap)
  }

  implicit def anyValDecoder[V, U](implicit ev: V <:< AnyVal,
    V: Unwrapped.Aux[V, U],
    decoder: Decoder[U]): Decoder[V] = {
    val _ = ev
    decoder.map(V.wrap)
  }
}

object AnyValCirceEncoding extends AnyValCirceEncoding
trait CircleJsonSupport extends de.heikoseeberger.akkahttpcirce.FailFastCirceSupport
  with AnyValCirceEncoding

Examples

Abstract API Sample, CircleJsonSupport integrated

trait RestWebApiContracts[TTable, TRow, TKey]
  extends
    RestWebApi[TTable, TRow, TKey] with
    RestWebApiBodyProcessor[TTable, TRow, TKey] with
    RestWebApiMessages[TTable, TRow, TKey] with
    RestWebApiJson[TTable, TRow, TKey] with
    RestWebApiEntityJsonAdapter[TTable, TRow, TKey] with
    CircleJsonSupport // has the support for the Circle any value type

Build fails

abstract class AbstractRestWebApi[TTable, TRow, TKey]
(components : ControllerComponents)
  extends
    AbstractController(components) with
    RestWebApiContracts[TTable, TRow, TKey] {
  val service : AbstractBasicPersistentService[TTable, TRow, TKey]

  def getAll : Action[AnyContent] = Action { implicit request =>
      // cannot create for List[TRow]
      implicit val listEncoder: Encoder.AsObject[List[TRow]] = deriveEncoder[List[TRow]]
      val records : List[TRow] = service.getAll
      val json = records.asJson(listEncoder).space2
      Ok(json)
    }
}

Build Success and works fine if the same code is written in the actual class

class CampaignsApiController @Inject()(
  campaignService : CampaignService,
  components      : ControllerComponents)
  extends AbstractRestWebApi[Campaign, CampaignRow, Int](components) {
    override def getAll : Action[AnyContent] = Action { implicit request : Request[AnyContent] =>
      val campaigns: List[CampaignRow] = campaignService.getAll
      val json = campaigns.asJson.spaces2
      Ok(json) // works perfect and returns json array
    }
}

Build Success and works fine if casting happens properly

abstract class AbstractRestWebApi[TTable, TRow, TKey]
(components : ControllerComponents)
  extends
    AbstractController(components) with
    RestWebApiContracts[TTable, TRow, TKey] {
  val service : AbstractBasicPersistentService[TTable, TRow, TKey]

  def getAll : Action[AnyContent] = Action { implicit request =>
      // cannot create for List[TRow]
      implicitvallistEncoder:Encoder[List[CampaignRow]]=deriveEncoder[List[CampaignRow]]
      val campaigns=service.getAll
      val json =campaigns.asInstanceOf[List[CampaignRow]].asJson(listEncoder).spaces2
      Ok(json) // perfectly
    }
}

Implicts failed

Base Class

  lazy implicit val listEncoder : Encoder[List[TRow]] = Encoder[List[TRow]]
  lazy implicit val encoder : Encoder[TRow] = Encoder[TRow]
  lazy implicit val decoder : Decoder[TRow] = Decoder[TRow]
  lazy implicit val listDecoder : Decoder[List[TRow]] = Decoder[List[TRow]]

Concreate class Class

  override implicit val listEncoder : Encoder[List[CampaignRow]] = Encoder[List[CampaignRow]]
  override implicit val encoder : Encoder[CampaignRow] = Encoder[CampaignRow]
  override implicit val decoder : Decoder[CampaignRow] = Decoder[CampaignRow]
  override implicit val listDecoder : Decoder[List[CampaignRow]] = Decoder[List[CampaignRow]]

Also doesn't help

Case Class for CampaignRow (Slick Class)

case class CampaignRow(campaignid: Int, campaignname: String, contentcategoryid: String, totalbudgetcpm: Double = 0.0, spendalready: Double = 0.0, remainingamount: Double = 0.0, startdate: Option[Double] = Some(0.0), enddate: Option[Double] = Some(0.0), impressioncount: Int = 0, demandsideplatformid: Int, isrunning: Int = 0, priority: Int = 999, isretricttousergender: Int = 0, expectedusergender: Option[String], publisherid: Option[Int])

Build errors faced during

- could not find implicit value for parameter encoder: io.circe.Encoder[List[TRow]]
- could not find Lazy implicit value of type io.circe.generic.encoding.DerivedAsObjectEncoder[List[TRow]]
      implicit val listEncoder: Encoder.AsObject[List[TRow]] = deriveEncoder[List[TRow]]

[travisbrown]

@aukgit I'm not sure I understand the exact issue, but here are some examples that I hope might help.

First, if you have CampaignRow defined as above, you can use generic derivation to define instances like this:

import io.circe.{Decoder, Encoder}, io.circe.generic.semiauto._

implicit val encodeCampaignRow: Encoder[CampaignRow] = deriveEncoder
implicit val decodeCampaignRow: Decoder[CampaignRow] = deriveDecoder

And then:

scala> import io.circe.syntax._
import io.circe.syntax._

scala> CampaignRow(1, "abc", "def", demandsideplatformid = 0, expectedusergender = None, publisherid = None).asJson
res3: io.circe.Json =
{
  "campaignid" : 1,
  "campaignname" : "abc",
  "contentcategoryid" : "def",
  "totalbudgetcpm" : 0.0,
  "spendalready" : 0.0,
  "remainingamount" : 0.0,
  "startdate" : 0.0,
  "enddate" : 0.0,
  "impressioncount" : 0,
  "demandsideplatformid" : 0,
  "isrunning" : 0,
  "priority" : 999,
  "isretricttousergender" : 0,
  "expectedusergender" : null,
  "publisherid" : null
}

Once you have that, you should never explicitly need to define a List[CampaignRow], since Circe automatically provides e.g. Encoder[List[SomeType]] for any concrete SomeType with an implicit Encoder instance in scope. Calling deriveEncoder[List[CampaignRow]] is almost certainly not doing what you want—it will derive an encoder for the List ADT, which will produce JSON like {"::": [...]}, not an ordinary JSON array.

So you should just make sure you have an encoder in scope for CampaignRow and then call asJson on the list of CampaignRow values, and it should just work.

If you're working with a generic type instead of a concrete class like CampaignRow, you'll need an Encoder constraint on the generic type. That should look something like this:

abstract class AbstractRestWebApi[TTable, TRow: Encoder, TKey]
(components : ControllerComponents)
  extends
    AbstractController(components) with
    RestWebApiContracts[TTable, TRow, TKey] {
  val service : AbstractBasicPersistentService[TTable, TRow, TKey]

  def getAll : Action[AnyContent] = Action { implicit request =>
      val records : List[TRow] = service.getAll
      val json = records.asJson.space2
      Ok(json)
    }
}

Which is syntactic sugar for the following:

abstract class AbstractRestWebApi[TTable, TRow, TKey]
(components : ControllerComponents)
(implicit encodeTRow: Encoder[TRow])
  extends
    AbstractController(components) with
    RestWebApiContracts[TTable, TRow, TKey] {
  val service : AbstractBasicPersistentService[TTable, TRow, TKey]

  def getAll : Action[AnyContent] = Action { implicit request =>
      val records : List[TRow] = service.getAll
      val json = records.asJson(encodeTRow).space2
      Ok(json)
    }
}

You'd have to include this constraint on any methods or other classes that instantiate this class with a generic type, all the way up to the point where there's a concrete type.

Hope that helps!

[aukgit]

Thank you @travisbrown for the quick help.
I will give this (implicit encodeTRow: Encoder[TRow]) a TRY

For now, I have found a workaround using def, instead of val.

As I mentioned before, these below ones failed:

Failed Example

// abstract class
  lazy implicit val listEncoder : Encoder[List[TRow]] = Encoder[List[TRow]]
  lazy implicit val encoder : Encoder[TRow] = Encoder[TRow]
  lazy implicit val decoder : Decoder[TRow] = Decoder[TRow]
  lazy implicit val listDecoder : Decoder[List[TRow]] = Decoder[List[TRow]]


// Concreate class
  override implicit val listEncoder : Encoder[List[CampaignRow]] = Encoder[List[CampaignRow]]
  override implicit val encoder : Encoder[CampaignRow] = Encoder[CampaignRow]
  override implicit val decoder : Decoder[CampaignRow] = Decoder[CampaignRow]
  override implicit val listDecoder : Decoder[List[CampaignRow]] = Decoder[List[CampaignRow]

Success Example

// abstract class
  def listEncoder : Encoder[List[TRow]]
  def encoder : Encoder[TRow]
  def  decoder : Decoder[TRow]
  def listDecoder : Decoder[List[TRow]]

// Concreate class
  override def listEncoder : Encoder[List[CampaignRow]] = Encoder[List[CampaignRow]]

  override def encoder : Encoder[CampaignRow] = Encoder[CampaignRow]

  override def decoder : Decoder[CampaignRow] = Decoder[CampaignRow]

  override def listDecoder : Decoder[List[CampaignRow]] = Decoder[List[CampaignRow]]

Probably I will use an Encoder encapsulate class to solve this issue.

class GenericCodec[TRow : Encoder] {
  def listEncoder : Encoder[List[TRow]] = Encoder[List[TRow]]
  def encoder : Encoder[TRow] = Encoder[TRow]
  def decoder : Decoder[TRow] = Decoder[TRow]
  def listDecoder : Decoder[List[TRow]] = Decoder[List[TRow]]
}

val genericCodec = new GenericCodec[CampaignRow]

[aukgit]

Hello @travisbrown ,

I have one more question, probably a silly one, however, I couldn't find a good answer yet.

For generics what is the difference amount these below categories

category1: class Generic[+A] vs class Generic[A]

category2: class Generic[-A] vs class Generic[A])

[aukgit]

@travisbrown

In the future can we make dynamic decoders and encoders which may fail in runtime if not successful

Proposal,

implicit val dynamicCodec: DynamicCodeRuntimeCodec[T] = new DynamicCodeRuntimeCodec[T]

May fail on run time if cannot parse to T

[travisbrown]

@aukgit There's no reason you couldn't have a decoder that relied on runtime reflection to decode an arbitrary type (although you'd probably need an implicit ClassTag[T]). I think I've written an example of this before, although I can't find it at the moment. It's pretty far from the spirit of this kind of type-ful programming, though, and I don't know of anyone who actually does this.

[aukgit]

Will you please kindly point me to that?

[travisbrown]

Sorry, as I said I can't find it at the moment. IIRC it was something I posted in a Gist or Stack Overflow or somewhere like that, but searching for variations of "manifest classtag circe scala" doesn't turn it up. There's a similar example in this old blog post of mine. In any case I definitely don't recommend it, and especially if you're just getting started with Scala I'd recommend steering clear of runtime reflection for as long as you can.

[aukgit]

Hi @travisbrown

Thank you for taking your valuable time and replying.

Here is what I tried and having some issues.

package shared.io.traits.jsonParse

import io.circe.generic.codec.DerivedAsObjectCodec
import io.circe.generic.decoding.DerivedDecoder
import shapeless.{Lazy, tag}

class JsonDecoders[T]
(implicit decode : Lazy[DerivedDecoder[T]], decodeCodec : DerivedAsObjectCodec[T])
{

  import io.circe._
  import io.circe.generic.semiauto._

  def defaultDecoder() : Decoder[T] = deriveDecoder[T]

  def codec : Codec.AsObject[T] = deriveCodec[T](decodeCodec)
}

object WApp extends App {

  import shared.com.ortb.persistent.schema.Tables._

  val re = new JsonDecoders[AuctionRow] // Works fine, gets compiled.
  val re2 = new JsonDecoders[List[AuctionRow]] // fails to compiled
  /**
Error:(30, 13) could not find Lazy implicit value of type io.circe.generic.decoding.DerivedDecoder[List[shared.com.ortb.persistent.schema.Tables.AuctionRow]]
  val re2 = new JsonDecoders[List[AuctionRow]]
**/
}

My question of how to create the List or Iterable type of T for that JsonDecoder class.

Or,

Probably using reflection.??

[aukgit]

Things gets fixed if I import all these

import io.circe.generic.semiauto._
import io.circe._
import io.circe.generic.auto._
import io.circe.parser._
import io.circe.generic.auto._
import io.circe.generic.encoding.DerivedAsObjectEncoder
import io.circe.generic.encoding._
import io.circe.{ Decoder, Encoder }, io.circe.generic.semiauto._
import io.circe.{Decoder, Encoder}
import io.circe.generic.semiauto._
import io.circe.syntax._
import io.circe.Decoder.AccumulatingResult
import io.circe.generic.JsonCodec
import io.circe.{Decoder, _}
import io.circe.generic.codec.DerivedAsObjectCodec
import io.circe.generic.decoding.DerivedDecoder
import io.circe.generic.semiauto.{deriveDecoder, _}
import shapeless.Lazy
import shared.com.ortb.implicits.implementations.CirceJsonSupport

Which one to keep as organize import also fails to build again.

[aukgit]

import io.circe.generic.auto._ this one fixes it.

[aukgit]

For anyone who is going to use the generic version of decoders, codecs for T and List[T]

Example Code : https://bit.ly/2RXr7gT

package shared.io.traits.jsonParse
import io.circe.generic.auto._ // must to import.
import io.circe.generic.codec.DerivedAsObjectCodec
import io.circe.generic.decoding.DerivedDecoder
import io.circe.generic.encoding.DerivedAsObjectEncoder
import io.circe.generic.semiauto.{deriveDecoder, _}
import io.circe.{Decoder, Encoder, _}
import shapeless.Lazy
import shared.com.ortb.implicits.implementations.CirceJsonSupport

class JsonCirceEncoder[T]
(
  implicit val decodeCodec : DerivedAsObjectCodec[T],
  implicit val decoder : Lazy[DerivedDecoder[T]],
  val encoder : Lazy[DerivedAsObjectEncoder[T]],
  val decodeListCodec : DerivedAsObjectCodec[List[T]]){
  implicit def defaultCodec : Codec.AsObject[T] = deriveCodec[T](decodeCodec)

  implicit def defaultListCodec : Codec.AsObject[List[T]] =
    deriveCodec[List[T]](decodeListCodec)


  implicit def defaultDecoder : Decoder[T] = deriveDecoder[T]

  implicit def defaultListDecoder : Decoder[List[T]] = deriveDecoder[List[T]]

  implicit def defaultEncoder : Encoder[T] = deriveEncoder[T]

  implicit def defaultListEncoder : Encoder[List[T]] = deriveEncoder[List[T]]

}

object ExampleApp extends App with CirceJsonSupport {

  import shared.com.ortb.persistent.schema.Tables._

  val re = new JsonCirceEncoder[AuctionRow]

}