Skip to content

codex-storage/nim-serde

Repository files navigation

nim-serde

Easy-to-use json serialization capabilities, and a drop-in replacement for std/json.

Quick examples

Opt-in serialization by default:

import pkg/serde/json

type MyType = object
  field1 {.serialize.}: bool
  field2: bool

assert MyType(field1: true, field2: true).toJson == """{"field1":true}"""

Opt-out deserialization by default:

import pkg/serde/json

# All fields deserialized, as none are ignored
type MyType1 = object
  field1: bool
  field2: bool

let jsn1 = """{
                "field1": true,
                "field2": true
              }"""
assert !MyType1.fromJson(jsn1) == MyType1(field1: true, field2: true)

# Don't deserialize ignored fields in OptOut mode
type MyType2 = object
  field1 {.deserialize(ignore=true).}: bool
  field2: bool

let jsn2 = """{
                "field1": true,
                "field2": true,
                "extra": "extra fields don't error in OptOut mode"
              }"""
assert !MyType2.fromJson(jsn2) == MyType2(field1: false, field2: true)

# Note, the ! operator is part of https://github.com/codex-storage/questionable, which retrieves a value if set

Serialize all fields of a type (OptOut mode):

import pkg/serde/json

type MyType {.serialize.} = object
  field1: int
  field2: int

assert MyType(field1: 1, field2: 2).toJson == """{"field1":1,"field2":2}"""

Alias field names in both directions!

import pkg/serde/json

type MyType {.serialize.} = object
  field1 {.serialize("othername"),deserialize("takesprecedence").}: int
  field2: int

assert MyType(field1: 1, field2: 2).toJson == """{"othername":1,"field2":2}"""
let jsn = """{
                "othername":       1,
                "field2":          2,
                "takesprecedence": 3
              }"""
assert !MyType.fromJson(jsn) == MyType(field1: 3, field2: 2)

Supports strict mode, where type fields and json fields must match

import pkg/serde/json

type MyType {.deserialize(mode=Strict).} = object
  field1: int
  field2: int

let jsn = """{
                "field1": 1,
                "field2": 2,
                "extra":  3
              }"""

let res = MyType.fromJson(jsn)
assert res.isFailure
assert res.error of SerdeError
assert res.error.msg == "json field(s) missing in object: {\"extra\"}"

Serde modes

nim-serde uses three different modes to control de/serialization:

OptIn
OptOut
Strict

Modes can be set in the {.serialize.} and/or {.deserialize.} pragmas on type definitions. Each mode has a different meaning depending on if the type is being serialized or deserialized. Modes can be set by setting mode in the serialize or deserialize pragma annotation, eg:

type MyType {.serialize(mode=Strict).} = object
  field1: bool
  field2: bool

Modes reference

serialize deserialize
SerdeMode.OptOut All object fields will be serialized, except fields marked with {.serialize(ignore=true).}. All json keys will be deserialized, except fields marked with {.deserialize(ignore=true).}. No error if extra json fields exist.
SerdeMode.OptIn Only fields marked with {.serialize.} will be serialized. Fields marked with {.serialize(ignore=true).} will not be serialized. Only fields marked with {.deserialize.} will be deserialized. Fields marked with {.deserialize(ignore=true).} will not be deserialized. A SerdeError error is raised if the field is missing in json.
SerdeMode.Strict All object fields will be serialized, regardless if the field is marked with {.serialize(ignore=true).}. Object fields and json fields must match exactly, otherwise a SerdeError is raised.

Default modes

nim-serde will de/serialize types if they are not annotated with serialize or deserialize, but will assume a default mode. By default, with no pragmas specified, serde will always serialize in OptIn mode, meaning any fields to b Additionally, if the types are annotated, but a mode is not specified, serde will assume a (possibly different) default mode.

# Type is not annotated
# A default mode of OptIn (for serialize) and OptOut (for deserialize) is assumed.

type MyObj = object
  field1: bool
  field2: bool

# Type is annotated, but mode not specified
# A default mode of OptOut is assumed for both serialize and deserialize.

type MyObj {.serialize, deserialize.} = object
  field1: bool
  field2: bool

Default mode reference

serialize deserialize
Default (no pragma) OptIn OptOut
Default (pragma, but no mode) OptOut OptOut

Serde field options

Type fields can be annotated with {.serialize.} and {.deserialize.} and properties can be set on these pragmas, determining de/serialization behavior.

For example,

import pkg/serde/json

type
  Person {.serialize(mode=OptOut), deserialize(mode=OptIn).} = object
    id {.serialize(ignore=true), deserialize(key="personid").}: int
    name: string
    birthYear: int
    address: string
    phone: string

let person = Person(
              name: "Lloyd Christmas",
              birthYear: 1970,
              address: "123 Sesame Street, Providence, Rhode Island  12345",
              phone: "555-905-justgivemethedamnnumber!⛽️🔥")

let createRequest = """{
  "name": "Lloyd Christmas",
  "birthYear": 1970,
  "address": "123 Sesame Street, Providence, Rhode Island  12345",
  "phone": "555-905-justgivemethedamnnumber!⛽️🔥"
}"""
assert person.toJson(pretty=true) == createRequest

let createResponse = """{
  "personid": 1,
  "name": "Lloyd Christmas",
  "birthYear": 1970,
  "address": "123 Sesame Street, Providence, Rhode Island  12345",
  "phone": "555-905-justgivemethedamnnumber!⛽️🔥"
}"""
assert !Person.fromJson(createResponse) == Person(id: 1)

key

Specifying a key, will alias the field name. When seriazlizing, json will be written with key instead of the field name. When deserializing, the json must contain key for the field to be deserialized.

ignore

Specifying ignore, will prevent de/serialization on the field.

Serde field options reference

serialize deserialize
key aliases the field name in json deserializes the field if json contains key
ignore
  • OptOut: field not serialized
  • OptIn: field not serialized
  • Strict: field serialized
  • OptOut: field not deserialized
  • OptIn: field not deserialized
  • Strict: field deserialized
  • Deserialization

    serde deserializes using fromJson, and in all instances returns Result[T, CatchableError], where T is the type being deserialized. For example:

    type MyType = object
     field1: bool
     field2: bool
    
    let jsn1 = """{
                   "field1": true,
                   "field2": true
                 }"""
    
    assert !MyType.fromJson(jsn1) == MyType(field1: true, field2: true)

    If there was an error during deserialization, the result of fromJson will contain it:

    import pkg/serde/json
    
    type MyType {.deserialize(mode=Strict).} = object
      field1: int
      field2: int
    
    let jsn = """{
                    "field1": 1,
                    "field2": 2,
                    "extra":  3
                  }"""
    
    let res = MyType.fromJson(jsn)
    assert res.isFailure
    assert res.error of SerdeError
    assert res.error.msg == "json field(s) missing in object: {\"extra\"}"

    Custom types

    If serde can't de/serialize a custom type, de/serialization can be supported by overloading % and fromJson. For example:

    type
      Address* = distinct array[20, byte]
      SerializationError* = object of CatchableError
    
    func `%`*(address: Address): JsonNode =
      %($address)
    
    func fromJson(_: type Address, json: JsonNode): ?!Address =
      expectJsonKind(Address, JString, json)
      without address =? Address.init(json.getStr), error:
        return failure newException(SerializationError,
          "Failed to convert '" & $json & "' to Address: " & error.msg)
      success address

    Serializing to string (toJson)

    toJson is a shortcut for serializing an object into its serialized string representation:

    import pkg/serde/json
    
    type MyType {.serialize.} = object
      field1: string
      field2: bool
    
    let mt = MyType(field1: "hw", field2: true)
    assert mt.toJson == """{"field1":"hw","field2":true}"""

    This comes in handy, for example, when sending API responses:

    let availability = getAvailability(...)
    return RestApiResponse.response(availability.toJson,
                                    contentType="application/json")

    std/json drop-in replacment

    nim-serde can be used as a drop-in replacement for the standard library's json module, with a few notable improvements.

    Instead of importing std/json into your application, pkg/serde/json can be imported instead:

    - import std/json
    + import pkg/serde/json

    As with std/json, % can be used to serialize a type into a JsonNode:

    import pkg/serde/json
    
    assert %"hello" == newJString("hello")

    And %* can be used to serialize objects:

    import pkg/serde/json
    
    let expected = newJObject()
    expected["hello"] = newJString("world")
    assert %*{"hello": "world"} == expected

    As well, serialization of types can be overridden, and serialization of custom types can be introduced. Here, we are overriding the serialization of int:

    import pkg/serde/json
    
    func `%`(i: int): JsonNode =
      newJInt(i + 1)
    
    assert 1.toJson == "2"

    parseJson and exception tracking

    Unfortunately, std/json's parseJson can raise an Exception, so proper exception tracking breaks, eg

    ## Fails to compile:
    ## Error: parseJson(me, false, false) can raise an unlisted exception: Exception
    
    import std/json
    
    {.push raises:[].}
    
    type
      MyAppError = object of CatchableError
    
    proc parseMe(me: string): JsonNode =
      try:
        return me.parseJson
      except CatchableError as error:
        raise newException(MyAppError, error.msg)
    
    assert """{"hello":"world"}""".parseMe == %* { "hello": "world" }

    This is due to std/json's parseJson incorrectly raising Exception. This can be worked around by instead importing serde and calling its JsonNode.parse routine. Note that serde's JsonNode.parse returns a Result[JsonNode, CatchableError] instead of just a plain JsonNode object as in std/json's parseJson:

    import pkg/serde/json
    
    {.push raises:[].}
    
    type
      MyAppError = object of CatchableError
    
    proc parseMe(me: string): JsonNode {.raises: [MyAppError].} =
      without parsed =? JsonNode.parse(me), error:
        raise newException(MyAppError, error.msg)
      parsed
    
    assert """{"hello":"world"}""".parseMe == %* { "hello": "world" }

    Known issues

    There is a known issue when using mixins with generic overloaded procs like fromJson. At the time of mixin call, only the fromJson overloads in scope of the called mixin are available to be dispatched at runtime. There could be other fromJson overloads declared in other modules, but are not in scope at the time the mixin was called. Therefore, anytime fromJson is called targeting a declared overload, it may or may not be dispatchable. This can be worked around by forcing the fromJson overload into scope at compile time. For example, in your application where the fromJson overload is defined, at the bottom of the module add:

    static: MyType.fromJson("")

    This will ensure that the MyType.fromJson overload is dispatchable.

    The basic types that serde supports should already have their overloads forced in scope in the deserializer module.

    For an illustration of the problem, please see this narrow example by @gmega.