Ignore null value and use the default value when deserializing from json string

[ghost]

Ignore null value and use the default value when deserializing from json string

[NightlyNexus]

See #762

[ghost]

It's not what i want.

My bean is:

class Foo @SuppressWarnings("unused") constructor() {
    @Json(name = "id")
    var id: Int = 0

    @Json(name = "name")
    var name: String = ""
}

And the serve return json :

{
"id": 1,
"name": null
}

About the "name" filed, I want to skip the null value and use the the default "", and also null array filed.
How can i deal with those cases?

[NightlyNexus]

there are a few ways to achieve what you want. here's a relatively short way that comes to mind:

class Foo {
  var id: Int = 0
  internal var name: String? = ""
  fun getName(): String = name ?: ""
  fun setName(name: String) {
    this.name = name
  }
}

@Test fun foo() {
  val encoded = """
    {
      "id": 1,
      "name": null
    }"""
  val decoded = Foo()
  val moshi = Moshi.Builder().add(KotlinJsonAdapterFactory()).build()
  val adapter = moshi.adapter(Foo::class.java)
  adapter.fromJson(encoded)!!.run {
    assertThat(id).isEqualTo(1)
    assertThat(getName()).isEqualTo("")
  }
  assertThat(adapter.toJson(decoded)).isEqualTo("""{"id":0,"name":""}""")
}

[leleliu008]

@NightlyNexus although it works fine, but I think it isn't a best solution. Because the Foo class generated by tools, not by hand usually, your solution is not convenient. I think the best solution is provide a ignore JSONNULL strategy by moshi lib itself.

[ZacSweers]

@ildar2 to answer your FR in #1000, the best solution is for you to either

• Work with your server folks on sending down more reasonable json. Using null values to indicate absence in a map isn't right since the key is what determines presence
• If the API is external, write a delegating adapter that reads the jsonValue map first and removes all keys with null values before passing it to the delegate.

I don't think it's in our interest to effectively break moshi's behavior to accommodate unscrupulous APIs that conflate absence and nullability.

[condesales]

@ZacSweers
In an ideal world, that would be the rule, but we all know that's not always the case.
The rule I follow in all apps I write is: "never trust the API responses"
So all my fields for API response models are nullable and I add default values in case they're not provided.

[ildar2]

@condesales default values are used only if api responce misses this field. If the field was explicitly null, then your field will be null, regardless of kotlin nullability. But you can work around that with reserialization - null fields get stripped. Here is a passing test, where preLoc?.mobileLoading was initially null, but after reserialization become default value, therefore we can set fields non-null, avoiding nullability issues:

    @Test
    fun `test null field should return default value gson`() {
        var preLoc = Gson().fromJson(
            "{\"mobile.loading\" : null, \"mobile.continue\" : \"Continue\"}",
            PreLocalize::class.java
        )
        assertNull(preLoc?.mobileLoading)
        val strippedOfNulls = Gson().toJson(preLoc)
        preLoc = Gson().fromJson(strippedOfNulls, PreLocalize::class.java)
        assertEquals("Continue", preLoc?.mobileContinue)
        assertEquals("City", preLoc?.commonCity)
        assertEquals("Loading", preLoc?.mobileLoading)
    }

[ZacSweers]

in case they're not provided.

This is the case (absent key) that Moshi's Kotlin support handles. However, null as a value is not such a case, as it's the server explicitly declaring the value.

[leleliu008]

@ZacSweers @condesales What does null mean? Obviously, different people have their own opinions about this question. Those people who want to ignore the null value, maybe they treat the null as a meaningless value. questions do not always have one answer.

[ZacSweers]

It has one answer in Moshi and this is Moshi's opinion :).

[leleliu008]

@ZacSweers a clear answer.

[ZacSweers]

his opinion that we can just ask for reasonable JSON across the entire internet is naive.

That's not what I said, so let me be clear.

Moshi's opinion in Kotlin is that present keys with literal null values are null (and therefore not candidates for default values) while absent keys are absent (and therefore candidates for default values). Moshi's got no opinion on how you do or don't collaborate with whoever is sending you JSON, so please don't take it that way. If you want nulls to indicate absence and can't make the sending API adhere to this, you can do a number of other options too pre-format this json before handing it to moshi.

Here's one that works via intermediary delegating adapter that manually removes nulls from the json blob:

@Retention(RUNTIME)
@Target(CLASS)
annotation class DefaultIfNull

class DefaultIfNullFactory : JsonAdapter.Factory {
  override fun create(type: Type, annotations: MutableSet<out Annotation>,
      moshi: Moshi): JsonAdapter<*>? {
    if (!Types.getRawType(type).isAnnotationPresent(
            DefaultIfNull::class.java)) {
      return null
    }

    val delegate = moshi.nextAdapter<Any>(this, type, annotations)

    return object : JsonAdapter<Any>() {
      override fun fromJson(reader: JsonReader): Any? {
        @Suppress("UNCHECKED_CAST")
        val blob = reader.readJsonValue() as Map<String, Any?>
        val noNulls = blob.filterValues { it != null }
        return delegate.fromJsonValue(noNulls)
      }

      override fun toJson(writer: JsonWriter, value: Any?) {
        return delegate.toJson(writer, value)
      }
    }
  }
}

class NullSkipperTest {

  @DefaultIfNull
  data class ClassWithDefaults(val foo: String, val bar: String? = "defaultBar")

  @Test
  fun skipNulls() {
    //language=JSON
    val json = """{"foo": "fooValue", "bar": null}"""

    val adapter = Moshi.Builder()
        .add(DefaultIfNullFactory())
        .add(KotlinJsonAdapterFactory())
        .build()
        .adapter(ClassWithDefaults::class.java)

    val instance = adapter.fromJson(json)!!
    check(instance.bar == "defaultBar")
  }
}

[djensen47]

Ah thank you for the clarification!

P.S., sorry, didn't mean seem so harsh ... definitely appreciate the great work you all do.

[ColtonIdle]

@ZacSweers this question comes up a lot. Maybe we can add a section under the Kotlin heading in the docs to more thoroughly explain

"It understands Kotlin's non-nullable types and default parameter values"

When you said that Moshi respects explicit nulls sent by the server... This made perfect sense to me. In the perfect world I do indeed want default values only when a key is absent, and the rest of the time... I want to respect null.

Unfortunately I work with a backend that has given us a "guarantee" of omitting nulls, yet every now and then one sneaks in. In this case, I rather ship my release builds with an adapter that will just strip out any nulls so that we don't hit any crashes in production. But I'm okay with omitting a StripNullAdapter in our development stage.

Do you think it'd be worthwhile to call this out in the docs (specifically moshis opinion on this) and potentially provide the strip null adapter for users to quickly find it... Also potentially mentioning that it may be best to only enable in release mode?

Just my opinion since I keep hitting needless crashes because my prod backend keeps breaking the contract.

Edit: just saw you provided an annotation and not an adapter. So my point still stands, but maybe highlight the option to perform this on the adapter level or class level with annotation

Edit 2: wait. Your code is an annotation and an adapter factory? Consider me lost. 😁 Wouldn't you only need one of those?

Edit 3: okay. Reread the code. Looks like your adapter only defaults the values if the annotation is present. Makes sense. I did try to run it and it crashes on the as Map<> line. Will try to fix that though

[ColtonIdle]

@ZacSweers by any chance do you think this modification to your adapter you posted above, is completely awful or terrible for some reason?

Basically I'm in an old legacy app, with like 200+ models, and so I'm looking for a way to touch as little code as possible. I just want nulls from my backend to be removed so that they turn into the default values in kotlin. Hoping to not have to use any annotations or anything like that. Hopefully one adapter to do the trick.

The below seems to work without any real negative effect on perf. Do you think it's completely terrible, or it's probably the best I can get?

class DefaultIfNullFactory : JsonAdapter.Factory {
    override fun create(type: Type, annotations: MutableSet<out Annotation>,
                        moshi: Moshi): JsonAdapter<*>? {
        val delegate = moshi.nextAdapter<Any>(this, type, annotations)
        return object : JsonAdapter<Any>() {
            override fun fromJson(reader: JsonReader): Any? {
                    val blob1 = reader.readJsonValue()
                try {
                    val blob = blob1 as Map<String, Any?>
                    val noNulls = blob.filterValues { it != null }
                    return delegate.fromJsonValue(noNulls)
                } catch (e: Exception) {
                    return delegate.fromJsonValue(blob1)
                }
            }
            override fun toJson(writer: JsonWriter, value: Any?) {
                return delegate.toJson(writer, value)
            }
        }
    }
}

[sam-devstudio]

his opinion that we can just ask for reasonable JSON across the entire internet is naive.

That's not what I said, so let me be clear.

Moshi's opinion in Kotlin is that present keys with literal null values are null (and therefore not candidates for default values) while absent keys are absent (and therefore candidates for default values). Moshi's got no opinion on how you do or don't collaborate with whoever is sending you JSON, so please don't take it that way. If you want nulls to indicate absence and can't make the sending API adhere to this, you can do a number of other options too pre-format this json before handing it to moshi.

Here's one that works via intermediary delegating adapter that manually removes nulls from the json blob:

@Retention(RUNTIME)
@Target(CLASS)
annotation class DefaultIfNull

class DefaultIfNullFactory : JsonAdapter.Factory {
  override fun create(type: Type, annotations: MutableSet<out Annotation>,
      moshi: Moshi): JsonAdapter<*>? {
    if (!Types.getRawType(type).isAnnotationPresent(
            DefaultIfNull::class.java)) {
      return null
    }

    val delegate = moshi.nextAdapter<Any>(this, type, annotations)

    return object : JsonAdapter<Any>() {
      override fun fromJson(reader: JsonReader): Any? {
        @Suppress("UNCHECKED_CAST")
        val blob = reader.readJsonValue() as Map<String, Any?>
        val noNulls = blob.filterValues { it != null }
        return delegate.fromJsonValue(noNulls)
      }

      override fun toJson(writer: JsonWriter, value: Any?) {
        return delegate.toJson(writer, value)
      }
    }
  }
}

class NullSkipperTest {

  @DefaultIfNull
  data class ClassWithDefaults(val foo: String, val bar: String? = "defaultBar")

  @Test
  fun skipNulls() {
    //language=JSON
    val json = """{"foo": "fooValue", "bar": null}"""

    val adapter = Moshi.Builder()
        .add(DefaultIfNullFactory())
        .add(KotlinJsonAdapterFactory())
        .build()
        .adapter(ClassWithDefaults::class.java)

    val instance = adapter.fromJson(json)!!
    check(instance.bar == "defaultBar")
  }
}

@ZacSweers while I use this DefaultIfNull annotation it shows error on field saying
"The annotation is not allowed on target member property with backing field" and if I resolve it by Add Annotation Target option it just starts giving error while api response comes

[ZacSweers]

@saqibmirza2007 that's not enough to help you, and sort of out of scope of this original issue. Should post it on stackoverflow with the exact error messages and sample code if you want help

[sam-devstudio]

@ZacSweers thanks for your reply I have sorted out the issue but can you please tell me that can I use this if I am using retrofit android which automatically parses response.

[ColtonIdle]

@saqibmirza2007 I would recommend not using that factory. I have had major issues in production because of it. My findings in the edit here. https://stackoverflow.com/questions/60377758/moshi-factory-to-ignore-null-values-and-use-kotlin-default-value-when-deserializ

[ZacSweers]

@ColtonIdle your problem isn't really with the factory, seems your problem is with how numbers work in JSON and caring about the intermediate number representations in that map. A downstream adapter will be able to handle those number conversions just fine, I think that post is misleading. Any API using the jsonValue API would have this behavior, and I use it for a number of projects without any issue.

[ColtonIdle]

@ZacSweers I put together a small test that shows my issue. There's a good chance I'm not understanding something correctly or using something correctly, but would love to at least know that for sure.

Here is my factory. Similar to yours except I removed the fact that you need to be opted in via an annotation.

class DefaultIfNullFactory : JsonAdapter.Factory {
    override fun create(type: Type, annotations: MutableSet<out Annotation>,
                        moshi: Moshi): JsonAdapter<*>? {
        val delegate = moshi.nextAdapter<Any>(this, type, annotations)
        return object : JsonAdapter<Any>() {
            override fun fromJson(reader: JsonReader): Any? {
                val blob1 = reader.readJsonValue()
                try {
                    val blob = blob1 as Map<String, Any?>
                    val noNulls = blob.filterValues { it != null }
                    return delegate.fromJsonValue(noNulls)
                } catch (e: Exception) {
                    return delegate.fromJsonValue(blob1)
                }
            }
            override fun toJson(writer: JsonWriter, value: Any?) {
                return delegate.toJson(writer, value)
            }
        }
    }
}

Here are my 3 test cases

class NullFactoryTestTest {

    data class ClassWithDefaults(val foo: String, val bar: String? = "defaultBar")

    /**
     * This test passes.
     */
    @Test
    fun `default value for null`() {
        //language=JSON
        val json = """{"foo": "fooValue", "bar": null}"""

        val adapter = Moshi.Builder()
            .add(DefaultIfNullFactory())
            .add(KotlinJsonAdapterFactory())
            .build()
            .adapter(ClassWithDefaults::class.java)

        val instance = adapter.fromJson(json)!!
        check(instance.bar == "defaultBar")
    }

    /**
     * This test fails with:
     * java.lang.IllegalStateException: 1.5836742E12 was not "1583674200000"
     */

    //I only want a string here because this is what I had when I used gson,
    // and so we want to keep the same types here as we migrated from gson to moshi
    @Test
    fun `long from backend but I want a String`() {
        //language=JSON
        val json = """{"foo": "fooValue", "bar": 1583674200000}"""

        val adapter = Moshi.Builder()
            .add(DefaultIfNullFactory())
            .add(KotlinJsonAdapterFactory())
            .build()
            .adapter(ClassWithDefaults::class.java)

        val instance = adapter.fromJson(json)!!
        check(instance.bar == "1583674200000", { """${instance.bar} was not "1583674200000" """ })
    }

    /**
     * This test passes
     */
    @Test
    fun `long from backend but I want a String but I removed the remove null factory`() {
        //language=JSON
        val json = """{"foo": "fooValue", "bar": 1583674200000}"""

        val adapter = Moshi.Builder()
//            .add(DefaultIfNullFactory())
            .add(KotlinJsonAdapterFactory())
            .build()
            .adapter(ClassWithDefaults::class.java)

        val instance = adapter.fromJson(json)!!
        check(instance.bar == "1583674200000")
    }
}

If it helps, I can just put up a sample repo so theres no miscommunication here.

As you can see, in the second test that fails, my server sends me a long timestamp for a birthday and my class that this gets adapted to has the timestamp defined as a String.

This works totally fine with no DefaultIfNullFactory(). I get the String on the other side with an untouched value. Hoorah!

If I add in the DefaultIfNullFactory(), then I still get a String, but it's been modified out from under me.

Can you see where my confusion comes from or am I going insane? 😄 I have a test that passes (and that means my android code is working), but then I added the DefaultIfNullfactory, and now my android app doesn't work because it's not expecting scientific notation in my string.

I also updated my SO question to say that I may not be right in my verdict of it not working.

[sam-devstudio]

@ColtonIdle Thanks for your reply. I am using it with retrofit and it was not working well maybe I have missed something.
Now I am thinking to use Moshi 1.9+ as it treats kotlin classes separately so might be issue gets fixed but I am still unsure about that as environment did'nt set up well util now

[uchhabra3]

I have made some changes

https://github.com/uchhabra3/Moshi-Issue-843-Solution

If anyone want to check and comment
Side effects, bugs etc

[star-andy]

I have made some changes

https://github.com/uchhabra3/Moshi-Issue-843-Solution

If anyone want to check and comment
Side effects, bugs etc

@uchhabra3 use in moshi 1.11.0 not work

class MoshiCreate {
    fun getMoshi(): Moshi {
        return Moshi.Builder()
            .add(MyKotlinJsonAdapterFactory())
            .add(MyStandardJsonAdapters.FACTORY)
            .addLast(KotlinJsonAdapterFactory())
            .build()
    }
}

[star-andy]

his opinion that we can just ask for reasonable JSON across the entire internet is naive.

That's not what I said, so let me be clear.

Moshi's opinion in Kotlin is that present keys with literal null values are null (and therefore not candidates for default values) while absent keys are absent (and therefore candidates for default values). Moshi's got no opinion on how you do or don't collaborate with whoever is sending you JSON, so please don't take it that way. If you want nulls to indicate absence and can't make the sending API adhere to this, you can do a number of other options too pre-format this json before handing it to moshi.

Here's one that works via intermediary delegating adapter that manually removes nulls from the json blob:

@retention(RUNTIME)
@target(CLASS)
annotation class DefaultIfNull

class DefaultIfNullFactory : JsonAdapter.Factory {
override fun create(type: Type, annotations: MutableSet,
moshi: Moshi): JsonAdapter<*>? {
if (!Types.getRawType(type).isAnnotationPresent(
DefaultIfNull::class.java)) {
return null
}

val delegate = moshi.nextAdapter<Any>(this, type, annotations)

return object : JsonAdapter<Any>() {
  override fun fromJson(reader: JsonReader): Any? {
    @Suppress("UNCHECKED_CAST")
    val blob = reader.readJsonValue() as Map<String, Any?>
    val noNulls = blob.filterValues { it != null }
    return delegate.fromJsonValue(noNulls)
  }

  override fun toJson(writer: JsonWriter, value: Any?) {
    return delegate.toJson(writer, value)
  }
}

}
}

class NullSkipperTest {

@DefaultIfNull
data class ClassWithDefaults(val foo: String, val bar: String? = "defaultBar")

@test
fun skipNulls() {
//language=JSON
val json = """{"foo": "fooValue", "bar": null}"""

val adapter = Moshi.Builder()
    .add(DefaultIfNullFactory())
    .add(KotlinJsonAdapterFactory())
    .build()
    .adapter(ClassWithDefaults::class.java)

val instance = adapter.fromJson(json)!!
check(instance.bar == "defaultBar")

}
}

@ZacSweers use in moshi 1.11.0 not work

class MoshiCreate {
    fun getMoshi(): Moshi {
        return Moshi.Builder()
            .add(DefaultIfNullFactory())
            .build()
    }
}

OR

class MoshiCreate {
    fun getMoshi(): Moshi {
        return Moshi.Builder()
            .add(DefaultIfNullFactory())
            .addLast(KotlinJsonAdapterFactory())
            .build()
    }
}

Neither of the above methods will work

[morder]

this is my working and fast solution

package com.squareup.moshi

class DefaultIfNullFactory : JsonAdapter.Factory {
    override fun create(
        type: Type,
        annotations: MutableSet<out Annotation>,
        moshi: Moshi
    ): JsonAdapter<*>? {

        val delegate = moshi.nextAdapter<Any>(this, type, annotations)

        if (!annotations.isEmpty()) return null
        if (type === Boolean::class.javaPrimitiveType) return delegate
        if (type === Byte::class.javaPrimitiveType) return delegate
        if (type === Char::class.javaPrimitiveType) return delegate
        if (type === Double::class.javaPrimitiveType) return delegate
        if (type === Float::class.javaPrimitiveType) return delegate
        if (type === Int::class.javaPrimitiveType) return delegate
        if (type === Long::class.javaPrimitiveType) return delegate
        if (type === Short::class.javaPrimitiveType) return delegate
        if (type === Boolean::class.java) return delegate
        if (type === Byte::class.java) return delegate
        if (type === Char::class.java) return delegate
        if (type === Double::class.java) return delegate
        if (type === Float::class.java) return delegate
        if (type === Int::class.java) return delegate
        if (type === Long::class.java) return delegate
        if (type === Short::class.java) return delegate
        if (type === String::class.java) return delegate

        return object : JsonAdapter<Any>() {

            override fun fromJson(reader: JsonReader): Any? {
                return if (reader.peek() == JsonReader.Token.BEGIN_OBJECT) {
                    delegate.fromJson(JsonReaderSkipNullValuesWrapper(reader))
                } else {
                    delegate.fromJson(reader)
                }
            }

            override fun toJson(writer: JsonWriter, value: Any?) {
                return delegate.toJson(writer, value)
            }
        }
    }
}

class JsonReaderSkipNullValuesWrapper(
    private val wrapped: JsonReader
) : JsonReader() {

    private var ignoreSkipName = AtomicBoolean(false)
    private var ignoreSkipValue = AtomicBoolean(false)

    override fun close() {
        wrapped.close()
    }

    override fun beginArray() {
        wrapped.beginArray()
    }

    override fun endArray() {
        wrapped.endArray()
    }

    override fun beginObject() {
        wrapped.beginObject()
    }

    override fun endObject() {
        wrapped.endObject()
        ignoreSkipName.compareAndSet(true, false)
        ignoreSkipValue.compareAndSet(true, false)
    }

    override fun hasNext(): Boolean {
        return wrapped.hasNext()
    }

    override fun peek(): Token {
        return wrapped.peek()
    }

    override fun nextName(): String {
        return wrapped.nextName()
    }

    override fun selectName(options: Options): Int {
        val index = wrapped.selectName(options)
        return if (index >= 0 && wrapped.peek() == Token.NULL) {
            wrapped.skipValue()
            ignoreSkipName.set(true)
            ignoreSkipValue.set(true)
            -1
        } else {
            index
        }
    }

    override fun skipName() {
        if (ignoreSkipName.compareAndSet(true, false)) {
            return
        }
        wrapped.skipName()
    }

    override fun nextString(): String {
        return wrapped.nextString()
    }

    override fun selectString(options: Options): Int {
        return wrapped.selectString(options)
    }

    override fun nextBoolean(): Boolean {
        return wrapped.nextBoolean()
    }

    override fun <T : Any?> nextNull(): T? {
        return wrapped.nextNull()
    }

    override fun nextDouble(): Double {
        return wrapped.nextDouble()
    }

    override fun nextLong(): Long {
        return wrapped.nextLong()
    }

    override fun nextInt(): Int {
        return wrapped.nextInt()
    }

    override fun nextSource(): BufferedSource {
        return wrapped.nextSource()
    }

    override fun skipValue() {
        if (ignoreSkipValue.compareAndSet(true, false)) {
            return
        }
        wrapped.skipValue()
    }

    override fun peekJson(): JsonReader {
        return wrapped.peekJson()
    }

    override fun promoteNameToValue() {
        wrapped.promoteNameToValue()
    }
}

[venator85]

@morder your solution seemed the best to me but it doesn't compile in moshi 1.11.0 since the JsonReader constructor is package private.

[morder]

@morder your solution seemed the best to me but it doesn't compile in moshi 1.11.0 since the JsonReader constructor is package private.

yep, change the package name to com.squareup.moshi
please look at the first line of my code

[ArcherEmiya05]

Check my answer here and see if it will help you

[nordfalk]

Here is what I'll use, based on @ColtonIdle 's suggestion:

class DefaultIfNullFactory : JsonAdapter.Factory {
    override fun create(type: Type, annotations: MutableSet<out Annotation>, moshi: Moshi): JsonAdapter<*>? {
        val delegate = moshi.nextAdapter<Any>(this, type, annotations)
        return object : JsonAdapter<Any>() {
            override fun fromJson(reader: JsonReader): Any? {
                val blob = reader.readJsonValue()
                if (blob is Map<*, *>) {
                    val noNulls = blob.filterValues { it != null }
                    return delegate.fromJsonValue(noNulls)
                }
                return delegate.fromJsonValue(blob)
            }
            override fun toJson(writer: JsonWriter, value: Any?) {
                return delegate.toJson(writer, value)
            }
        }
    }
}

Perhaps its not as fast as @morder but speed isnt a big issue in my use case.