How to deserialize collections without type information

[kentcb]

I'm using built_value for JSON serialization in the context of integrating with an API. I'm struggling a bit because built_value seems to require type information in the incoming JSON stream, which it won't get from the API. I think I've solved this for cases where I'm deserializing a DTO, but I've hit a snag with respect to APIs that returns collections.

For example, the API might return this JSON:

[
    {
        "id": 42,
        "name": "Kent"
    },
    {
        "id": 13,
        "name": "Belinda"
    }
]

And I need to parse that into a BuiltList<UserDto> (where UserDto is just a built value with id and name).

Now I thought the following would suffice to parse this:

var json = ...;
final parameters = <FullType>[
  new FullType(UserDto.serializer.types.first),
];
final fullType = new FullType(BuiltList, parameters);
final deserialized = serializers.deserialize(convert.json.decode(json), specifiedType: fullType);

However, this gives me:

Bad state: No builder for BuiltList<dynamic><UserDto>.

What am I doing wrong here?

[kentcb]

OK, I figured out the main problem. My generated serializers collection did not include the following:

..addBuilderFactory(const FullType(BuiltList, const [const FullType(UserDto)]), () => new ListBuilder<UserDto>())

I guess this is because my DTOs don't actually contain BuiltList<UserDto> anywhere. Instead, it is only my API endpoint that expects it (i.e. one of my APIs returns a list of users). My plan is to manually add it for now, unless there's a better solution?

I also need to figure out how to dynamically determine a FullType given only a generic type, T. 🤔

[kentcb]

Even though it's not as clean a design for my APIs, I'm leaning heavily towards using serializeWith / deserializeWith, since I don't want the type on the wire. However, I still can't figure out a simple way to just say "deserialize this JSON using a BuiltList<UserDto> serializer". I can't figure out how to obtain the BuiltList serializer. Heck, the IDE doesn't even seem to recognize that it exists in package:built_value/serializer.dart 🤔

[zoechi]

What about

new ListBuilder<UserDto>(json.map(
    (user) => serializers.deserialize(convert.json.decode(json), specifiedType: const FullType(UserDto))
)).build();

?

[kentcb]

I seem to be going around in circles with this. In the end, this is the best I've been able to come up with that seemingly solves the problem:

/// Serializes [object] to JSON.
String serialize(Object object, FullType fullType) {
  // TODO: it might be that we *do* want to allow this, but I have no use case as yet.
  assert(object != null, "Unable to serialize null.");
  assert(fullType != null);

  return json.encode(serializers.serialize(object, specifiedType: fullType));
}

/// Deserializes [json] into a [T].
T deserialize<T>(String json, FullType fullType) {
  assert(json != null);
  assert(fullType != null);

  final result = serializers.deserialize(convert.json.decode(json), specifiedType: fullType);

  if (result is T) {
    return result;
  } else {
    final error = "Result of deserialization is of type ${result?.runtimeType}, not the expected type $T.";
    throw new Exception(error);
  }
}

I had really hoped that the FullType could be inferred from the generic parameter T, but it seems like Dart's type system isn't strong enough to pull this off (therefore I made serialize take Object instead of T, since the latter adds no value here).

Are there any suggestions on how this could be improved?

[zoechi]

I had really hoped that the FullType could be inferred from the generic parameter T, but it seems like

I think this is the whole purpose why FullType exists in the first place, otherwise built_value would only take a generic type as parameter.
This might not be necessary anymore when --preview-dart-2 is default everywhere (not sure, didn't have a closer look into that topic myself yet)

[kentcb]

I think this is the whole purpose why FullType exists in the first place

Yeah, that's the same conclusion I came to - guess I just hoped maybe I was wrong. Oh well, I think passing in FullType is far easier than expecting the caller to resolve the correct serializer per your first comment (which I only just saw now, sorry). Not ideal, but at least I can explain why it has to be this way in my docs.

I will leave this open for a couple of days in case there is any further feedback. Thanks!

[zoechi]

Sure, @davidmorgan will be able to provide more fundamental feedback for sure

[davidmorgan]

Hmm, in your code snippet here

#438 (comment)

what class are the serialize and deserialize methods on? I'm trying to figure out what you want at a high level :)

Generally, Dart has limited support for runtime manipulation of types, unless you're in the VM. This is so that dart2js/flutter can throw away most of the type information; it forces you to use codegen to do some things.

[kentcb]

@davidmorgan yeah, sorry - I forgot some context. I'm writing a mixin that I can use from any class that wants to integrate with back-end APIs (I'm using a vertical architecture so the API integration for each feature sits in a file within each feature's folder). So the full code is:

/// A mixin that provides JSON (de)serialization support.
abstract class SerializationMixin {
  /// Serializes [object] to JSON.
  ///
  /// [fullType] must fully describe the type being serialized. Due to limitations in Dart's
  /// type system, it cannot be inferred from a generic type parameter `T`.
  ///
  /// More information at https://github.com/google/built_value.dart/issues/438.
  String serialize(Object object, FullType fullType) {
    // TODO: it might be that we *do* want to allow this, but I have no use case as yet.
    assert(object != null, "Unable to serialize null.");
    assert(fullType != null);

    return json.encode(serializers.serialize(object, specifiedType: fullType));
  }

  /// Deserializes [json] into a [T].
  ///
  /// [fullType] must fully describe the type being serialized. Due to limitations in Dart's
  /// type system, it cannot be inferred from the generic type parameter [T].
  ///
  /// More information at https://github.com/google/built_value.dart/issues/438.
  T deserialize<T>(String json, FullType fullType) {
    assert(json != null);
    assert(fullType != null);

    final result = serializers.deserialize(convert.json.decode(json), specifiedType: fullType);

    if (result is T) {
      return result;
    } else {
      final error = "Result of deserialization is of type ${result?.runtimeType}, not the expected type $T.";
      throw new Exception(error);
    }
  }
}

Then a particular API integration would look something like:

class Login extends TheApiInterface with SerializationMixin, HttpMixin {
  @override
  Future<UserDto> authenticate(String name, String password) async {
    final body = (new AuthenticateRequestDtoBuilder()
          ..name = name
          ..password = password)
        .build();
    final response = await postAndVerifySuccess("/authenticate", body: body);
    return deserialize<UserDto>(response.body, const FullType(UserDto));
  }
}

I was hoping that final invocation could be simplified to:

return deserialize<UserDto>(response.body);

But I just couldn't find a way to make it work without "doubling up" on the type information.

[dave26199]

I see. There is one feature that may help you, serializers.deserializeWith.

You would write e.g.

serializers.deserializeWith(UserDto.serializer, response.body)

this works by taking both T and FullType from UserDto.serializer. You can also do some variation on this yourself if you like:

https://github.com/google/built_value.dart/blob/master/built_value/lib/src/built_json_serializers.dart#L32

[kentcb]

@dave26199 thanks, but per this comment above I wasn't able to figure out how to use serializers in scenarios where I wanted to deserialize a BuiltList<SomeDto>. Further, I couldn't figure out how to obtain the serializer from only the T in generic code.

  T deserialize<T>(String json) {
    assert(json != null);

    // how would I do this?
    final serializer = ???;
    return serializers.deserializeWith(serializer, json) as T;
  }

Or are you suggesting to do away with the generic helper altogether?

[davidmorgan]

Hmm, you're right, that doesn't quite work right now. I was thinking you might be able to create a Serializer instance for each DTO, e.g. a SomeDtoBuiltListSerializer analogous to this:

https://github.com/google/built_value.dart/blob/master/built_value/lib/src/built_list_serializer.dart

...which would effectively provide a way to bundle the type literal BuiltList<SomeDto> and FullType(BuiltList, [SomeDto]) together. But this doesn't work because the way Serializer specifies the FullType is to via just a Type object, which isn't enough.

What we actually need is a new class, RealFullType, which has both the generic type and the full type. i.e.

var userDtoList = new RealFullType<BuiltList<SomeDto>>(new FullType(BuiltList, [new FullType(SomeDto)])))

then we can use this variable to do what you want: we can pass it in and both 1) extract T, 2) get the FullType so we can find the serializer.

It should be possible to try creating such a class in your code; you'd then have a library of constants and pick the right one when you need it. Then you would have

return deserialize(userDtoList, response.body);

How does that sound?