Design and open discussion around better Dart serialization experiences.
This is not an official Google or Dart design or process is primarily to experiment and collect feedback from both internal and external stakeholders and users. There is a chance this might result in no changes, or changes could occur at some future point in time.
Please read CONTRIBUTING.md before opening a PR
Table of contents
Dart is toted as a modern object-oriented client-side programming language (with some but more limited usage in standalone tools and servers), but lacks a well defined story for serialization - something that is paramount to adoption and healthy developer ergonomics.
With the introduction of Flutter, Dart users have the ability to produce high-fidelity iOS and Android applications, but have also run into the issue of having a typed object model that serializes to/from data formats like JSON.
A prior review of options in Dart that is a bit of out date is available on the Dart website as an article: "Serialization in Dart". It's a good place to start if you don't know much about the subject.
Not surprisingly, users are effectively using Dart with JSON and other formats like protocol buffers, today, both internally and externally, but often with some sort of downside.
The simplest option - just don't use types. This often produces the lowest overhead in terms of both runtime and code-size, and can even scale OK for small projects or in cases where the serialization format is fluid.
import 'dart:async';
import 'dart:convert';
import 'package:http/http.dart' as http;
Future<Map<String, dynamic>>> fetchAccount(int id) async {
var response = await http.get('account/$id');
return JSON.decode(response.body) as Map<String, dynamic>;
}main() async {
var account = await fetchAccount(101);
print('Account #101 is for: ${account['name']'});
}- Batteries included: you rarely have to go beyond the core libraries.
- Works equally well in all platforms (browser/server/mobile).
- Produces the lowest overhead in terms of both code-size and runtime.
- Toolability: Good luck renaming
account['name']toaccount['first_name']. - Can't statically validate against a schema or format.
- Doesn't scale well to large teams.
- Exposes a mutable and iterable data structure for everything.
- Small teams or single developers/prototypers.
- Applications with unstructured data (so
MaporListis actually OK).
Dart's runtime reflection library (dart:mirrors) can read from and
write to structured classes at runtime, and use metadata like annotations for
developers to be able to add extra information.
import 'dart:convert';
import 'dart:mirrors';
class Account {
final int id;
Account({this.id});
}
class Serializer<T> {
const Serializer<T>();
T decode(String json) {
var type = reflectType(T);
var constructor = type.constructors.first;
var parameters = constructor.namedParameters;
return type.newInstance(
namedParameters: JSON.decode(json),
);
}
}
main() {
var account = const Serializer<Account>().decode(r'''
{
"id": 101
}
''');
print('Account #101 is for: ${account.id'});
}- Batteries mostly included: trivial to write a simple serialization library.
- Potential to remove mirrors usage using source code transformation.
- Arbitrary requirements on classes (public constructor, etc).
- Serious platform issues: disabled for Flutter, unsuable code size on the web.
- Runtime performance suffers compared to statically typed code.
- Difficult to debug: reflection-based systems harder to reason about.
- Source code transformation is mostly terrible for good build systems.
Of course, hand-written code and classes can do precisely what you want. For small projects or systems that don't change often (or for precisely optmizing for your business requirements) this might make the most sense:
import 'dart:convert';
class Account {
final int id;
Account({this.id});
}
class AccountSerializer {
const AccountSerializer();
Account decode(String json) {
var map = JSON.decode(json) as Map<String, dynamic>;
return new Account(id: map['id']);
}
}
main() {
var account = const AccountSerializer().decode(r'''
{
"id": 101
}
''');
print('Account #101 is for: ${account.id'});
}- You get exactly the behavior and code you want.
- Any medium-sized+ data model is going to be time consuming/error prone.
- As the data model changes must change both class and serializer.
- Hard to ever support other data formats with writing even more code.
- Dart feels like it has platform/language issues to those who read the code.
- Too many to count :)
A time-tested option, simply generate Dart code either ahead-of-time or during the development process from another data source, such as a schema, configuration file, or even Dart source code (static source analysis).
Most internal users at Google use this strategy (in some form or another, though the most common is protocol buffers) - but this also relies on fact we have bazel as a standard build system.
import 'package:my_json_generator/my_json_generator.dart';
part 'main.g.dart';
class Account {
final int id;
Account({this.id});
}
@generate
abstract class AccountSerializer {
const factory AccountSerializer() = AccountSerializer$Generated;
Account decode(String json);
}
main() {
var account = const AccountSerializer).decode(r'''
{
"id": 101
}
''');
print('Account #101 is for: ${account.id'});
}- Nicer ergonomics compared to hand writing (once build system in place).
- Possible to get very close to (depending on requirements) hand-written code.
- Getting more popular in web community with introduction of CLIs.
- Difficulty of designing the "perfect system" (definition varies).
- Static analysis errors: until you generate
main.g.dart, at least. - Dart lacks a complete standard build system that works equally well for all.
- For frameworks like Flutter that are "batteries included", this falls short.
A recent alternative, made possible with the javascript interop library, is to use anonymous javascript objects which can be strongly typed and work well with code completion.
It requires direct access to the native javascript serialization functions
@JS()
library serialise.interop;
import 'package:js/js.dart';
@JS('JSON.parse')
external dynamic fromJson(String text);
@JS('JSON.stringify')
external String toJson(dynamic object);@JS()
@anonymous
class Account {
external int get id;
external set id(int value);
external factory Simple({ int id });
}main() {
Account account = fromJson(r'''
{
"id": 101
}
''');
print('Account #101 is for: ${account.id'});
}Examples and a performance comparison
- No reliance on mirrors or source transformation.
- Compiles to less javascript than a hand-written class.
- Very fast since it uses native browser functions.
- Only works in the browser and therefore only applicable to client-side applications.
- The interop package only supports properties and not fields so the class definition is a little more verbose.
- Small teams or single developers/prototypers.
Who (entities or individuals) are effected or have a business need in this proposal. Note that being a user of serialization is likely not enough to be considered a stakeholder - though the goal of this document is to solicit feedback from indidiual users.
NOTE: The following list is entirely assumptive at this moment:
TBD
TBD
TBD
TBD
NOTE: Languages with dynamic typing without any form of static analysis (i.e. JavaScript, Ruby, Python) are excluded - in-that often the platform's built-in serialization is enough to avoid needing anything else.
It's also not that interesting to compare against - and optimizers like Google's closure often have different requirements than the language itself.
TypeScript has a structural type system that is easy to overlay ontop of
(unstructed) formats like JSON. As an example we can pretend that a received
JSON blob representing a User has static types:
interface User {
name: string;
age: number;
created: Date;
}
fetchById(id: int): Promise<User> {
return http.get('/users/${id}').map((response) => JSON.parse(response));
}
function run() {
fetchById(101).then((user) => {
console.log('Hello! ${user.name} is ${user.age} year(s) old.');
});
}If decorators are introduced (see proposal) then a lightweight macro-like syntax will exist as well to generate boilerplate. As an example:
@serializable()
class User {
constructor(name: string) {
this._name = name;
}
private _name: string;
@serialize()
get name() {
return this._name;
}
}
function run() {
const p = new Person('André');
console.log(JSON.stringify(p));
}Using the Gloss library you get some
helper functions and syntax, but nothing too magical.
import Gloss
struct RepoOwner: Decodable {
let ownerId: Int?
let username: String?
// MARK: - Deserialization
init?(json: JSON) {
self.ownerId = "id" <~~ json
self.username = "login" <~~ json
}
}Or for translating to JSON:
import Gloss
struct RepoOwner: Glossy {
let ownerId: Int?
let username: String?
// MARK: - Deserialization
// ...
// MARK: - Serialization
func toJSON() -> JSON? {
return jsonify([
"id" ~~> self.ownerId,
"login" ~~> self.username
])
}
}An example of the Kotson library:
import com.github.salomonbrys.kotson.*
val gson = GsonBuilder().registerTypeAdapter<Person>(personSerializer).create()import com.github.salomonbrys.kotson.*
val gson = Gson()
// java: List<User> list = gson.fromJson(src, new TypeToken<List<User>>(){}.getType());
val list1 = gson.fromJson<List<User>>(jsonString)
val list2 = gson.fromJson<List<User>>(jsonElement)
val list3 = gson.fromJson<List<User>>(jsonReader)
val list4 = gson.fromJson<List<User>>(reader)An example of the ServiceStack library:
(You can try this example live in your browser)
using System.Linq;
using ServiceStack;
using ServiceStack.Text;
public class GithubRepository
{
public string Name { get; set; }
public string Description { get; set; }
public string Url { get; set; }
public string Homepage { get; set; }
public string Language { get; set; }
public int Watchers { get; set; }
public int Forks { get; set; }
public override string ToString() => Name;
}
var orgName = "ServiceStack";
var orgRepos = $"https://api.github.com/orgs/{orgName}/repos"
.GetJsonFromUrl(httpReq => httpReq.UserAgent = "Gistlyn")
.FromJson<GithubRepository[]>()
.OrderByDescending(x => x.Watchers)
.Take(5)
.ToList();
"Top 5 {0} Github Repositories:".Print(orgName);
orgRepos.PrintDump();
// Save a copy of this *public* Gist by clicking the "Save As" below TODO.
Go comes with the encoding/json package in the standard library which allows serialization and deserialization of data types.
Here's an example:
(you can try this example live in your browser)
package main
import (
"encoding/json"
"fmt"
)
const personJSON = `{
"Name" : "John Doe",
"Age" : 50
}`
func main() {
// deserialize a JSON string to a 'person' type
deserializedPerson := deserializePerson(personJSON)
fmt.Printf("Deserialized: %+v\n\n", deserializedPerson)
// serialize a 'person' type to a JSON string
serializedPerson := serializePerson(deserializedPerson)
fmt.Printf("Serialized: %s", serializedPerson)
}
type person struct {
Name string
Age int
}
func deserializePerson(s string) person {
var p person
json.Unmarshal([]byte(s), &p)
return p
}
func serializePerson(p person) string {
b, _ := json.Marshal(p)
return string(b)
}Below are partial, theoritical, and non-exhaustive potential solutions to help make Dart's serialization and JSON story better. Nothing has been agreed on - and we're open to other ideas. Likely the "solution" will involve many things
- not just one.
Users write code like this:
class User {
final int id;
User({this.id});
}And can invoke User.new (and named parameters using a JSON/Map):
main() {
new User(~{'id': 5});
}While this doesn't strictly fix the JSON/serialization issues, it does make an abstract class/interface be a more implicitly useful data model. For example:
main() {
var user = new User {id: 5};
}struct User {
int id;
}And perhaps it could participate in a structural type system where something
like a Map or JsObject (for web users) could be "cast" (represented as)
this struct:
main() {
var json = {id: 5} as User;
print(json.id);
}typedef Json = String | num | List<Json> | Map<Json, Json> | Null;TODO
TODO
I.e. something that can be better optimized/tree-shaken across platforms.
A seamless code generation story, perhaps with better analyzer integration, incremental builds, and IDE-awareness would allow the least amount of changes to the language. We would likely still need a canonical serialization library.
TODO
TODO