Project goals

[phpnode]

Hey, this is a cool project! I've been working on something really similar - I even called it flow-runtime until I saw you'd grabbed the name :). I was wondering what your goals for the project are? Do you intend to support all of flow's features such as type parameters, object indexers, functions etc or will you keep to a smaller subset? I think there's tremendous value in having a canonical way to represent types in JS, the ecosystem really needs a project like this.

[phpnode]

Just pushed my very work in progress thing here in case you're interested - https://github.com/codemix/runtime-types

[gcanti]

will you keep to a smaller subset?

Hi,

Yes, the goal of this library is to be complementary to Flow, so it will be a small subset:

• main goal: validate the IO boundary (for example API payloads)
• replace tcomb + tcomb-validation
• allow for runtime type intrspection
• extract static types from runtime types (when hopefully Flow does not detect errors on Object #11 is fixed)

// example of type extraction
import * as t from 'flow-runtime'
import type { TypeOf } from 'flow-runtime'

const Person = t.object({
  name: t.string,
  age: t.number
})

 // this is equivalent to
 // type PersonT = { name: string, age: number };
 type PersonT = TypeOf<typeof Person>;

[phpnode]

The ability to validate runtime declared types statically is very nice, but do you foresee people using the DSL directly or will you offer a babel plugin as with tcomb? My goals are very similar, but I want to achieve full compatibility with flow because then the system is capable of representing and validating any possible JS value. I'm about 85% done with my project but it would be nice if we could combine them or collaborate in some way because they're essentially the same thing to outside observers, would you consider expanding the scope to aim for full flow compat?

[gcanti]

but do you foresee people using the DSL directly or will you offer a babel plugin as with tcomb

If #15 will not be fixed, I think that the only solution in order to by DRY is to provide a babel plugin (but I'd prefer not to if possible).

I want to achieve full compatibility with flow because then the system is capable of representing and validating any possible JS value

This is interesting, what are the values that Flow is not able to validate so you need runtime type checking (beyond the IO boundary)?

On the other hand static type checking is strictly more powerful than RTC for some tasks: checking function input / output and phantom types to name a few

[phpnode]

It's not that flow is lacking features, it's that I want to be able to use the type definitions that I'm writing anyway for other purposes - input validation, serialization, reflection etc and in order to do that I need to support everything flow supports (which isn't that much more than you already have here).

On the other hand static type checking is strictly more powerful than RTC for some tasks: checking function input / output and phantom types to name a few

I do agree, but by the same token, runtime checking is strictly more powerful in other cases. They're complementary as you say.

It's also very nice to be able to add constraints to types in a way that flow does not allow, e.g. you could define a type called uint32 which only permits numbers of that size, or an EmailAddress type that only accepts valid email addresses. Flow can't do that but it can be incredibly useful.

The other major use case is when working with a codebase that is not all written with flow - every time control transfers into the legacy code you lose type safety, it's hard to verify that the type definitions you're writing for code that interacts with legacy are actually correct etc. If you can accurately assert types at runtime then that really makes it a lot easier to introduce flow to an existing code base.

More than anything, I think it would be incredibly valuable to have a canonical way to represent types in JS at runtime, and to be canonical it has to be able to handle any kind of possible value.

[phpnode]

If you're interested in collaborating one way forward would be for me to port the babel plugin to target this library, at the moment it produces output that's very similar, right down to the lib name:

  type Demo = {
    (foo: string): string;
    (bar: boolean): boolean;
    foo: string;
    bar: number;
    baz: number | string;
    [key: string]: number;
    [index: number]: boolean;
  };

compiles to

  import t from "flow-runtime";
  const Demo = t.type("Demo", t.object(
    t.callProperty(t.function(
      t.param("foo", t.string()),
      t.return(t.string())
    )),
    t.callProperty(t.function(
      t.param("bar", t.boolean()),
      t.return(t.boolean())
    )),
    t.property("foo", t.string()),
    t.property("bar", t.number()),
    t.property("baz", t.union(t.number(), t.string())),
    t.indexer("key", t.string(), t.number()),
    t.indexer("index", t.number(), t.boolean())
  ));

[gcanti]

I want to be able to use the type definitions that I'm writing anyway for other purposes

I've always been obsessed by this idea, that's why I'm so interested in talking about this topic. I'm afraid that static types and runtime types are "different languages" though. Let me explain and show some problems I faced in the recent months (with babel-plugin-tcomb).

1. First of all, a reliable conversion from static types to runtime types is hard, perhaps not even possible

import type { SomeType } from 'external-library'

type MyType = {
  some: SomeType
};

How can I convert SomeType to a runtime type? Should I parse the external-library libdef? Recursively? (maybe external-library imports another libdef). What if SomeType is a function type? Type checking a function at runtime seems useless if you are a Flow user. What if is parametric?

type MyType<A> = {
  some: SomeType<A>
};

What if SomeType is a class but you have not access to the constructor?

(...I'll think about it, and try to write down the other issues in the next few days)

[gcanti]

Also /cc @minedeljkovic as you are a "power user" of babel-plugin-tcomb:

we already have to use workarounds for some scenarios that I believe would not be needed with flow-runtime

[phpnode]

I've always been obsessed by this idea, that's why I'm so interested in talking about this topic

I'm glad it's not just me, this is exciting!

How can I convert SomeType to a runtime type? Should I parse the external-library libdef?

Probably not in v0.0.1 but sure, in future that's a possible strategy. Another option would be to try and infer the type but you obviously have to be careful about not producing something too narrow.

If you declare a function or class with type annotations then we could also associate the type with the value itself, e.g.

function foo (): string { return "bar"; }

could compile to (without checks):

function foo () { return "bar"; }
foo[Symbol.typeDefinition] = t.function(t.return(t.string());

So we could also look for this symbol after importing. If we have no type information we'd just fall back to any, as Flow does.

You are absolutely right that there will be gaps in such a system, but if flow can cover 99% of the cases statically, and we can cover 99% of cases at runtime then the odd cases where we have to allow any do not negate the overall benefits.

[phpnode]

The other option for the scenario you mentioned above is to just rely on flow itself when we don't know enough about an imported type. We can use type-at-pos for this and cache the results in a script that can be loaded at runtime. Similarly for type definitions we can just run flow ast ./flow-typed/* and cache those too. This would allow us to cover cases like Generator or TypedArray which aren't accessible to JS. In pursuit of this I pushed something that can parse flow config files here: https://github.com/codemix/runtime-types/tree/master/src/flowConfigParser

[gcanti]

I think that the only solution in order to by DRY is to provide a babel plugin

Another option is to write a bunch of transformers to / from a JSON representation, but again there are many technical issues: after the experience with babel-plugin-tcomb I'm starting to think that an half backed solution is worse than no solution at all.

Probably not in v0.0.1 but sure, in future that's a possible strategy. Another option would be to try and infer the type but you obviously have to be careful about not producing something too narrow

@phpnode this seems definitely out of scope of this library, when I say that they are complementary I think of the mathematical meaning: the intersection between runtime and static type checking should be as small as possible. There are 3 things Flow can't do

• validate the IO boundary
• runtime type introspection
• refinements (*)

(*) actually you can do refinements with Flow, is just a bit awkward

All those 3 things are currently covered by this lightweight library.

The only problem I see is how to be DRY: you need a static type for Flow and a runtime type for IO validation, but in general I can't think of an automatic process that translates from one to another, besides the banal one

type Person = {
  name: string,
  surname: ?string
};

to

const Person = t.object({
  name: t.string,
  surname: t.maybe(t.string)
})

What if Person has a type parameter?

type Person<A> = {
  name: string,
  surname: A
};

Having kind * -> * one could translate that to a function

function getPersonType(A: Type<*>): Type<*> {
  return t.object({
    name: t.string,
    surname: A
  })
}

but before taking this route I'd love to see many examples in the wild, so I guess that the best thing to do now is to release a v0.1 and see what happens.

I don't mean to bash your work with runtime-types, again I want to stress that I'm very interested in what you are trying to achieve. I'm just cautious and I want to take a step at time

[phpnode]

@gcanti I can't really see the difficulty with the example you provided - the lib already supports type parameters, e.g.

  const demo = <T> (input: T): T => input;

Compiles to

  import t from "flow-runtime";
  const demo = input => {
    const T = t.typeParameter("T");
    let _inputType = T;
    const _returnType = t.return(T);
    t.param("input", _inputType).assert(input);
    return _returnType.assert(input);
  };

This might not be the most beautiful output in the world but it can match flow semantics. The type parameter instance captures the type it first receives and uses it for all subsequent checks.

I think I understand where you're coming from, I'm the author of babel-plugin-typecheck and I've run into many of the same headaches that you will have experienced with tcomb. It's why I'm so insistent that flow compatibility needs to be exact, because anything else makes the approach unusable. But it's definitely possible to achieve it, and I don't think doing so needs to bloat the library code itself (although obviously it would require some tooling around it)

[gcanti]

Yes and it's a clever transformation, but I wonder why I'd want to do that, when Flow works just fine and statically.

Runtime type checking is a great tool when you can't adopt a static type checker or as a gentle introduction to Flow (the main goals of babel-plugin-tcomb).

Let's put it in another way, I'm a "die-hard-static-types-guy" so the (admittedly opinionated) rationale behind this library is the following:

1. you should use a static type checker (Flow in our case) for everything, I mean everything. Even if sometimes can be awkward,
2. since IO validation is not doable by a static type checker (by definition) instead of writing custom validators from scratch here's a tool to define them with a bunch of combinators and a concise syntax, which happens to be compatible with Flow (and while we're at it, contains roughly the same core features of tcomb + tcomb-validation with less LOC)

Currently 2) is the main (and only) goal of flow-runtime (*)

In addition, you may use the same combinators in order to get runtime type introspection (and runtime refinements).

(*) perhaps I should rename my project flow-io so you can use the name flow-runtime for yours, which seems more a "flow re-implemented in javascript through runtime type checking" (in case let me know, no problem for me)

[phpnode]

It's true that runtime type checking is not particularly useful if your codebase is fully annotated, Flow has got so much better in the last 18 months. But if your codebase isn't totally annotated then it's still useful, as with babel-plugin-typecheck and babel-plugin-tcomb it can provide a gentler introduction to types and with things like constraints it can be a real benefit during development. I still think there's a place for it.

However the primary motivation behind my library is very similar to yours - it's about being about to use the types for things that flow cannot do, not about replicating what flow already does. Really I just want types to be accessible at runtime, the fact that the types can be used to verify function args etc is almost irrelevant and I don't expect people will use that in production - for me that's more about verifying the completeness and compatibility of the system.

I agree 100% with your goals, it's just that they're not my only goals - do a search for "types" or "validation" on npm, there are hundreds of results, all very slightly different, almost all entirely incompatible. It's a travesty! There's also the forthcoming Typed Objects proposal for ES which will introduce another way to define types. Having a unified way to do all of this would really enable a lot of cool things and save an enormous amount of effort. Speaking of which, I just looked and it turns out we each started work on babel-plugin-tcomb and babel-plugin-typecheck within 30 days of each other, and here 18 months later, again we've independently started work on something very similar within 30 days of each other. It's kind of you to offer to change your project's name but I think we should work together if we can? It's early enough that combining the projects wouldn't be very difficult.

---

Edit: Here's the transformation for the actual example you posted, i.e. on a type rather than a function:

  type Person<A: string> = {
    name: string;
    surname: A;
  };

->

  import t from "flow-runtime";
  const Person = t.type("Person", Person => {         // Inside the function Person is PartialType<T>
    const A = Person.typeParameter("A", t.string());  // declare our type parameter, in this case bound to string.
    return t.object(                                  // The return value completes the PartialType
      t.property("name", t.string()),                 // We could also allow a simple map of property names to types here.
      t.property("surname", A)                  
    );
  });
  // Person is now TypeAlias<ObjectType<<A: string>() => {name: string, surname: A}>>

[gcanti]

Thanks for the example. And then, how do you use Person? In order to do IO validation I guess it should be monomorphized. I'm thinking of the more general polymorphic type:

type Person<A> = {
  name: string;
  surname: A;
};

[phpnode]

To reference Person we could do a transformation like this:

import {reify} from 'flow-runtime';
type Person<A> = {
  name: string;
  surname: A;
};
const PersonType = (reify: Type<Person>);  // reify is just a placeholder, has type any
PersonType.validate({name: 'bob', surname: false});

->

import t, {reify} from 'flow-runtime';
const Person = t.type('Person', Person => {
  const A = Person.typeParameter('A');
  return t.object(
    t.property('name', t.string()),
    t.property('surname', A)
  );
});
const PersonType = Person;
PersonType.validate({name: 'bob', surname: false});

Which is a bit hacky but has the nice property that flow understands it.

I might be missing something but why does Person need to be monomorphized (and how would that look)? In this example we can replace A with any because they're equivalent, but if there were more than one reference to A the lib just checks that they are of the same shape.

Slightly related, how would you describe a self-referential type in your lib? e.g.

type User = {
  friends: User[];
}

I use a similar mechanism as for type parameters -

const User = t.type('User', User => t.object(t.property('friends', t.array(User)));

[gcanti]

The same result writing the definitions by hand (current implementation on master)

import {
  Type,
  ObjectType
} from '../src/index'

import * as t from '../src/index'

type Person<A> = {
  name: string,
  surname: A
};

function getPersonType<T>(A: Type<T>): ObjectType<Person<T>> {
  return t.object({
    name: t.string,
    surname: A
  })
}

// monomorphization
const PersonType = getPersonType(t.string) // <= type-at-pos says PersonType is ObjectType<Person<string>>

const apiJSON = {name: 'bob', surname: false} // <= comes from a fetch
t.validate(apiJSON, PersonType)

reify is just a placeholder, has type any

Yep https://github.com/gcanti/babel-plugin-tcomb#runtime-type-introspection

but if there were more than one reference to A the lib just checks that they are of the same shape

Mmhh this is a slippery ground, Flow likely would augment the type with a union.

Slightly related, how would you describe a self-referential type in your lib?

In a very similar way

const User = t.recursion('User', User => t.object({ friends: t.array(User) }))

[phpnode]

Ah I see, I've been calling that a "type parameter application" but I probably have my terminology wrong. Right now

  type Person<A> = {
    name: string,
    surname: A
  };

  type PersonType = Person<string>;

->

  const Person = t.type("Person", Person => {
    const A = Person.typeParameter("A");
    return t.object(
      t.property("name", t.string()),
      t.property("surname", A)
    );
  });

  const PersonType = t.type("PersonType", t.ref(Person, t.string()));

The important bit here is t.ref() which takes a type or value and 0 or more type instances which serve as parameters for that type. It returns a new type with those type parameters applied.

Yep https://github.com/gcanti/babel-plugin-tcomb#runtime-type-introspection

Wow, that is basically identical.

Mmhh this is a slippery ground, Flow likely would augment the type with a union.

Doesn't it simply check that whenever a value is "written" to a particular type parameter, all writes to that instance must be (loosely) compatible? That's the semantics we have here. Maybe I'm misunderstanding.

[gcanti]

Doesn't it simply check that whenever a value is "written" to a particular type parameter, all writes to that instance must be (loosely) compatible

I was referring to the previous example where A was not specified, it depends on what you mean by "(loosely) compatible", but maybe I'm off the track

type T<A> = {
  a: A,
  b: A
};

declare function f<A>(x: T<A>): void;

f({ a: 1, b: 's' }) // <= A is number | string

[phpnode]

oh I see, by "loosely compatible" I meant it would consider the value "abc" to be string not "abc", but this is obviously different. I don't think it would be so difficult to fix this - if the type parameter is unbound subsequent writes should make a union.

[gcanti]

@phpnode just renamed to flow-io so you can use flow-runtime

I guess that the best thing to do now is to release a v0.1 and see what happens

I'm going to release a v0.1 soon, with the only scope of IO validation written "by hand" (Flow types + runtime types). Then we can learn from real world use cases what can be possibly automated (or if we can join the forces on a common project).

Your goal, while interesting, seems too broad for my current use case. Being a (semi)happy user of Flow, I only need to cover the remaining bits for now. Thanks for the interesting discussion, it helped me to focus on the very goal of this project.

[phpnode]

@gcanti understood, appreciate you changing the name and thanks for the enlightening discussion! I'll keep watching this project and let you know when flow-runtime is released, hopefully we'll be able to join the projects in some way in future - I'll certainly be copying some aspects of flow-io, especially the error reporting and static runtime type validation which are both really nice. All the best and thanks again.

[gcanti]

let you know when flow-runtime is released

Thanks, I'll keep an eye on your project and feel free to reach out if you want to discuss some issues you encountered

I'll certainly be copying some aspects of flow-io

👍 it's what open source is all about from my POV, cross pollination of (hopefully) good ideas

[DjebbZ]

Joining the conversation a bit late :)

While I completely understand your goals gcanti (focus on the IO, the only problem you really have with Flow), I also agree with phpnode about having one way to represent types, accessible at runtime. It may allow something that is missing from all your (wonderful) tcomb toolkit : generative/property-based testing (à la QuickCheck). The Clojure community understand it very well. Since v1.9 Clojure includes a library similar to tcomb, but compatible with their core generative testing library. It makes expressing domain problems and testing them thoroughly extremely nice. But Clojure doesn't have static typing (typed clojure doesn't count).

Javascript has a lib for generative testing, jsverify, but it has its own lonely way of expressing types. I'm not bashing it at all, just saying it lives in a silo when it comes to representing types in Javascript.

You (gcanti) are bringing something else to the table with babel-plugin-tcomb and flow-io : bridge the gap between static and runtime check. If there could be a unified way to represent types at runtime, based on Flow syntax, we could use it to make generative testing in Javascript. Having this would put Javascript on top of the game : static typing + runtime typing + generative testing with the same syntax. When you include the rest of the tcomb toolkit (for docs, schema, etc.), one can truly achieve something unprecedented in current mainstream programming languages.

As you say, one step at a time. It's been a long time I've wanted to tell you this. Flow is very good, tcomb is too. Javascript really needed them. It also needs a good generative testing library that interoperates with a type representation, and Flow's/yours with tcomb may just it.

Thanks for reading.

[gcanti]

If there could be a unified way to represent types at runtime

@DjebbZ Well there is, technically the purest form is a predicate

const string: (value: mixed) => boolean
  = value => typeof value === 'string'

Things get more complicated when you want better error messages

Validation function

const string: (value: mixed) => Either<Error, string> 
  = value => ...

and a name

Type<T>

interface Type<T> {
  name: string,
  validate: (value: mixed) => Either<Error, T>
}

and runtime type introspection

ObjectType<T>

interface ObjectType<T> {
  name: string,
  validate: (value: mixed) => Either<Error, T>,
  props: { [key: string]: Type<any> };
}

Having a real unified way means either

• a library so popular that is the de-facto standard
• a specification (something like fantasy-land for runtime types)

Related

• rtype
• Typed Objects
• jsig

[gcanti]

@DjebbZ I think that ultimately the single source of truth should be the Flow syntax and the types defined for Flow because

• you write them anyway for leveraging static type checking
• the syntax is pretty good and concise

Once you have defined the types, you can generate many useful things through some transformations of the babylon AST

• source code -> AST
• AST -> runtime types
• AST -> documentation
• AST -> automatic IO Validation only in development
• AST -> generative tests
• AST -> GraphQL Schemas
• AST -> JSON Schema

You could also generate the types from external sources, for example

• JSON Schema -> AST
• GraphQL Schemas -> AST

The AST, or a simplified JSON DSL representing Flow types, may be, as a common language, the key to interoperability.

This is an little experiment https://github.com/gcanti/flow-io/tree/transformers/src (see the ast, builders and transformers folders)

and here some tests to get the general idea

• https://github.com/gcanti/flow-io/blob/transformers/test/fromJsonToRuntimeTypes.js
• https://github.com/gcanti/flow-io/blob/transformers/test/fromTypesToJson.js

[phpnode]

The AST, or a simplified JSON DSL representing Flow types, may be, as a common language, the key to interoperability.

The issue with this is that it's very useful to be able to quickly tell whether a particular value is actually a value or a type, and that's hard to do safely with POJOs. Also, someType.assert(someValue) is convenient and different types have different properties or capabilities, e.g. it's nice to be able to do someObjectType.getProperty('foo') or someFunctionType.getParam('input'). It would be possible to allow a shorthand for making a POJO into a type though:

const string = {
  name: 'string',
  validate () { ... }
};

// string is just a value

const stringT = t.type(string);

// stringT is Type<typeof string>

stringT.assert('something');

[volkanunsal]

The AST, or a simplified JSON DSL representing Flow types, may be, as a common language, the key to interoperability.

I agree with @gcanti on this. The transformer examples are very thought provoking, but would be great to have more real world examples. Are they intended for production use? It seems like they rely on babylon runtime, which is generally not a part of frontend code...

[phpnode]

In case anyone's interested I published the first version of flow-runtime - https://codemix.github.io/flow-runtime

[volkanunsal]

I like the idea of flow-runtime, but I just spent a couple of weeks removing babel-plugin-tcomb from my apps due to subtle differences between flow types and runtime types. So I'm wondering whether the same thing can happen with flow-runtime as well?

…

On Wed, Jan 4, 2017 at 3:51 PM Charles Pick ***@***.***> wrote: In case anyone's interested I published the first version of flow-runtime - https://codemix.github.io/flow-runtime — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#19 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAJQMFJhhQXEvY5MneKZdOfDWlTcf9JDks5rPAZNgaJpZM4Kvp4D> .

[phpnode]

@volkanunsal it can happen but it would be a bug so if you find such a case please file an issue, i'm striving for full flow compatibility

[gcanti]

@phpnode hi, I'm glad to hear that, I'd love to add a link to flow-runtime in the readme, would you like to send a PR?