Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Typescript types #1

Closed
Qix- opened this issue Sep 10, 2020 · 5 comments
Closed

Typescript types #1

Qix- opened this issue Sep 10, 2020 · 5 comments
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@Qix-
Copy link
Owner

Qix- commented Sep 10, 2020

cc @dragomirtitian - I was the one who asked this StackOverflow question - perhaps you can further help me.

Also cc @blakeembrey, since you've helped me in the past with ZEIT's Vercel's arg package :)

Here are the types we've come up with so far, thanks to @dragomirtitian's help:

type UnionToIntersection<U> = (U extends any ? (k: U) => void : never) extends (
	k: infer I
) => void
	? I
	: never;

type MakeAggregateObject<T> = {
	[P in keyof T]: T[P] extends (
		...p: infer P
	) => AsyncGenerator<any, infer R, never | infer R>
		? (...p: P) => Promise<[boolean, R]>
		: never;
};

declare function scaly<T extends [any] | any[]>(
	objects: T & Partial<UnionToIntersection<T[number]>>[]
): MakeAggregateObject<UnionToIntersection<T[number]>>;

export = scaly;

However, it breaks down pretty quickly once you start using the library as it's intended.

Take the following example:

import scaly from 'scaly';

const mongoLayer = {
	async *getName(eid: string) {
		return 'mongo:' + eid;
	}
};

const redisLayer = {
	async *getName(eid: string) {
		console.log('REDIS GOT', yield);
	},
	async *getError() {
		yield 'this is an error';
	}
};

const memcacheLayer = {
	async *getName(eid: string) {
		console.log('MEMCACHE GOT', yield);
	},
	async *getError() {
		// Should never get hit, as the next layer is `redisLayer`,
		// which `yield`s an error (as per the Scaly specification).
		// Scaly treats `yield expr` more like a `throw` in that it never
		// resumes the generator instance.
		console.log('THIS SHOULD NEVER SHOW UP', yield);
	}
}

const DB = scaly([
	memcacheLayer, // Hit LRU first ...
	redisLayer,    // ... followed by Redis ...
	mongoLayer     // ... followed by MongoDB.
]);

async function main() {
	console.log(await DB.getName('foo'));
	console.log(await DB.getError());
}

main().catch(err => {
	console.error(err.stack);
	process.exit(1);
});

The above should output:

MEMCACHE GOT mongo:foo
REDIS GOT mongo:foo
[ true, 'mongo:foo' ]
[ false, 'this is an error' ]

But typescript is choking on the types:

index.ts:10:9 - error TS7055: 'getName', which lacks return-type annotation, implicitly has an 'any' yield type.

10  async *getName(eid: string) {
           ~~~~~~~

index.ts:19:9 - error TS7055: 'getName', which lacks return-type annotation, implicitly has an 'any' yield type.

19  async *getName(eid: string) {
           ~~~~~~~

index.ts:22:9 - error TS7055: 'getError', which lacks return-type annotation, implicitly has an 'any' yield type.

22  async *getError() {
           ~~~~~~~~

index.ts:27:18 - error TS2345: Argument of type '[{ getName(eid: string): AsyncGenerator<any, void, any>; getError(): AsyncGenerator<any, void, any>; }, { getName(eid: string): AsyncGenerator<any, void, any>; getError(): AsyncGenerator<...>; }, { ...; }]' is not assignable to parameter of type '[{ getName(eid: string): AsyncGenerator<any, void, any>; getError(): AsyncGenerator<any, void, any>; }, { getName(eid: string): AsyncGenerator<any, void, any>; getError(): AsyncGenerator<...>; }, { ...; }] & Partial<...>[]'.
  Type '[{ getName(eid: string): AsyncGenerator<any, void, any>; getError(): AsyncGenerator<any, void, any>; }, { getName(eid: string): AsyncGenerator<any, void, any>; getError(): AsyncGenerator<...>; }, { ...; }]' is not assignable to type 'Partial<{ getName(eid: string): AsyncGenerator<never, string, unknown>; } & { getName(eid: string): AsyncGenerator<any, void, any>; getError(): AsyncGenerator<string, void, unknown>; } & { ...; }>[]'.
    Type '{ getName(eid: string): AsyncGenerator<never, string, unknown>; } | { getName(eid: string): AsyncGenerator<any, void, any>; getError(): AsyncGenerator<string, void, unknown>; } | { ...; }' is not assignable to type 'Partial<{ getName(eid: string): AsyncGenerator<never, string, unknown>; } & { getName(eid: string): AsyncGenerator<any, void, any>; getError(): AsyncGenerator<string, void, unknown>; } & { ...; }>'.
      Type '{ getName(eid: string): AsyncGenerator<never, string, unknown>; }' is not assignable to type 'Partial<{ getName(eid: string): AsyncGenerator<never, string, unknown>; } & { getName(eid: string): AsyncGenerator<any, void, any>; getError(): AsyncGenerator<string, void, unknown>; } & { ...; }>'.
        Types of property 'getName' are incompatible.
          Type '(eid: string) => AsyncGenerator<never, string, unknown>' is not assignable to type '((eid: string) => AsyncGenerator<never, string, unknown>) & ((eid: string) => AsyncGenerator<any, void, any>) & ((eid: string) => AsyncGenerator<any, void, any>)'.
            Type '(eid: string) => AsyncGenerator<never, string, unknown>' is not assignable to type '(eid: string) => AsyncGenerator<any, void, any>'.
              Call signature return types 'AsyncGenerator<never, string, unknown>' and 'AsyncGenerator<any, void, any>' are incompatible.
                The types returned by 'next(...)' are incompatible between these types.
                  Type 'Promise<IteratorResult<never, string>>' is not assignable to type 'Promise<IteratorResult<any, void>>'.
                    Type 'IteratorResult<never, string>' is not assignable to type 'IteratorResult<any, void>'.
                      Type 'IteratorReturnResult<string>' is not assignable to type 'IteratorResult<any, void>'.
                        Type 'IteratorReturnResult<string>' is not assignable to type 'IteratorReturnResult<void>'.
                          Type 'string' is not assignable to type 'void'.

 27 const DB = scaly([
                     ~
 28  memcacheLayer, // Hit LRU first ...
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...
 30  mongoLayer     // ... followed by MongoDB.
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 31 ]);
    ~


Found 4 errors.

Which... you weren't kidding about the hairy error output.

The tl;dr here is that yield and return are both entirely optional, and the types are affected based on how the function uses them:

  • return; (no return value) is acceptable, always. It's a special signal.
  • return expr; is acceptable, but the expr type must match between all layers. It is what becomes the Promise tuple's second type (hence the edit to the SO question) - hence the Promise<[boolean, infer R]>.
  • yield (with nothing yielded) is acceptable and yields the same type as the return type (the infer R bit).
  • yield expr; is acceptable, but never returns (not sure if this makes a difference with typescript) and can be any (a string, Error, whatever) - it is treated as a "graceful throw" in Scaly's terms.

Of course, the methods can also throw but Scaly treats those as irrecoverable/internal/fatal errors, not user-generated errors.


The existing types get close (remove getError() and the call to it, and the types are happy and correct) but I'm really afraid we won't be able to achieve type correctness for this weird API without requiring we add : AsyncGenerator<...> return type annotations to everything (which would be highly annoying).

The other thing I want to avoid is e.g. one implementation decides it wants to yield an error, that all of the other implementations now have to update their type definitions. It would be great if it could error when there's a violation of type symmetry where it counts, but doesn't error when otherwise inconsequential things like that happen.

For example, the Mongo layer might all of a sudden yield new SomeMongoErrorType('error') for getName(). I don't want to have to update the Redis layer's getName() signature to include SomeRedisErrorType | SomeMongoErrorType. Fast forward to 20 layers, and these types become unwieldly.

Am I dreaming of a utopia, or is this possible with Typescript?

@Qix- Qix- added enhancement New feature or request help wanted Extra attention is needed labels Sep 10, 2020
@Qix-
Copy link
Owner Author

Qix- commented Sep 10, 2020

cc @dylan1p

@dragomirtitian
Copy link

dragomirtitian commented Sep 10, 2020

@Qix- I think it might be possible, but It's not really clear to be what the rules should be. AsyncGenerator has 3 parameters:

  • T - the yielded item type. So if we use yield expr, the type of expr will become T. In the case above when you use yield (without an expression) that will result in T being undefined. When no yield is used, you get never as T.
  • TReturn - the type of the return. When return is used the type of the expression passed to return will become TReturn
  • TNext - The return of the yield expression. The type of the target where you assign yield will become this type. In the examples above since you assign yield as the second parameter to console.log TNext is any. Or unknown if it no yield is used. (If you were to only use yield in let x: number = yield;, TNext would be number)

So what matters to the compatibility of these generators ? The current issue is that TReturn is void from memcacheLayer but it's string from mongoLayer. Does TReturn not matter for assignability ? Or is not returning a value always acceptable ?

What about the other parameters ? Is the value returned by yield ever important ? Does it matter to the compatibility of the objects? What about the yielded values ? How do those figure into it.

@Qix-
Copy link
Owner Author

Qix- commented Oct 24, 2020

Sorry for the delay on this; this is turning out to be a really tricky comment to word correctly since the types are quite hairy. Hopefully you're still interested in helping @dragomirtitian 😅

Before continuing, let me define a term so that we're on the same page: while the string keys and their function values are just that - functions - I'll refer to the set of functions that share the same key among any of the objects in which that key is present as a "method".

const cat = {
    meow: async function *() {},
    walk: async function *() {}
};

const human = {
    speak: async function *() {},
    walk: async function *() {}
};

const result = scaly([human, cat]);

In the above, there are two layers - human and cat. Between the two layers are three methods - meow, walk and speak. Scaly is just a library that manages control flow between multiple functions that compose the overall method walk, in this case, since walk shares two layers.

Hence, walk has two functions, but is a single method.

Further, I'll refer to the async generator declarations as input methods and the single "aggregate" resulting method as the output method.

  • The output method's return type is [boolean, T] where T is the input methods' return type if the first tuple boolean is true, or one of the various yield expr types if the first tuple boolean is false, with one exception...
  • The input method's return type can also be undefined (whereas the only case where the output method's return type is void is if the overall method's return type is also void; otherwise, you'll never get undefined by calling the output method)
  • The input method can yield expr, where expr is anything except for undefined. The type does not have any relation to the method's return type. This form of the yield expression never returns, either (we treat it as a throw expression, more or less).
  • The input method can yield undefined (or just yield of course), which is a special case that might produce a value from another layer's input method implementation. The type of expression yield undefined is thus the same as the method's return type.

Let me annotate an example with inline comments explaining why the types are what they are.

import scaly from 'scaly';

/*
	Three output methods:
	    - async getName(string) : string
	    - async getLocation(string): [string, string]
	    - async setName(string) : string|null
*/

const gpsLayer = {
	async *getLocation(id: string) {
		try {
			const loc: [string, string] = await someGPSClient.find(id);
			return loc;
		} catch (err) {
			if (err.code == 'UNKNOWN_ID') {
				// "throw" a recoverable error by `yield`-ing a non-undefined value.
				// This form of `yield expr` never resumes (returns).
				// Note that we yield an `Error` object here, not a `string`.
				yield err; // yield expr (where typeof(expr) is anything except undefined)
			} else {
				// otherwise, throw a real exception.
				throw err; // not covered by typescript I don't think.
			}
		}
	}
};

const mongoLayer = {
	async *getName(id: string) {
		const name: string|undefined = await someMongoClient.find({id}, {name: 1});
		if (!name) yield 'unknown user ID'; // "throw" a recoverable error again
		return name;
	},

	async *setName(id: string, name: string) {
		const oldName: string|undefined = await someMongoClient.upsertAndGetOldValue({id}, {name});
		return oldName || null; // return the string if it's there, or `null` if it's falsey
	}
};

// This is where the use of `undefined` becomes important.
const redisLayer = {
	async *getName(id: string) {
		const name: string|undefined = await someRedisClient.get(`${id}:name`);

		// If it's a cache-hit, then simply return it
		if (name) {
			return name; // typeof(name) == string, same as mongoLayer's `getName()` method type
		} else {
			// Otherwise, produce the value from the next layer.
			// The result of a naked `yield` is never `undefined` because
			// Scaly will error if no result can be produced from any of the
			// subsequent layers.
			const newName: string = yield; // same as the method return type, never `undefined`.
			await someRedisClient.set(`${id}:name`, newName);

			// In this case, the return value can be `undefined` since Scaly
			// already has the result value. There is no need to `return newName`
			// again a second time.
			return undefined;
		}
	},

	async *getLocation(id: string) {
		const loc : [string, string]|undefined = await someRedisClient.get(`${id}:location`);

		if (loc) {
			return loc; // typeof(loc) == [string, string], same as gpsLayer's `getLocation()`
		} else {
			// Same thing here.
			const newLoc: [string, string] = yield;
			await someRedisClient.set(`${id}:location`, newLoc);
			return undefined;
		}
	},

	async *setName(id: string, name: string) {
		// This is the strange one; in our implementation here,
		// we just want to observe the calls to `setName()` without
		// producing a result and letting the later layers
		// have a chance at using the parameter, too.
		//
		// Even though we don't return a result in the implementation,
		// the return type should still be string|null|undefined since that's
		// what the other layers' inner method return types are (specifically,
		// mongoLayer.setName(string, string): string|null).
		//
		// Therefore, there are two ways we can write this
		// implementation:

		{
			// The first way is to handle it ourselves first,
			// then let the function return `undefined` to
			// indicate we don't care about the result
			// nor do we have a result or an error to produce,
			// and that the next layers, if any, should be invoked:
			await someRedisClient.set(`${id}:name`, name);
			return undefined;
		}

		{
			// Alternatively, we can let the deeper layers handle
			// the request first, and if there is no error result
			// produced, we can then handle it *after* the other
			// layers have handled it:
			yield; // discard the result (which would otherwise be type `string|null`)
			await someRedisClient.set(`${id}:name`, name);
			return undefined;
		}
	}
};

const DB = scaly([
	redisLayer,
	mongoLayer,
	gpsLayer
]);

/*
	typeof(DB) == {
		getName: async function (string): [true, string] | [false, string],
		setName: async function (string, string): [true, string|null] | [false, unknown],
		getLocation: async function (string): [true, [string, string]] | [false, Error]
	}
*/

The resulting object's function values' return types are a tuple of [boolean, T], as I mentioned before. The T depends on the value of boolean, though I don't know how expressable this is in Typescript. If the boolean is true, then the second tuple value's type is the method's return type. If it is false, then it's one of the yield expr types.

Hopefully this makes sense. I perfectly understand if this is something you're not interested in doing 😅 Scaly is definitely the most complicated libraries I've personally seen when it comes to type munging so I can understand not wanting to do it.

I'm mostly interested in base-level correctness - for example, if you can only express the resulting methods' return types as [boolean, result_types | error_types] as opposed to the [true, result_types] | [false, error_types] pseudocode I have above, that is just fine; just so that valid uses of Scaly pass any sort of Typescript type checking at the very least. Anything beyond that would be awesome but not necessary :)

As always, any information is helpful even if it's short of the full typings, haha. Though I think it's a fun puzzle for those interested in cryptic Typescript type definitions!

Thanks again for the response and the help on SO!

@Qix-
Copy link
Owner Author

Qix- commented Dec 8, 2020

cc @adamhaile - S.js/Surplus must have some crazy hairy typing, maybe you can shed some light on how to approach this. :D

@Qix-
Copy link
Owner Author

Qix- commented Sep 2, 2021

Got a new job where I'll be writing Typescript and needed a good project to refresh my understanding. After a few hours, I can say I've slapped these together.. decently enough.

Released as 1.0.0. Please open new issues for any weirdness.

@Qix- Qix- closed this as completed Sep 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants