Add Type Validator to Build Tools and Add Protocol Definitions Example

[anthony-murphy]

The Type Validator does two things:

• Generates a type data file, which includes typing data for previous versions no longer generating type data, just pulling directly from package
• Generates tests that ensure types are compatible across versions

The generate tests do not actually get run, they just get compiled, which ensures there are not compile breaks introduced.

I've manually run the Type Validator for protocol-definitions and generated the type data along with the test. once the validation tool is checked in, i plan to integrate this tool directly into the build:test npm script.

#6540

[anthony-murphy]

const enums are not stable across versions in this version of typescript. in version 4.3 of typescript this works. in general enum are not extenable

[heliocliu]

Some questions, not sure if they'll be handled/answered in the next part. Are you planning to run this for all packages? How will this work for intentional breaking changes on the types? Is the idea to always run against the latest release for now (and not vs major/minor changes specifically)?

Looks pretty cool over all. Wondering if this can be leveraged for broader API validation for breaking changes 🤔

[anthony-murphy]

Some questions, not sure if they'll be handled/answered in the next part. Are you planning to run this for all packages? How will this work for intentional breaking changes on the types? Is the idea to always run against the latest release for now (and not vs major/minor changes specifically)?

Looks pretty cool over all. Wondering if this can be leveraged for broader API validation for breaking changes 🤔

@heliocliu These are good questions, and i don't have full clarity on them yet, but i'll share my thinking so far.

Right now, i'm only planning to run this on our definition packages, but i don't see any reason we can't run it on more packages, or maybe even all if we think it is necessary.

For breaking changes, I don't have a full design yet, but you can see that i can access js doc tags like deprecated. My thought is we enforce some deprecation flow based on semver. So we can pretty easily ensure that something is deprecated in the previous major version before we allow a breaking change. Then to articulate the break it'd be really nice if we could leverage some doc tag: https://tsdoc.org/ or https://www.typescriptlang.org/docs/handbook/jsdoc-supported-types.html . Ideally this would also be something we can generate our breaking.md from. However, i haven't found a good way to do it yet, so we may end up needing another file.

For actually specifying what broke we have a couple options, the easiest is to just use a big hammer, and not test whatever is broken aka don't test forward or back compat. But i think we should probably be able to split those, and specify what was broken, back or forward compat, and only test that. Beyond that is the most complex and would require figuring out exactly what broke, and precisely test that, but this runs into the issue that we now need to deeply understand the type, like all it's properties, and i like the current simplicy of just considering the exported type, and not its structure.

[heliocliu]

I really like the approach here of looking at just the exported type and not going too deep with the properties since that appears to be sufficient for detecting breaking changes at least. I think this lets us cover transitive breaking changes as well - if any type receives a breaking change, any dependent package should also fail this check if that type is used in an exported API signature (given some constraints like no re-exported types and bad any typing etc).