Automatic Backward/Forward .d.ts compatibility system

[cordiosd]

Problem: When you are forced to upgrade a library for a security concern or because it has become non functional due to some other concern and that upgrade now uses a different version of typescript to generate its .d.ts files that sit next to the .js files, your entire code base can become problematic. In some cases there is no possible solution outside of forking the offending libraries and manually modifying it to work within the current state of your system based on the current version of the tsc you are using.

Typescript was supposed to be only helpful (i.e. never harmful) in improving the system over javascript. And outside of this problem, that has always been the case. This brings that into question as the Javascript can be fine but the .d.ts files can make it so there is no path forward with a specific compiler version as some dependendencies can have .d.ts files from older incompatible versions and other dependencies have .d.ts files from newer incompatible versions.

Suggested Solution: Make it so that multiple .d.ts files are generated whenever there would be incompatibilities for older versions of the typescript compiler. In the header of the standard .d.ts file include compiler metadata about the version this was generated for and the other versions that were generated. The multiple .d.ts files per .js file would have one version of the .d.ts file per incompatibile syntax.
When all future compilers read the .d.ts file they would look for this metadata and choose the most appropriate .d.ts file for their build.

Suggested configuration options within the tsconfig.json file:
Setting 1: Enable the ability in the tsconfig.json file to generate multiple .d.ts files.
Setting 2: Specify the oldest tsc version that should be considered to minimize the number of .d.ts files
Setting 3: Specify the tsc version that the default .d.ts file will be created for. Newer versions of the compiler would read and inspect the meta data in this file and then use the appropriate .d.ts file for its capability.

My Experience
The following is the situation that spawned this issue for me:

We have a build tool that used an old version (3.X) of the npm inquiry module. We found that once we upgraded node to 10.14.2, that node stopped working properly (node programs hung in future calls to console.log()) after any CLI UI from this library was accessed. We had been on tsc version 2.5.3 to avoid issues with tsc incompatibility. The version 6.x of inquiry came with .d.ts files that were incompatible with our typescript version. Being a good excuse to get the benefits of the new type checking we updated to 3.4.5 (after having a problem with 3.5.X). This solved our issue perfectly (or so we thought). All of our many apps/libraries now built with less than 3 source code changes and 2 tsconfig.json file changes. That was until we decided to perform a production build. In this case the 1.X version of ngbootstrap would no longer build with the newer tsc 3.4.5. And updating ngbootstrap can't happen without updating angular as the angular .d.ts files changed a type to a generic. Increasing the angular version in our web apps would have a laundry list of steps and testing to take care of updating these to version 7. But our cordova apps, dependent on Ionic3 requires an actual recreation of a newer Ionic4 app and then manually moving your old code over to the new framework. All of this to fix an issue with a command line tool unrelated to these projects but that shares a library is a serious problem.

At this point it became clear that our best path for now is to downgrade from node 10.14.2 back to 10.11.0. But this brings up a serious question about the viability of tsc when using libraries from other developers that we don't have control over and do not want to take over every time this problem rears its ugly head. Writing code that has an unknown cost of maintaining in the future while just trying to apply maintenance updates due to versioning conflicts just did not happen very much prior to npm type code library package distribution from parties without an obligation to maintain their libraries.

The urgency we hit with the problem was exasperated by our having all of our projects automatically use the latest build of all of our modules. This technique has been a substantial benefit in writing code in the proper library to share between multiple applications. Had we moved each of our modules forward in the slower semantic versioning process as if each module was created by a 3rd party, we would have hit this same issue and it would have caused more work, but in a slower fashion. In other words is would have a larger amount of work, but the timing of the fixes would occur over time. We would have had to update a core library to use the newer version of typescript so that the library could be used with a newer application. Other applications not currently being developed would have continued to use the old version until they needed a fix. As soon as they also needed a change to a core library, the problems we hit all at once would have to be resolved also.

Conclusion
There is no path forward for developers, like those who developed the inquiry module even if they wanted to to create a .d.ts file that could work with all versions of tsc. They have no way to maintain their old work. With the suggestions described here, they would not need to and everyone could use the latest typescript compiler as soon as it came out without worrying about problems caused to users of their modules or their use of other modules.

[MartinJohns]

Related: #22605

[RyanCavanaugh]

#22605 is the intended fix for everything going forward.

For the reverse case, we do want someone (hopefully in the community) to write a tool that converts new TypeScript syntax in .d.ts files to the closest available downlevel analogue, though this not something that can realistically be done perfectly in all cases.

Generating older-version-compatible files isn't something we're willing to sign up for in terms of complexity vs payoff.

[cordiosd]

Ryan,

Thank you for the feedback and response.

Reviewing #22605, I am concerned about the manual and necessary feedback loop / knowledge required for an author to know that the .d.ts file generated will not be accessible to someone using an older version of typescript as well as the low incentive for many package creators to take this step. Critical bug fixes won't be consumed if the consumer can't also update their tsc version. I believe that without the tools or pipeline automatically creating compatible .d.ts versions that they won't be created in a large number of cases and that will lead to either a reduction in the utilization of typescript or the reduction in the utilization of smaller packages as using many smaller packages is less stable for a longer term application lifecycle when using typescript.

The domain knowledge and skills required to both create and maintain a downlevel .d.ts generator are most cost effectively created by those writing the compiler. I understand that adding support for backward compatibility is taking resources away from other "sexier" features developers probably want to work on. But I believe it is key to the most rapid adoption of the latest versions of the typescript compiler by all of the developers who create using the typescript compiler.

Perhaps I have been spoiled by the C# team's attentiveness to backward compatibility.
https://docs.microsoft.com/en-us/dotnet/framework/migration-guide/version-compatibility

My takeaway from my recent projects are:

1. If you are going to do any large scale Javascript development you must use Typescript.
2. If you are going to use Typescript, you need to minimize your dependency on other packages you have no control over. I.e. only use large packages that provide substantial value like angular or react. Hopefully only use 1.
3. For all other code, either develop it yourself or inspect and copy other packages that have problems solved that you would like to use but can't trust they will work in 6 months within your environment. This is because your code, tsc, and the rest of the open source community moves forward at their own independent rates with varying incentives to maintain the code and will likely be introducing incompatibilities whose resolution is likely greater than the upfront cost of developing the code yourself.

As for item 2, is there an escape hatch I am unaware of? For example, if I found I was dependent on a package that had an incompatible .d.ts file in the package, is there some method for me to tell the typescript compiler for this node_modules folder to use .d.ts files from another location so that I could copy and paste the .d.ts files into my repository and fix the offending .d.ts files manually? If not, how hard would that be to add?

I am a huge fan of Typescript. Prior to Typescript, I think the only tool for doing any efficient large scale Javascript development was the Closure Compiler. And without the intellisense and development tools that come with Typescript, I question how efficient that was also, as I never used it in a project.

[typescript-bot]

This issue has been marked as "Too Complex" and has seen no recent activity. It has been automatically closed for house-keeping purposes.