TSD Future

[blakeembrey]

Hey everyone!

I'm going to be jumping into this project and seeing what we can do to improve the sharing of TypeScript definitions. Development isn't (quite) going to happen instantly, so I wanted to submit a short issue that describes some of my immediate goals and see if there are other ideas of what they want to see moving forward. Feel free to create new issues with feature requests and we can label them for TSD Future.

Goal

The goal of TSD future is to seamlessly support any TypeScript definition files, from any source. By default, you should still be able to search the definitely typed repository and install from there. However, more focus will be trained onto support independent definitions, specifically ones bundled with modules. This is a primary focus because as the ecosystem grows, a single repo is going to be a challenge to scale and I would like to give more power to module authors.

Currently TSD (tsd@next) supports a feature called tsd link. It's a great first step, but comes with many drawbacks - mostly because it's just a really simple path copy and paste. It needs a bit more to be useful. In the future, you'll be able to tsd install and it will look up your dependencies via package.json or bower.json and it will install definition files. However, it'll then recursively resolve dependencies from there. Any dependencies the module has likely have their own definition files that they're depending on.

This will make tsd.d.ts less editor friendly, but the goal is to avoid having to manually edit this file and check in dependencies any more. Just as you don't check in your node_modules or bower_components, you shouldn't have to check in your type dependencies. As we resolve dependencies down the tree, if we encounter tsd.json files they'll also factor into the recursive resolution. This is important since not every module can or will adapt over night. If there are conflicts between sub-dependency definitions, we will provide a dialog for selecting the version that should be used (resulting in that flat structure we wanted, a la bower).

Doing this, we'll pull closer to the seamless usage of definition files. Similarly, the commands will be pulled closer to npm and bower. All that will be required from now on is tsd install. To remove old dependencies, you can tsd prune. Finally, tsd uninstall will remove a dependency from tsd.json. The tsd.d.ts file will always be kept in sync with hard module dependencies and the tsd.json.

To support all this, I think some changes to tsd.json are required. I'm going to change this last (avoid it if I can), but I think it's too tied to the one source approach which doesn't scale. From now on, manual definition file installations tsd install <source> should create an entry like package.json with "name": "source", where source can be any file/url (tsd.json or d.ts file). For example, the source when installed from the definitely typed repo will the raw git files. This allows anyone to host their own definitions. tsd install <module> will still fall back to the definitely typed repo and tsd search <module> will still search the repo. It is possible for this to be automated as well.

Having the name -> source mapping allows us a little more flexibility. For example, we can now check in our custom definition files and have them resolve according to the recursive algorithm since they are defined. On top of that, we can create aliases (which are supported by build tools like browserify and webpack) from one module to another. Finally, we can eventually support the module output format by TypeScript and automatically compile that into the namespaced module. That would personally be very cool to do, since it's a pretty big pain point for third-party module developers.

Ok, I think I'm done rambling but there will definitely be more to come.

Tracking

• TSD to have command parity with NPM and Bower TSD to have parity with NPM and Bower commands #142 Alphabetize entries in installed field of tsd.json #119 Modify install command to perform 'reinstall' if invoked with no arguments #122 Create uninstall command #6 tsd install should install every definition listed in tsd.json #116
• Perform a complete, recursive defintion resolution Packages with a / in the name don't work. #134 TS6053 File not found when import tsd.d.ts from bower distributed package #105 Does tsd link not work? #130 Resolve dependencies for linked bundles #123 Package linker's folder scan uses hardcoded folders #109 tsd reinstall does not remove stale typings #151
• Make sure we're running on all node versions >= 0.8 and iojs Update to be compatible with Node.js 0.12 #144
• Fix general inconsistencies First argument ignored when using flags #143 tsd --save flag modifies permissions of parent directories unnecessarily #125 'undefined' being written to tsd.d.ts #120 Overlap path value install library --save each time in tsd.d.ts #112 Fix CLI query on Max etc #62 Report errors more gracefull #75 Simplify installed property in tsd.json #113 Generating tsd.json changes its parent directory's permission to 0744  #95
• Automatically install unknown dependency definitions tsd link --implicit proposal #115
• Install dependencies from any source query definitions from multiple repo #97 Support non-Github repos. #135 Allow install from seperate repo like bower #87
• Start work on a wrapper that will accept any definition file and declare as a module name

Testing

I will start work in the next few days on the future branch.

npm install -g DefinitelyTyped/tsd#future

Related

There's a few proposals in TypeScript right now to fix the definitions situation that will probably (hopefully) make most of this redundant (microsoft/TypeScript#2338, microsoft/TypeScript#2839).

[5angel]

Glad to know this project still has traction. So when do you plan deaing with #144?

[blakeembrey]

@5angel It definitely still has traction, but I'm not currently focused on previous versions of TSD for a few reasons (namely still not having publish access to NPM yet, since the original author is MIA). That issue doesn't mean it's not working with 0.12 or any other version, it's just the 0.5.x branch was pinned to certain node versions (see https://github.com/DefinitelyTyped/tsd/blob/master/package.json#L37). You can still install, it should only be a warning. If you install with tsd@next it won't even warn you since it specifies >= 0.10.

[5angel]

@blakeembrey phew, thank you. It was thinking about cloning the rep and making a pull request (:

[basarat]

namely still not having publish access to NPM yet,

@blakeembrey We (you and me) have that now. I'll change the tag on 0.6 to be latest (instead of next). I'll also remove beta from it. That will ensure that people can continue to use what we have right now while you work on vnext 👍 🌹

[blakeembrey]

@basarat Fantastic, thank you 😄 I saw it and I'm glad everything is alright.

[basarat]

Alright done ... phew. 0.6.0 is now latest so npm install tsd -g should just work.

[Bartvds]

Author here, sorry for going AWOL and dropping off the grid, stuff went different then expected 😞

I kinda dropped the ball here, sadly can't commit OS coding time anymore so @blakeembrey has my full blessing to go ahead on this and take ownership. Thanks @basarat for being admirably more dependable contributor and getting stuff rolling.

Today I got an email from Angular guys who needed a sub-dependency mistake fixed: minitable. So I pushed an update for it to npm earlier but for some unknown reason I pinned the exact dependency here in TSD. So I think we need an TSD v0.6.1 soonish.. a PR will turn up any time.

Anyway, I'll have a look around the tickets soonish and see what's up. Its been a while since I ran TypeScript and node development though (doing python now (:open_mouth:) And I'll put some text together about project status and stuff.

[blakeembrey]

@Bartvds Nice to see your ok! I was a little worried, even though I'm yet to meet you. Feel free to keep push any TSD versions and I'll make sure it stays maintained until the next version is ready 😄

[Bartvds]

Cool, thanks. I got distracted. Not sure if I get back in TypeScript yet but at least TSD could have second revival (2nd yes).

It was a bit of wayback machine but here is TSD 0.6 bootcamp to help you going: Since I don't know when I got focus again I just typed a lot now:

• 0.3 and 0.4 where from maintainer before me, also got limboed.
• 0.5 was my full rewrite to link directly to github (I had too much time back then). 0.5.7 has been used up till this week.
• 0.6 was major cleanup to imports style and more npm modules (not all, not enough). it got stuck in limbo for release for a long time.

The main issues TSD faces and tries to solve are about user interface and conventions of the DefinitelyType repos. There was (is?) not a 100% consensus about how package naming and versioning are handled.

TSD works using the filenames in first level folder as the module name (npm name recommended), it also considers some legacy nested folders. The main folders are usefull for convenient bundeling but over whole of DT there is not a universal semantic pattern of putting definitions together in folders. Then parses for semvers (filename is leading). The information in actual header of file is mostly cosmetic for listing/search.

Then there is the use of bundles/linked definitions; TSD link and dts-bundle are an attempt but appear flawed. There is a prototype in 0.6 to read information from package.json, an approach to scan for tsd.json could be cool. It gets a bit fuzzy what defs/types you actually need, you might not need all of a packages tsd.json but only what is visible on it's exported API.

Versioning is an issue. It is a mix of the path/name/semver and git's commit system. Keep in mind the semver in path is of the module it describes, not of the definition. There is no guarantee a next commit on same (semvered) filename doesn't break peoples builds, so for safety TSD pins on filename in a specific commit.

There were ideas of adding metadata to either the headers or a separate file in DT repos, and some other ideas in various related repos (here, DT itself, definition-header, definition-tester etc).

Since 0.6 was oldish even before it got lost I was trying a prototype of a cleaner version that uses streams all the way. It supported adapters, to read trees and blobs from file system, github and local git. Most of it worked but I sorta got stuck on streamy http cache and some stream plumbing problems. Could be solved though.

Anyway, some random stuff about code:

• it's a lot of own code, at least it's strict formatted and linted (and typed 😉) and I hope you like generic promises..
• /src code is basically a few main packages:
  • tsd has the type-definition specific stuff (and main app)
  • xm folder has (own) old utils. i tried to replace much of it with npm modules in 0.6 but only got so far. usefull but mixed quality. would be nice to get rid of.
  • expose is an experimental CLI command router and does fancy layout stuff, generate help, and outputcolor/plain/html etc. not essential but looks cool.
  • http is a http downloader with local file cache, it does e-tags and gzip and some http options (proxy etc). it prunes cache periodically. a small flaw is that it buffers downloads (works ok in our use-case).
  • git has wrappers to access blobs and trees from github over http, it uses the http cache.
• the typings folder is epic, i used to edit DT contributions as hobby.
• tests are a odd located: the actual cases are in src/spec, and src/test holds support code and custom assertions. some fixtures are located elsewhere.
• some tests are a little involved, but most TSD/CLI specific ones allow to add many cases by repeating the pattern: after verifying updates you can copy new test output to the fixtures easily.
• the gruntfile uses a custom grunt config wrapper/generator to create partial-tests. it's still just regular grunt running the tasks.
• most of actual logical interface is implemented in tsd/API.ts (should be exported and documented for re-use).
• API uses a structure from tsd/logic/Core.ts holding main services (index, http, git, downloaders etc).
• sub-systems usually know about the structure of Core and may reach into/through it to get their own work done (fun times but TypeScript compiler keeps it together), still try to glue in API.
• there is a settings/env/path thing in tsd/context/Context.ts. its not pretty but has the stuff. annoyingly it does IO and holds a copy of the tsd.json information in memory (race much).
• cli is setup from tsd/CLI.ts, it should use the API. API then glues together different services..
• .. so basically start there and follow the calls, but don't get stuck in custom utils.
• i felt there is a little too much code and too many dependencies and typed wrappers for what it does. although it is expandable in it's way; just slap on new services and add the glue.
• I'm Dutch and way back before 0.5 I just stopped writing games in Flash AS3..

.. anyway, most of it is kinda replacable except /src/tsd, and even that is basically just looping directory trees and parsing the paths for information. Then filter/compare and move blobs and paths around (don't underestimate amount of functionality needed though, it is a lot of things adding up, it's kinda like a mini VCS).

Cheers. 👍

[blakeembrey]

Thanks for the amazing introduction! 😄

[clavecoder]

VS15 and VSCode depend on TSD now, correct? It looks like you guys have a lot of rewrite to do. Can you get it done? Are you preparing for the io.js merge, for example? I'm feeling queasy about depending on tsd for my projects after all this discussion and the degree of rewrite in #170.

[blakeembrey]

VS15 and VSCode depend on TSD now, correct?

To be honest, I haven't heard or seen of this. Do you have a source?

Can you get it done?

Yes, absolutely. There's a lot of depth to the project, but it's not huge. I've just been incredibly busy since I started to rewrite and explore on the future branch.

Are you preparing for the io.js merge, for example?

That will definitely be prepared for, but it's unlikely we'll need to take any major action.

I'm feeling queasy about depending on tsd for my projects after all this discussion and the degree of rewrite in #170.

At the end of the day, it's completely up to you. TSD is currently stable on 0.6 and any future versions will have a completely seamless upgrade path. Even in #170, the first thing I implemented was a tsd init --upgrade command to seamlessly update to it - and that will always be the goal, making it as easy to use as possible.

In any case, the project does need refactoring and does need new ideas. Whether it's directly from #170 or we just pull some things from it, it's there to present some alternatives and directions. We need this because, as a new user even myself, it's very difficult to actually get started. In my first week using TypeScript I ran into issues getting definitions, then I ran into issues where they were out of date and had to hack in references to custom directories and copy and paste code, then I wanted to publish definitions of my own projects that depend on other definitions and that's not possible. Basically, I quickly ended up in a mess that tooling should have fixed. That fix is the goal of this.

[clavecoder]

Thanks for your very helpful answer. It's pretty clear that the promise and the reality of TS are a bit far apart. I think I still want to do it for core projects but it might be too much of a mess for application programmers to tackle right now.

[blakeembrey]

@kenbrubaker I've been using it for all new projects and it's been great, minus the few getting started humps. I actually think it's better for application developers than library maintainers right now, just because of the distribution issues around it still.

[mtraynham]

@blakeembrey Can I add a few concerns to how tsd link is addressed?

One of the exceptions that can be thrown by TypeScript is having a module conflict. You cannot have two angular.d.ts files. Based on that premise, you should not include tsd install-ed project definition files into the bower/package.json files, because a down-stream module may have two dependencies that both depend on angular.d.ts. It has to be resolved by either: a) a flat dependency tree or b) every project needs to ensure all d.ts dependencies are resolved and modules should only provide their own .d.ts. You may have hinted at this already.

One of the other issues with tsd link is how the path linking works. Currently, link updates the tsd.d.ts file to point at the ../bower_components/moduleA/moduleA.d.ts path. But say that particular module also depends on xyz.d.ts that I can retrieve from tsd install. It almost seems better to instead copy the "linked" files to the typings directory, rather than "linking" them with a path. I see it being no different than installing from tsd, where my "typings" directory has moduleA/moduleA.d.ts. This would better support a single structure for tsd (i.e. typings/module/module.d.ts)

[zakhenry]

@mtraynham the approach I have been taking is to not allow external dependencies and explicitly removing (via gulp) the references comment from my linked d.ts submodules. This does mean I have to make sure my main module has (at lease) the same dependencies as the submodule. It is a far from elegant solution but it works in the meantime.

I think that resolution of submodules should just work - ie duplicates are allowed but errors are not thrown. There is the possibility of conflicting typings files (eg if two versions are used), but typings files should theoretically be possible to merge (In PHPStorm when there is duplicate typings the type completion just reads from both files)

I think having a flat file structure would be a mistake as there is a chance that the submodule depends on a typings file that has a total conflict (same name, different api), or a versioning difference (angular 1.3 vs 1.4 for example).

[mtraynham]

@xiphiaz Fair points, but I don't know if I can agree with that last comment:

I think having a flat file structure would be a mistake as there is a chance that the submodule depends on a typings file that has a total conflict (same name, different api), or a versioning difference (angular 1.3 vs 1.4 for example).

bower already uses this structure, and npm 3 will be maximally flat. One of the more pertinent issues is resolving version conflicts for the browser. Even if the future is to package npm dependencies, instead of bower, it would be important that a module, with two dependencies that use different versions of angular, likely resolve to a single version of angular. With the current state of most client libraries, I don't forsee close scoped dependencies (one's that don't pollute the global namespace) being widely available... In this case, merging definitions seems a bit shady as what you ultimately bundle may not match the definition.

[blakeembrey]

@mtraynham @xiphiaz Absolutely agree with everything here. Unfortunately I haven't been able to work on it too much lately, but I think a flat structure will always be required.

However, in future iterations I think the goal will be to remove any possible conflict - for example, generating a single d.ts file from your dependency with its own dependencies inlined (without global modules). That would remove any conflict from ever happening. Having the declare module 'name' is kind of an anti-pattern here since it breaks interoperability with everything else, so the end goal would be to have that declare generated for you based on your dependencies. Of course, the only way things can work right now is with the declare, so we have a ways to go in between. Generating the d.ts will also be a huge improvement for module authors, since you'll finally be able to use the d.ts files generated by TypeScript - and I think this should be the primary goal of future work, making things work seamlessly.

As for copying versus implicitly linking and such, this shouldn't really matter. Preferably you'll publish your .d.ts library with no /// <reference>s and instead rely on imports and tsconfig.json.

[pleerock]

does not look like we have a good progress there

[blakeembrey]

@pleerock You'll have to elaborate on that statement, I'm not sure what you're talking about.

[eggers]

Any plans for something like --save-dev so that we can separate test dependencies into their own bundle?

[blakeembrey]

Yes, it's currently supported in branch I'm working on. There is no available release yet, as I'm closing up with the final piece which turned out to be quite tricky - that is, rewriting dependency typings into a single file to remove potential namespace conflicts.

[blakeembrey]

I spent a lot of time trying to solve the features in DefinitelyTyped alongside the things I needed to solve with this issue creation, but it's just not possible with the current structure of DT. To avoid trying to make breaking changes to a monolith project, like DT, I'm releasing the work for this issue under a separate repository and organization: https://github.com/typings/cli.

The goal of typings is to use external module declarations (not ambient module declarations) that can resolve over any source to create a typed dependency tree without conflicts. A lot of the requests made in TSD exist there and it's built closer to an NPM-like API. I welcome anyone to try it and see what they think, but the registry needs a lot more typings to be implemented before the real value is visible.

Live examples: https://github.com/TypeStrong/tsconfig/blob/master/typings.json, https://github.com/TypeStrong/ts-node/blob/master/typings.json and the registry at https://github.com/typings/registry.

[zakhenry]

@blakeembrey awesome! This is a much more maintainable pattern. What is the plan for those typings that you have added for some projects under the https://github.com/typings org? Is the idea that they will be removed when the respective source project include their own typings.json in their own project?

[blakeembrey]

@xiphiaz Absolutely. Everything is an external module declaration so it should actually be extremely straightforward to open a bunch of PRs for this already and those module will start working with TypeScript module resolution. Also, anyone can create a typing and make a PR to the registry to include their entry until the module merges it 😄 On top of that, having separate repos means that maintenance of typings should be much simpler than having to subscribe to issues and PRs for over a 1000 typings.

[zakhenry]

@blakeembrey good to hear, is the plan to always have the registry or be able to have the project define it's dependencies on tags, like how bower does not have an official registry - it just inherits the git tag semvers so the typings developer doesn't have to publish to a registry; they just have to maintain sane tags.

[blakeembrey]

@xiphiaz it can use tags or the commit hash. I'm using the commit hash right now since it's pretty much immutable which is what you want from the typings. The registry actually come after the CLI was done, but the plan is to always have the registry as the "official" source to find typings. There's also some other nifty things that can (only) be done by using a registry, such as auto-updating typings according the package.json and implicit installs of all missing typings.

Currently, the registry is used to populate the installation and it uses whatever the registry says for that version. To implement some of the more advanced features, I might have to implement a registry:... scheme (see typings/typings#28) so that you can know when typings are out of date with the installed versions in package.json.