[WIP] Add support for --types #729

kitsonk · 2018-09-11T12:07:13Z

Resolves #632

This PR allows the following:

$ deno --types

Which will pipe the default library for deno to the console, which deno uses to validate code against internally. End users can then pipe this to a file for use in their projects and use within an editor. For example, an end user could do this:

$ deno --types > lib.deno.d.ts

And then configure a tsconfig.json with the following, which would then mirror the way deno will evaluate the code:

{
  "compilerOptions": {
    "allowJs": true,
    "module": "amd",
    "moduleResolution": "node",
    "noEmit": true,
    "noLib": true,
    "target": "esnext"
  },
  "files": [
    "./lib.deno.d.ts",
    "./myProject.ts"
  ]
}

Note I cannot think of a good way of creating a unit test for this at the JS level and it took me manually doing the export and checking the file to validate it and make sure it works. Some changes to errors.ts had caused some issues with the resulting types, which were only detectable that way. It might be possible to run the types through tsc to validate them (essentially do what is suggested above as part of the test suite) but I wanted to get feedback before pursuing that.

kitsonk · 2018-09-11T16:32:05Z

I have created and example here: https://github.com/kitsonk/deno_example using the export via --types for a project that passes the type checking under tsc and gives intellisense in VSCode. It is worth checking it out to see what there are 3 external libraries that we inline in deno that have to be present when using tsc or VSCode to work properly. They are:

@types/flatbuffers
@types/text-encoding
typescript

@ry the CI failure seems to be a temporary Travis environmental issue versus an issue with the PR.

kitsonk · 2018-09-11T16:51:04Z

Just further to the above, I looked at the three externals. Two of them could just be appended onto the --types output, which would be @types/text-encoding and typescript, because they are both ambient/global declarations. While @types/flatbuffers is a UMD/modular declaration which cannot be output without some sort of manipulation to turn it into an ambient/global declaration.

The option could be to pull the type definition in directly to the project and do the conversion at source. I do not expect a lot of churn in flatbuffers that would make it difficult to maintain over the long term.

ry · 2018-09-11T16:55:16Z

@kitsonk Nice work! I was able to try it and it works. (Also restarted the CI server and it passed.)

Looking at the first couple lines of the output:

// Copyright 2018 the Deno authors. All rights reserved. MIT license.
/// <reference no-default-lib="true"/>
/// <reference lib="esnext" />
declare module "js/assets" { }
declare module "js/types" {
    export type TypedArray = Uint8Array | Float32Array | Int32Array;
    export interface ModuleInfo {
        moduleName: string | null;
        filename: string | null;
        sourceCode: string | null;
        outputCode: string | null;
    }

The declare module "js/types" suggests that inside Deno one could import "js/types", but this isn't the case. Additionally it is exporting a bunch of internal types from msg_generated.ts.

ry · 2018-09-24T14:23:35Z

@kitsonk can you give an update on the status of this?

kitsonk · 2018-09-24T21:55:42Z

@ry I am still working on it, it is just slow... manipulating the AST and doing the sort of dependency analysis is a lot more complicated and while the AST is there and easy to manipulate.

If we wanted to, the "ugly" working version we could commit while the elegant build is going to be a bit longer.

kitsonk · 2018-09-25T07:06:20Z

@ry I have updated this PR with a rebasing of the original --types support and the addition of my current WIP. It is still very rough, but sort of gives an idea of where I am headed.

kitsonk · 2018-09-27T07:32:37Z

@ry I know this is taking a long time, but I am making progress. Here is the output now of --types:

https://gist.github.com/kitsonk/6c11763faf833f5cf4ec14f58657d8f8

I think I am now finally on the right path to solve this problem and deliver something that will just adjust with deno for the foreseeable future.

ry · 2018-09-27T20:24:28Z

@kitsonk cool - keep me updated

kitsonk · 2018-09-29T12:10:34Z

Ok, it took a while, but now this is functionally complete, I just need to integrate it into the build. An example of what is being generated is available here: https://gist.github.com/kitsonk/6c11763faf833f5cf4ec14f58657d8f8

ry · 2018-09-30T14:38:54Z

@kitsonk Looks quite nice now! I think it could still be clearer if the number of these @internal modules was reduced, and interfaces/function signatures were moved directly to where they were exported. But I think that's something we can iterate on top of this.

How can I run it manually?

ry · 2018-09-30T14:40:37Z

tools/ts/build_lib.ts

+// flag, so we will just create a special `@internal` namespace and add in
+// just what we need "manually"
+const internalModule = libDTs.addNamespace({
+  name: `"@internal"`,


@internal/generated ?

ry · 2018-09-30T14:43:48Z

tools/ts/build_lib.ts

@@ -0,0 +1,253 @@
+import { join } from "path";


This file seems to be a top-level main script that depends on cwd. Because it's a lot of code, it would be good to put it inside a function with arguments as all the various inputs (e.g. cwd). This will potentially let us test it independently later.

I was going to move it to args passed from the build script, because depending on other environments it simply wouldn't work (like when the build path is outside of the root of deno like on travis) but I was just "hacking" it to work until I was ready to integrate it, which is what I am doing now.

ry · 2018-09-30T16:16:48Z

I wonder if there's any way we can validate that the generated declaration file corresponds to the actual runtime types?

kitsonk · 2018-09-30T20:43:00Z

@ry to run manually at the moment, from the root of the deno directory:

$ node ./tools/ts/build

Hmmm... I wonder how easy it would be to combine into one @internal namespace. The problem we could have is name collisions. I would suggest though that we figure out how to get this landed and then try to improve upon it.

As far as testing/validating runtime types, the basic level would be to author a .ts file that would compile without any emit errors. More advanced would authoring actual type assertion unit tests. I tried a long time ago to adapt some of the stuff that TypeScript uses internally for testing, but it was klunky. There maybe an opportunity with ts-simple-ast to write something cleaner which allows some better type assertions. Again, my advice would be for stage one, ensure that the unit tests for globals, deno and the compiler cover the APIs functionally authored in TypeScript, which will implicitly test the types and then we can refine.

kitsonk · 2018-10-01T01:32:06Z

@ry I think this is now complete and build_lib has been fully integrated into Deno and the previous generation of the default library has been removed. ./tools/build.py :gen_lib will just do what is required to output the the library.

Once there is a branch for the added packages on third_party, I can do a final rebase and organise the commits into a more logical grouping.

ry · 2018-10-01T15:46:55Z

tools/ts/build.js

+  skipProject: true
+});
+
+require("./build_lib");


The name of this file and directory are too generic tools/ts/build.js. Is there something more descriptive we could use?

Maybe tools/ts_declaration_builder/main.js

ry · 2018-10-01T15:47:27Z

tools/ts/build.js

+    strict: true,
+    target: "esnext"
+  },
+  skipProject: true


Why skipProject? Add a comment?

ry · 2018-10-01T15:49:52Z

tools/ts/build_lib.ts

+      silent = true;
+      break;
+  }
+});


I'd rather not have top level scripts here. Can you wrap this all in a main function, and pass process.argv to it?

ry · 2018-10-01T15:51:34Z

tools/ts/build_lib.ts

+
+// Add the `compiler` module
+addNamespaceDeclaration(
+  "compiler",


Why isn't the compiler module internal?

I researched it. See #876

ry · 2018-10-01T15:54:42Z

tools/ts/ast_util.ts

+const MSG_GENERATED_SPECIFIER = "gen/msg_generated";
+export const INTERNAL_GENERATED_SPECIFIER = "@internal/generated";
+
+let rootPath = process.cwd();


Superfluous since setRootPath() is called elsewhere? Also please avoid top-level functionality - it would be nice if everything was wrapped in functions so we can test it.

ry · 2018-10-01T15:56:45Z

tools/ts/ast_util.ts

+import { relative } from "path";
+import { readFileSync } from "fs";
+import Project, {
+  ExportDeclaration,


What does this syntax mean? Why is Project not inside the braces?

FWIW, I've been thinking to move Project to a named export and not have a default export (I'm the library author). It's the default export of the library right now and it's really only that way for historical reasons.

And it is supported ES6 import syntax. It is default import and then named imports. You can blame TC39 for default. Something that sounded good on paper...

Project is also a named export, I will convert to that.

oops, it isn't... DOH!

I'll do an update that adds it as a named export tonight. I'm not a fan of default imports/exports either.

Edit: Published in 16.0.2. Seems like my declaration file was lying that it was a named export! Now it's actually exported as a named export as well and added a deprecation notice to default.

kitsonk · 2018-10-02T20:49:32Z

@ry I updated my branch of third_party and now get the following:

Showing 1,557 changed files with 792,741 additions and 653,799 deletions.

So a net add of 139k SLOCs.

We need this specific version because ts-simple-ast depends on it. See denoland#729 (comment)

We need this specific version because ts-simple-ast depends on it. See #729 (comment)

kitsonk · 2018-10-08T01:33:12Z

@ry I think this is close to being ready to go, with another review and updating third_party. The changes have been squashed into a series of more logical standalone commits.

ry · 2018-10-08T15:57:25Z

tools/ts_library_builder/main.js

+
+// This modules bootstraps ts-node to allow on the fly transpiling of
+// TypeScript while under NodeJS and performs an interpretation of
+// process.argv to pass to the build_library main function.


I think it would be nice to replace this with a more general tools/ts_library_builder/README.md

Let me take a stab at describing this - please modify

This tool allows us to produce a single TypeScript declaration file that describes the complete Deno runtime, including global variables and the built-in "deno" module. The output of this tool, lib.deno_runtime.d.ts, serves several purposes: 1. It is passed to the typescript compiler js/compiler.ts, so that TypeScript knows what types to expect. 2. It is outputted to stdout by `deno --types`, so that users can easily have access to the complete declaration file. Editors can use this in the future to perform type checking. 3. Because JSDocs are maintained, this serves as a simple documentation page for Deno. We will use this file to generate HTML docs in the future. ## Design Ideally we wouldn't have to build this tool at all, and could simply use tsc to output this declaration file. However, we were not able to get it to output a clean declaration using `emitDeclarationOnly` and `outFile`. [.... maybe add more detail here.] ## TODO Currently the Jsdoc associated with interfaces is not properly emitted

ry · 2018-10-08T16:00:11Z

tools/ts_library_builder/build_library.ts

+    `${basePath}/js/globals.ts`
+  ]);
+
+  // TODO check diagnostics for any issues


Will diagnostics be printed to stdout? It would be good to have a simple assert here that fails if there are any diagnostics.

Note: The semantic, syntactic, global, options, and declaration (since declaration: true) diagnostics are available on emitResult below by calling emitResult.getDiagnostics().

Generally, diagnostics can be retrieved by calling inputProject.getPreEmitDiagnostics(), which returns the semantic, syntactic, global, config file parsing, options, and if enabled the declaration diagnostics. For specific diagnostics, they can be retrieved from the program (ex. inputProject.getProgram().getSemanticDiagnostics()).

Yeah, I was thinking about this and figured that actually I probably need to do this. In order to get it to work, I need to properly configure the output project so that it can resolve the other types it needs in order to be fully valid.

ry · 2018-10-08T16:01:32Z

tools/ts_library_builder/build_library.ts

+  // actually creating it, only in memory at this stage.
+  const libDTs = outputProject.createSourceFile(outFile);
+
+  // Deal with `js/deno.ts`


Consider breaking this section out into its own function.

ry · 2018-10-08T16:01:48Z

tools/ts_library_builder/build_library.ts

+    console.log(`Created module "deno".`);
+  }
+
+  // Deal with `js/globals.ts`


Consider breaking this section out into its own function.

ry

Great work - I'm ready to land this after a few things:

Add a readme file, I outlined one above.
Let's get all the changes to js/ landed separately (like @internal stuff) except for where you actually hook into lib.deno_runtime.d.ts

ry · 2018-10-08T16:08:52Z

BUILD.gn

+# Generates the core TypeScript type library for deno that will be
+# included in the runtime bundle
+
+run_node("gen_runtime_lib") {


Rename to deno_runtime_declaration

ry · 2018-10-08T16:09:28Z

BUILD.gn

-run_node("gen_declarations") {
+# Generates the core TypeScript type library for deno that will be
+# included in the runtime bundle
+


Remove \n

ry · 2018-10-08T16:09:56Z

BUILD.gn

+    args += [
+      "--debug",
+    ]
+  }


kitsonk · 2018-10-10T05:25:23Z

@ry ok, I think I have addressed all feedback. Also I am now checking diagnostics on the final lib.deno_runtime.d.ts and have checked that if there is some sort of error in there, the diagnostics get logged plus it will fail the generation of the library, and therefore fail the build.

There really aren't any commits in here now that make sense to be independent, unless you want to land the updates to third_party and the support for --types as seperate PRs, they are at least clean commits as part of this PR.

ry · 2018-10-10T13:34:16Z

This patch has my approval - thank you @kitsonk !

I will do another round of reviews with Bert, due to the size.

Continued in #952

We need this specific version because ts-simple-ast depends on it. See denoland/deno#729 (comment)

kitsonk changed the title ~~Add support for --types~~ [WIP] Add support for --types Sep 12, 2018

Fausto95 approved these changes Sep 14, 2018

View reviewed changes

kitsonk force-pushed the types branch from 6098cfe to 8855af5 Compare September 16, 2018 22:04

kitsonk force-pushed the types branch from 8855af5 to 69aef9b Compare September 25, 2018 07:04

kitsonk mentioned this pull request Sep 28, 2018

NamespaceDeclaration should support import and export declarations dsherret/ts-morph#436

Closed

kitsonk force-pushed the types branch 2 times, most recently from f0b4bee to 88baa9e Compare September 29, 2018 12:08

ry reviewed Sep 30, 2018

View reviewed changes

kitsonk force-pushed the types branch from 88baa9e to 531653d Compare October 1, 2018 01:27

kitsonk force-pushed the types branch from 531653d to 5e1e3de Compare October 1, 2018 01:34

ry reviewed Oct 1, 2018

View reviewed changes

kitsonk force-pushed the types branch from 7b02eeb to 0bbeb37 Compare October 2, 2018 22:45

ry added a commit to ry/deno that referenced this pull request Oct 3, 2018

Upgrade TS to 3.0.3

c55825d

We need this specific version because ts-simple-ast depends on it. See denoland#729 (comment)

ry mentioned this pull request Oct 3, 2018

Upgrade JS deps #891

Merged

ry added a commit that referenced this pull request Oct 3, 2018

Upgrade TS to 3.0.3

522ddbf

We need this specific version because ts-simple-ast depends on it. See #729 (comment)

kitsonk mentioned this pull request Oct 4, 2018

Move Platform from types to platform #906

Merged

kitsonk force-pushed the types branch 2 times, most recently from f6ef61a to 7763135 Compare October 8, 2018 01:30

kitsonk mentioned this pull request Oct 8, 2018

setHasDeclareKeyword does not work on all ambientable nodes dsherret/ts-morph#460

Closed

ry reviewed Oct 8, 2018

View reviewed changes

BUILD.gn

args += [

"--debug",

]

}

Copy link

Member

ry Oct 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

kitsonk added 4 commits October 10, 2018 16:05

Add ts-simple-ast ts-node @type/prettier to third_party

b3a8175

Add support for --types flag

2eff6a2

Add ts_library_builder

d38745f

Migrate from lib.deno_runtime.d.ts

92a32e8

kitsonk force-pushed the types branch from 99d19df to 92a32e8 Compare October 10, 2018 05:19

ry mentioned this pull request Oct 10, 2018

Add --types #952

Closed

ry closed this Oct 10, 2018

kitsonk mentioned this pull request Oct 11, 2018

Add support for --types #960

Closed

Tzikas added a commit to Tzikas/squares that referenced this pull request Sep 10, 2019

Upgrade TS to 3.0.3

d693e4f

We need this specific version because ts-simple-ast depends on it. See denoland/deno#729 (comment)

kitsonk deleted the types branch August 2, 2022 04:42

[WIP] Add support for --types #729

[WIP] Add support for --types #729

Conversation

kitsonk commented Sep 11, 2018 • edited

kitsonk commented Sep 11, 2018

kitsonk commented Sep 11, 2018

ry commented Sep 11, 2018

ry commented Sep 24, 2018

kitsonk commented Sep 24, 2018

kitsonk commented Sep 25, 2018

kitsonk commented Sep 27, 2018

ry commented Sep 27, 2018

kitsonk commented Sep 29, 2018

ry commented Sep 30, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ry commented Sep 30, 2018

kitsonk commented Sep 30, 2018

kitsonk commented Oct 1, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ry Oct 1, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherret Oct 1, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherret Oct 1, 2018 • edited

Choose a reason for hiding this comment

kitsonk commented Oct 2, 2018

kitsonk commented Oct 8, 2018

ry Oct 8, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherret Oct 8, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ry Oct 8, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ry left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kitsonk commented Oct 10, 2018

ry commented Oct 10, 2018

kitsonk commented Sep 11, 2018 •

edited

ry Oct 1, 2018 •

edited

dsherret Oct 1, 2018 •

edited

dsherret Oct 1, 2018 •

edited

ry Oct 8, 2018 •

edited

dsherret Oct 8, 2018 •

edited

ry Oct 8, 2018 •

edited

ry left a comment •

edited