Runtime Compiler API #3442

kitsonk · 2019-12-05T03:59:29Z

This PR provides 3 new Deno APIs which provide user/runtime access to the TypeScript compiler that is built into Deno. This should be a good base for creating interesting workloads on top of Deno.

kevinkassimo · 2019-12-05T04:09:54Z

bikeshedding: Deno.compileTS? (compile sounds more broad, even like compiling to binary, which might be a feature we support in the future)

kitsonk · 2019-12-05T04:42:57Z

bikeshedding: Deno.compileTS?

Well it isn't just TS, you can "compile" JavaScript and JSON with the current implementation. I think when we added the ability to register a custom compiler, that we would want to be able to just throw any of the supported media types at this and just have it return the results, irrespective of it being built in or not.

The only limitation I can see of the current API, is that when sources are provided (and they aren't resolved outside of Deno) the media type is determined by extension. There isn't a way to express a media type outside of that. An alternative version of the sources could be supported where it took an array of "source file" objects, which would allow specification of a media type, but that just increases the complexity.

kitsonk · 2019-12-06T20:36:10Z

Ok, the Deno.transpileOnly works now and the processing of compiler options that are passed also works. To make it functionally complete I need to do Deno.bundle. There is quite a bit of DRY that I need to address and other cleanups, as well as tests.

kitsonk · 2019-12-09T01:25:02Z

Ok, the PR is functionally complete now, but needs DRYing, clean-up, tests and docs.

kitsonk · 2019-12-11T04:59:57Z

~~Grrr... bundle_exports is broken and I don't know why...~~ ~~when I run the steps manually it actually works, but it fails as part of cargo test~~ ~~but I am not sure what I changed to cause it... 😡~~

It turned out it was a strong one, where root module specifiers sometimes included relative paths in the absolute URL (e.g. "file://foo/bar/../bar/baz.ts") but the resolved modules didn't, so the ability to determine the common root which is needed to figure out what module to instantiate in the bundle wasn't working. 🤷‍♂

cli/js/compiler_api.ts

bartlomieju · 2019-12-28T12:10:24Z

cli/js/compiler_api.ts

+ * @param sources An optional key/value map of sources to be used when resolving
+ *                modules, where the key is the module name, and the value is
+ *                the source content.  The extension of the key will determine
+ *                the media type of the file when processing.  If supplied,
+ *                Deno will not attempt to resolve any modules externally.


This API allows to programatically create TS programs programatically without writing file to disk before compilation 👍 nice

cli/js/compiler_imports.ts

std/manual.md

bartlomieju

One more small nitpick: js/compiler_*.ts - maybe move all compiler related files under common js/compiler/ directory (or js/compilers/ to resemble layout in Rust)?

All in all I think this PR brings a lot of great functionality that users are waiting for. I'm for landing this PR, but after Tokio 0.2 PR (#3418)

kitsonk · 2019-12-29T22:16:44Z

One more small nitpick: js/compiler_*.ts - maybe move all compiler related files under common js/compiler/ directory (or js/compilers/ to resemble layout in Rust)?

I thought about that, as that was my first instinct but then I remembered that @ry doesn't like sub-directories... 😁 so I was going to wait for some feedback before making that change.

nayeemrmn · 2019-12-29T22:45:08Z

cli/js/compiler_api.ts

+export function compile(
+  rootName: string,
+  sources?: Record<string, string>,
+  options?: CompilerOptions
+): Promise<[Diagnostic[] | undefined, Record<string, string>]> {


Diagnostics are null when there are none, so what would empty array diagnostics mean?

Are you getting an empty array? I believe I coded it so that it doesn't resolve with an empty array.

Why that kind of API? Empty array for no diagnostics follows naturally and is just as easy to check for. Unless this is some convention?

Empty array for no diagnostics follows naturally and is just as easy to check for.

Is it? Technically it is slightly slower, because property access is slower than a value check.

Unless this is some convention?

There are two conventions. One is the arguments to a callback convention, where err would be the first argument and undefined if there was no error. The second convention is the Rust enum Result type, where you have Err and Ok. While not directly translatable, a presence of a value would indicate that something didn't go ok.

So why would an empty array be preferable?

There are two conventions. One is the arguments to a callback convention, where err would be the first argument and undefined if there was no error.

When there is a single potential error, the no-error case is modelled with nullability. When there is (explicitly) a list of errors, the no-error case is modelled naturally as an empty list and nullability is redundant, isn't it? null | non-empty list is generally strange to see.

My first instinct compares it more to an stderr stream which is simply empty or not (especially when it's referred to as a diagnostic).

Also restructures the compiler TypeScript files to make them easier to manage and eventually integrate deno_typescript fully. Fixes denoland#2927

kitsonk · 2020-01-06T21:07:01Z

@ry rebased post rusty_v8 merge and still going strong.

ry

Hey @kitsonk, I really apologize for taking so long to get this in. My insufficient excuses for taking so long is that w've been refactoring lower-level code and it's difficult to switch modes when thinking about that.

My concerns with this patch are:

I keep thinking it would be nice to have the TypeScript compiler be less privileged and instead run as normal user code. This is clearly impossible in the near future - and it's not even clear if it would be possible under any circumstances. But it bothers me that we're unable to run the normal TypeScript compiler as user code still (like we do with, for example, prettier).
I'm slightly concerned with the mismatch between subcommand names and the APIs. "Deno.bundle" matches "deno bundle" but "Deno.compile" corresponds to "deno fetch".

That said, we do have a privileged compiler facilities which deviate from TS's in functionality. Currently they are only accessible from the command-line by executing Deno - certainly it's better to be able to programmatically access these APIs, as is achieved in this patch. These APIs which you've added are all accessed through the op interface, so I think this is all good.

This patch does add a bit of runtime code, so let's watch the benchmarks for changes in snapshot size or execution time regressions.

All that said - it LGTM - it's a well written patch with tests and docs.

Also restructures the compiler TypeScript files to make them easier to manage and eventually integrate deno_typescript fully.

kitsonk force-pushed the runtime_compiler branch from c491f49 to e64ecdf Compare December 6, 2019 20:33

kitsonk force-pushed the runtime_compiler branch from e64ecdf to 83d855f Compare December 9, 2019 01:22

kitsonk force-pushed the runtime_compiler branch from f24c9fb to 0047cc9 Compare December 11, 2019 04:27

kitsonk force-pushed the runtime_compiler branch 2 times, most recently from a2eadfb to ed065bd Compare December 13, 2019 03:55

kitsonk marked this pull request as ready for review December 13, 2019 04:01

kitsonk force-pushed the runtime_compiler branch from 920b6f7 to a16d4cd Compare December 13, 2019 05:15

kitsonk changed the title ~~[WIP] Runtime Compiler API~~ Runtime Compiler API Dec 13, 2019

nayeemrmn mentioned this pull request Dec 24, 2019

Is it possible to provide a cli command to convert ts file to js file ? #3543

Closed

bartlomieju mentioned this pull request Dec 26, 2019

Expose DENO_DIR #3551

Closed

bartlomieju reviewed Dec 28, 2019

View reviewed changes

cli/js/compiler_api.ts Show resolved Hide resolved

bartlomieju reviewed Dec 28, 2019

View reviewed changes

cli/js/compiler_imports.ts Show resolved Hide resolved

bartlomieju reviewed Dec 28, 2019

View reviewed changes

std/manual.md Outdated Show resolved Hide resolved

bartlomieju approved these changes Dec 28, 2019

View reviewed changes

bartlomieju requested a review from ry December 28, 2019 12:24

kitsonk force-pushed the runtime_compiler branch from a16d4cd to bb05b2b Compare December 29, 2019 21:56

nayeemrmn reviewed Dec 29, 2019

View reviewed changes

kitsonk force-pushed the runtime_compiler branch from da30f57 to a755177 Compare December 30, 2019 19:32

kitsonk added 5 commits January 7, 2020 06:13

Runtime compiler API.

54b6526

Also restructures the compiler TypeScript files to make them easier to manage and eventually integrate deno_typescript fully. Fixes denoland#2927

Add runtime compiler APIs to manual

72892fe

Minor format issue

7c23e3d

Update manual based on feedback

8d7a2b5

Fix a couple clippy issues

1384faf

Updates for rebase

6e60373

kitsonk force-pushed the runtime_compiler branch 2 times, most recently from 67a66f1 to 6e60373 Compare January 6, 2020 19:30

ry approved these changes Jan 8, 2020

View reviewed changes

ry merged commit d325566 into denoland:master Jan 8, 2020

bartlomieju mentioned this pull request Jan 8, 2020

refactor: cleanup cli/js/compiler.ts #3347

Closed

rsp mentioned this pull request Jan 15, 2020

Public API for compilers #1739

Open

klaushvam mentioned this pull request Jan 17, 2020

Inconsistency between types and code in new Compiler API #3702

Closed

caspervonb pushed a commit to caspervonb/deno_std that referenced this pull request Jan 21, 2021

Runtime Compiler API (denoland/deno#3442)

cbd00ba

Also restructures the compiler TypeScript files to make them easier to manage and eventually integrate deno_typescript fully.

caspervonb pushed a commit to caspervonb/deno_std that referenced this pull request Jan 24, 2021

Runtime Compiler API (denoland/deno#3442)

5a35c8b

Also restructures the compiler TypeScript files to make them easier to manage and eventually integrate deno_typescript fully.

caspervonb pushed a commit to caspervonb/deno_std that referenced this pull request Jan 24, 2021

Runtime Compiler API (denoland/deno#3442)

8ee5e2c

Also restructures the compiler TypeScript files to make them easier to manage and eventually integrate deno_typescript fully.

caspervonb pushed a commit to caspervonb/deno_std that referenced this pull request Jan 24, 2021

Runtime Compiler API (denoland/deno#3442)

64b58c9

Also restructures the compiler TypeScript files to make them easier to manage and eventually integrate deno_typescript fully.

caspervonb pushed a commit to caspervonb/deno_std that referenced this pull request Jan 31, 2021

Runtime Compiler API (denoland/deno#3442)

871a4b5

Also restructures the compiler TypeScript files to make them easier to manage and eventually integrate deno_typescript fully.

caspervonb pushed a commit to caspervonb/deno_std that referenced this pull request Jan 31, 2021

Runtime Compiler API (denoland/deno#3442)

b66497c

Also restructures the compiler TypeScript files to make them easier to manage and eventually integrate deno_typescript fully.

caspervonb pushed a commit to caspervonb/deno_std that referenced this pull request Jan 31, 2021

Runtime Compiler API (denoland/deno#3442)

ae439b1

Also restructures the compiler TypeScript files to make them easier to manage and eventually integrate deno_typescript fully.

caspervonb pushed a commit to caspervonb/deno_std that referenced this pull request Jan 31, 2021

Runtime Compiler API (denoland/deno#3442)

642937a

Also restructures the compiler TypeScript files to make them easier to manage and eventually integrate deno_typescript fully.

caspervonb pushed a commit to caspervonb/deno_std that referenced this pull request Feb 1, 2021

Runtime Compiler API (denoland/deno#3442)

c5ffa6c

Also restructures the compiler TypeScript files to make them easier to manage and eventually integrate deno_typescript fully.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Runtime Compiler API #3442

Runtime Compiler API #3442

kitsonk commented Dec 5, 2019 •

edited

Loading

kevinkassimo commented Dec 5, 2019 •

edited

Loading

kitsonk commented Dec 5, 2019

kitsonk commented Dec 6, 2019

kitsonk commented Dec 9, 2019

kitsonk commented Dec 11, 2019 •

edited

Loading

bartlomieju Dec 28, 2019

bartlomieju left a comment

kitsonk commented Dec 29, 2019

nayeemrmn Dec 29, 2019

kitsonk Dec 30, 2019

nayeemrmn Dec 30, 2019

kitsonk Dec 30, 2019

nayeemrmn Dec 30, 2019

kitsonk commented Jan 6, 2020

ry left a comment

Runtime Compiler API #3442

Runtime Compiler API #3442

Conversation

kitsonk commented Dec 5, 2019 • edited Loading

kevinkassimo commented Dec 5, 2019 • edited Loading

kitsonk commented Dec 5, 2019

kitsonk commented Dec 6, 2019

kitsonk commented Dec 9, 2019

kitsonk commented Dec 11, 2019 • edited Loading

bartlomieju Dec 28, 2019

Choose a reason for hiding this comment

bartlomieju left a comment

Choose a reason for hiding this comment

kitsonk commented Dec 29, 2019

nayeemrmn Dec 29, 2019

Choose a reason for hiding this comment

kitsonk Dec 30, 2019

Choose a reason for hiding this comment

nayeemrmn Dec 30, 2019

Choose a reason for hiding this comment

kitsonk Dec 30, 2019

Choose a reason for hiding this comment

nayeemrmn Dec 30, 2019

Choose a reason for hiding this comment

kitsonk commented Jan 6, 2020

ry left a comment

Choose a reason for hiding this comment

kitsonk commented Dec 5, 2019 •

edited

Loading

kevinkassimo commented Dec 5, 2019 •

edited

Loading

kitsonk commented Dec 11, 2019 •

edited

Loading