Webpack plugin

[buu700]

Nice work @fabiandev; I'm really excited to test this out at some point!

Anyway, just throwing this idea out there in case you or anyone else have time for it at some point: a webpack plugin to automatically handle inserting the import into each source file and building with tsr.

In addition to making it a lot simpler to get started by piggybacking on existing tooling, this would abstract away most of the work required for integrations like angular/angular-cli#6763 (at least for non-prod builds; some extra work might be needed to integrate tsr into Angular's AOT compiler).

[fabiandev]

Thank you! I already thought about providing a webpack wrapper around ts-runtime. However, at this point I don't really have time to investigate, but I'm open to any discussions, suggestions and pull requests.

By the way, the import import _t from 'ts-runtime/lib', based on the provided settings, is automatically inserted on top of every source file. Also the declaration file that holds external and ambient declarations is, by default, imported in every entry file. If this is what you were referring to?

But I definitely see the benefits of your proposal!

[buu700]

Thank you! I already thought about providing a webpack wrapper around ts-runtime. However, at this point I don't really have time to investigate, but I'm open to any discussions, suggestions and pull requests.

Awesome. I don't need it super urgently, but may take you up on submitting a PR at some point in the future if it hasn't already been done by someone else and/or been implemented in the Angular CLI by then.

By the way, the import import _t from 'ts-runtime/lib', based on the provided settings, is automatically inserted on top of every source file. Also the declaration file that holds external and ambient declarations is, by default, imported in every entry file. If this is what you were referring to?

Ah, yeah, thanks for clarifying that. I'd misunderstood "On top of every source file, the runtime type checking library is imported" as "... must be imported by you", not "... is imported as one of the following list of transformations applied by tsr". That makes a lot more sense now.

[fabiandev]

Okay, great!

Just to make it clearer, when transforming a file with the following code:

let a: string;

this is what you get by default:

import './tsr-declarations';
import _t from 'ts-runtime/lib';

let _aType = _t.string(), a;

What I'll have to change is, that tsr-declarations.js should not be imported/emitted if it is empty. However, it may be turned off in any case with --excludeDeclarationFile via the CLI.
Also it should be possible to opt out of the ts-runtime/lib import, I think.

Just noticed, that I will also have to align the CLI options with the API options. The project is still work in progress, but I'm glad for any feedback or issues spotted :)

[andykais]

any update on this? I would love to see a webpack integration (or even advice on how to integrate the existing tsr with a webpack build)

[fabiandev]

I will try to provide a guide on how to integrate this project in a webpack process soon. Please also see #19.

[fabiandev]

I just cam across ts-runtime-loader, didn't check it, but does it work for you?

[andykais]

so this does work to a degree, however its doing some really weird things behind the scenes (partially because of how ts-runtime works). It actually patches node's fs with an in memory file system, so that the files that ts-runtime's transform function writes are not to disk. Then it re-runs the whole of webpack a second time to replace the assets that ts-loader stripped of types back into runtime type checks. I actually ran into an error with it when I added multiple files from the same directory which had separate tsr-declaration files. A lot of what this loader has to do is hacky, and although its awesome someone is working on a solution, it is not a stable one. I would still petition that ts-runtime add a transformStrings function which input/outputs strings instead of files.

This would allow me (or anyone) to create a loader more in line with webpacks design principles of modularity and simplicity. This would keep different runtime files separated, and allow webpack to handle de-duping modules on its own. As another bonus, it could work with babel-loader and the @babel/typescript plugin

[fabiandev]

I see — as mentioned before, I didn't check the code or try it out. The thing with string in/string out is a bit tricky with ts-runtime, since it relies on resolving types, to be able to create the correct runtime type checks.

This means that we have to provide the full project, either via an entry file from the file system, or via a representation of it via transformReflection. This allows you to pass a FileReflection array, where the compiler will look for the file contents, instead of accessing the file system.

The latter could probably be used with webpack.

[fabiandev]

By the way, both transform functions, transform and transformReflection, return FileReflection[], which contains all the resulting transformed and compiled code.

[andykais]

The latter does sound like a possible solution with webpack. Loaders are supposed to be stateless, but it would be possible to read all files once, then reuse that for the remainder of the build. I dont believe there is a way around this, since webpack only expects one file in and one file out.

This leads to another issue though, which is, how will webpack know about the other files that are created by transform? Depending on what is required by a transformed file, it could have created tsr-declaration.js as well as other files. I see one possible solution, slightly more inline with what webpack prefers, and less likely to break other loaders, should something go wrong.

ts-runtime-loader/
  index.js
  helper-loader.js

// index.js
const path = require('path')
const helperLoader = require('./helper-loader')

const wrapRequires = (source, currentResourceFilename) => {
  source.replace(
    LOCAL_REQUIRE_REGEXP,
    moduleImport => `./${currentResourceFilename}.${moduleImport}!=!${helperLoader}!${currentResourceFilename}?{content:"${source}",isJSON5:true}`
  )
}
module.exports = function(source) {
  const transformed = transformStrings(source)
  const wrappedRequires = transformed.map(tr => {
    tr.text.replace(
      LOCAL_REQUIRE_REGEXP,
      moduleImport => {
        const moduleFile = transformed.find(tr => tr.name === moduleImport)
        if (moduleFile) {
          const escapedSource = escape(source)
          return `require('./${currentResourceFilename}.${moduleImport}!=!${helperLoader}!${currentResourceFilename}?{content:"${escapedSource}",isJSON5:true}')`
        } else {
          return moduleImport
        }
      }
  })
  const currentResource = wrappedRequires.find(tr => tr.name === this.resourcePath)
  return currentResource.text
}

// helper-loader.js
const { parseQuery } = require("loader-utils")

module.exports = function(source) {
  const { content } = parseQuery(this.resourceQuery)
  return content
}

the implementation isnt perfect here, but the jist of it is that we can transform each require statement that matches one of the other transformed files into a webpack loader request which holds the files content inside it as a query. Then, when webpack uses the helper-loader loader to resolve those other files, we return the content from the "loader request" query. This is webpacks way of creating new resources outlined here https://webpack.js.org/api/loaders/#inline-matchresource.

There are still a few small problems, I have a circular issue, I need to transform files before I set them as query parameters, but I need to put them as query parameters in order to transform them. The second issue being I dont know if I can properly escape all valid javascript into a query parameter.

The good news is though, webpack can still resolve which requires are common, and this requires no extra access to the filesystem.

[fabiandev]

This sounds like a way to go, thanks for putting effort into exploring that! If you make further progress I'd be happy about any contribution to this project.

[Venryx]

I'm considering working on this some myself, though am not sure yet whether the complexity level makes it feasible for me to complete with the motivation level I have.

In any case, I thought I'd join the conversation, and try to get a picture on what challenges are left to solve for a proper webpack loader to be created.

Regarding:

The second issue being I dont know if I can properly escape all valid javascript into a query parameter.

Couldn't you simply encode each character in the Javascript code into its number/character-code, separated by some known-safe character?

For example:

function EncodeAsQuerySafeString(code) {
    const codeAsNumbers = code.split("").map(ch=>ch.charCodeAt(0));
    return codeAsNumbers.join(",");
}
EncodeAsQuerySafeString("alert('test1');");
// result: "97,108,101,114,116,40,39,116,101,115,116,49,39,41,59"

And to decode:

function DecodeQuerySafeString(str) {
    return str.split(",").map(charCode=>String.fromCharCode(charCode)).join("");
}
DecodeQuerySafeString("97,108,101,114,116,40,39,116,101,115,116,49,39,41,59");
// result: "alert('test1');"

EDIT: You should also be able to use base64 encoding, which is more compact: https://stackoverflow.com/a/2820329/2441655