We ended up with two different versions of the reify package (0.19.0 and
0.19.1) in the last build of the dev bundle, which seems to have caused
the test failures.

So that emails from other users are not exposed.

Follow-up to #10452.

…0577)

Fixes #10576.

* Update ecmascript-runtime-{client,server} to core-js@3.1.4.

Also added a polyfill for Symbol.asyncIterator to server, modern, and
legacy, which should fix #9897.

* Add a test of for-await-of async iteration.

This should verify that #9897 is fixed.

Instead of merely supporting ECMAScript module syntax via Reify, we should
really be compiling unanticipated modules (typically within node_modules)
using the same logic that the rest of the application uses.

Note: this processing applies only to .js files for now, since that's what
the ImportScanner works with.

Should help with #10563.

Now that we have the ability to invoke Babel via compiler plugins for
individual modules encountered by the ImportScanner, it's possible the
ImportScanner will try to compile @babel/runtime/helpers/* modules.

These modules are not safe to recompile because they use native syntax
(such as typeof) in ways that do not need additional transformation or
simulation, and would be broken by applying the usual Babel transforms.

It may bother you that this list of packages is hard-coded, or that it
might grow over time. To ease those concerns, I would say:

  1. We can release new versions of the babel-compiler and ecmascript
     packages whenever we need to.

  2. These particular npm packages belong in this list because Babel
     itself assumes they have already been compiled, so there shouldn't be
     (m)any other packages that fit that narrow criterion.

In other words, this is just a list of packages that must be left
untouched in order to bootstrap the Babel compiler system, and the
babel-compiler package is where the details of that system primarily
reside, so that's where we should put this list, until/unless we find a
better solution.

Case in point: @babel/runtime/helpers/esm/typeof.js uses ECMAScript module
syntax (import, export), but must not be compiled with transforms like
@babel/plugin-transform-typeof-symbol, since it's part of the runtime
library depended upon by that transform.

This logic is an extreme implementation detail for sure, but at least
babel-compiler is the only code that needs to know about this complexity.

…m-ImportScanner

Support full ECMAScript compilation of modules unexpectedly imported from node_modules.

…ile (#10586)

Need to update version "node-gyp" to "5.0.1", due to deprecated packages into 3.7.0

Follow-up to #10420, per this comment by @conorstrejcek:
#10420 (comment)

Should help with #10512.

Since PR #10592 updated node-gyp to the latest version, we should update
node-pre-gyp as well.

See my comment in issue #10512 for motivation:
#10512 (comment)

I think the changes in b0cb1ce are still
worthwhile because they ensure ops within a transaction have distinct
Timestamps, with the caveat that we should avoid assuming those Timestamps
are necessarily unique.

…ndle. (#10596)

Should fix #10595.

Code from the application `node_modules` directory becomes part of the
`modules` package, so that it can be imported by any other package that
uses the module system, regardless of package load order.

Now that we compile code from `node_modules` using `babel-compiler` and
`meteor-babel` (#10585), `node_modules` code requires the same runtime
environment as any other Meteor JS code. For the most part, this need is
satisfied by the `@babel/runtime/helpers/...` modules, which are also
defined in the `modules` package because they come from `node_modules`.
However, in the legacy bundle, `meteorBabelHelpers.sanitizeForInObject` is
used to fix buggy for-in iteration in older Internet Explorers.

Thankfully, this extra helper code does not need to be included in the
modern or server bundles, but only in legacy code.

#10522 (comment)

Example:

  "meteor": {
    "mainModule": ...,
    "testModule": ...,
    "nodeModules": {
      "recompile": {
        "very-modern-package": ["client", "server"],
        "alternate-notation-for-client-and-server": true,
        "somewhat-modern-package": "legacy",
        "another-package": ["legacy", "server"]
      }
    }
  }

The keys of the meteor.nodeModules.recompile configuration object are npm
package names, and the values specify for which bundles those packages
should be recompiled using the Meteor compiler plugins system, as if the
packages were part of the Meteor application.

For example, if an npm package uses const/let syntax or arrow functions,
that's fine for modern and server code, but you would probably want to
recompile the package when building the legacy bundle. To accomplish this,
specify "legacy" or ["legacy"] as the value of the package's property,
similar to somewhat-modern-package above. These strings and arrays of
strings have the same meaning as the second argument to
api.addFiles(files, where) in a package.js file.

This configuration serves pretty much the same purpose as symlinking an
application directory into node_modules, but without any symlinking:
https://forums.meteor.com/t/litelement-import-litelement-html/45042/8?u=benjamn

The meteor.nodeModules.recompile configuration currently applies to the
application node_modules directory only (not to Npm.depends dependencies
in Meteor packages). Recompiled packages must be direct dependencies of
the application.

In PR #10585, I tried enabling full compiler plugin processing for any
modules imported from `node_modules` that were not otherwise handled by
compiler plugins, but doing that for all packages was just too slow, not
to mention potentially dangerous for modules whose code cannot be safely
recompiled by Babel.

The primary reasons for wanting to recompile node_modules are

  1. to enable "native" ECMAScript `import`/`export` syntax (given that
     Node.js still does not fully support ESM syntax yet), and

  2. to compile standard syntax like `const`, `let`, classes, and arrow
     functions for legacy browsers.

The first goal can be achieved by compiling unanticipated `node_modules`
code with Reify, which is much faster than Babel, in part because Reify
can avoid doing any parsing when the source contains no `import` or
`export` identifiers. This compilation is also cached on disk, so its
impact should only be felt during cold builds after a `meteor reset`.

The second goal can be accomplished using the new `package.json`
configuration option `meteor.nodeModules.recompile`, which causes the
recompiled packages to be handled by the normal compiler plugins system,
so that the `ImportScanner`'s fallback compilation is not necessary.
Before this option was introduced, this recompilation behavior could be
achieved by symlinking application directories into `node_modules`, but
that always felt like an esoteric hack.

This ensures we wrap modules with a function to rename the `module`
identifier to something more reliable when the ImportScanner compiles
unanticipated modules imported from node_modules.

This partially reverts f0d39b8 by simply
including .js and .mjs modules in the server bundle, rather than
delegating to pure Node evaluation. In practice, whether or not the
package has a "module" entry point (or "type":"module") in its
package.json was not a perfect indicator of whether it should be compiled
with Reify and bundled, or left untouched and handled by Node.

Truly native modules (such as those with a .node file extension) should
always be handled by Node, so module.useNode() definitely still has a role
to play in the server bundle.

With appropriate meteor.nodeModules.recompile configuration, that is.

This partially reverts commit 4dfc741.

Some server packages, especially those that rely on __dirname or
__filename (e.g. puppeteer), simply cannot be included in the server
bundle, and must be evaluated natively by Node.

As long as we have a Module.prototype._compile hook to process natively
evaluated modules with Reify, module.useNode() can still benefit from ESM
import/export syntax.

This allows us to take advantage of the features.compileForShell option
when compiling code in the `meteor shell` REPL.

This saves us from having to install @babel/parser in the server bundle.

There is currently a bug in Reify that prevents it from loading this
parser by default, but we can work around that on the Meteor side for now.

We don't need to build a whole new dev bundle just for this upgrade, since
we've already worked around the bug that it fixes, but it will get picked
up the next time we build the dev bundle for Meteor 1.8.2.

…ckage.json-config

Support meteor.nodeModules.recompile package.json configuration option.

Now that we have the meteor.nodeModules.recompile configuration (#10603),
we don't need to guess which packages might need to be recompiled.

Acorn seems to have trouble parsing code that @babel/parser parses
successfully, such as the dist/esm.js file in markdown-to-jsx@6.10.2.

…0604)

Follow-up to #10420, per this comment by @conorstrejcek:
#10420 (comment)

Should help with #10512.

See my comment in issue #10512 for motivation:
#10512 (comment)

I think the changes in b0cb1ce are still
worthwhile because they ensure ops within a transaction have distinct
Timestamps, with the caveat that we should avoid assuming those Timestamps
are necessarily unique.

The meteor/tools/isobuild/resolver.js changes are the static half of the
puzzle. The runtime half was implemented in install@0.13.0 with this
commit: benjamn/install@233aa75

…cript

Convert tools/fs/files.js to TypeScript.

These methods return a StatWatcher object with various useful methods, but
@types/node apparently gets this completely wrong, assuming void.

This completes the TypeScript conversion of the tools/fs directory. 🎉

Unfortunately, this conversion triggered an error due to one of the
shortcomings of the Babel implementation of TypeScript:

SyntaxError: /tools/tool-env/profile.ts: Namespace not marked type-only declare. Non-declarative namespaces are only supported experimentally in Babel. To enable and review caveats see: https://babeljs.io/docs/en/babel-plugin-transform-typescript
  278 | }
  279 |
> 280 | export namespace Profile {
      |                  ^
  281 |   export let enabled = !! process.env.METEOR_PROFILE;
  282 |
  283 |   export function time<TResult>(bucket: string, f: () => TResult) {
    at File.buildCodeFrameError (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/core/lib/transformation/file/file.js:261:12)
    at transpileNamespace (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/plugin-transform-typescript/lib/namespace.js:25:25)
    at PluginPass.TSModuleDeclaration (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/plugin-transform-typescript/lib/index.js:271:32)
    at newFn (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/visitors.js:193:21)
    at NodePath._call (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/path/context.js:53:20)
    at NodePath.call (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/path/context.js:40:17)
    at NodePath.visit (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/path/context.js:88:12)
    at TraversalContext.visitQueue (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/context.js:118:16)
    at TraversalContext.visitMultiple (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/context.js:85:17)
    at TraversalContext.visit (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/context.js:144:19)

TypeScript is now supported by the actual TypeScript compiler, rather than
by @babel/preset-typescript: meteor/babel#25

babel/babel#10170 (comment)

As reported by @arggh in this comment:
#10522 (comment)

Should fix this problem reported by @arggh:
#10522 (comment)

Implement an official TypeScript compiler plugin.

This makes type checking possible without an external tool like VSCode.

Thanks to @menewman for tackling this small but important module!

The typescript package was previously installed as a dependency of
meteor-babel, but it clearly deserves to be a direct dependency.

Sometimes, before dev_bundle/bin/.meteor-commands.json has been written, a
command like `./meteor npx tsc` can fail because `npx` is not blessed in
the same way `node` and `npm` are. Since `npx` is a core tool that we want
folks to be able to use without confusion, I think it makes sense to
include it in the list of known subcommands.

For strings we convert to Uint8Array, which does not support forEach on
any version of IE, and is not polyfilled by default.

This reverts commit 5c32331.

This test is passing reliably on the release-1.8.2 branch, so we should
keep running it there.

* Updated code to use modern JS
* Added types
* Stopped using 2 underscore functions (1 remaining)

Among other benefits, this should address #10622.

https://nodejs.org/en/blog/release/v8.16.1/
https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V8.md#8.16.1

Most importantly, this change means that changes to files not used by the
server bundle will not trigger a server restart.

Fixes #10449 by implementing the strategy I described in this comment:
#10414 (comment)

The previous implementation simply avoided calling watchSet.addFile for
potentially unused files, trusting that addFile would be called later if
the file was eventually used. However, this strategy left the contents of
watchSet.files incomplete for tasks such as IsopackCache._checkUpToDate,
which require full information about all files, even the ones that might
not be used by the bundle. The new strategy maintains metadata about
potentially unused files in a separate data structure, which will be
merged/cloned/serialized/deserialized along with other WatchSet data.

Part of #10686.

Avoid restarting server in development when changed file(s) not used by server bundle.

Should fix #10674.
May help with #10608.

cc @houshuang @klaussner

Most of these changes came for free with the update of meteor-babel to
version 7.6.0, but a few remaining spots needed to be updated.

Because json5 is more tolerant of non-standards-compliant input, it has no
trouble with \t tab characters outside of string literals, so this change
should fix issue #10688.

Note that json5 is already installed in dev_bundle/lib/node_modules/json5,
because it is a dependency of @babel/core, so we don't need to rebuild the
dev bundle immediately, though it would be a good idea to do so before the
next beta/RC release.

Background: benjamn/optimism#50