feat: decentraland libraries & enable sourcemaps support for scenes

[menduz]

What?

This pull request adds debugging capabilities to the generated ECS code. Source maps and breakpoints are now working properly.

build-ecs

• facing a problem in the source map generation, ugilfy-js is replaced by terser. keeping strict ES5 as input/output.
• added support for bundleDependencies in the scene's package.json to load "Decentraland Libraries" (more on libraries in a moment)
• added support to compile both "Decentraland libraries" and scenes
• added proper diagnostics for every file in every project, previously it only reported crashes and errors in "index.ts" only.
• added -p | --project argv to run in a folder
• two files are now emited every time: scene.js and scene.min.js (this should changed to emit the minified only when NOT watching changes OR prod mode)
• many debugging checks were added (mostly to ensure file existence and consistent files i.e. [scene.json#main] == [package.json#main])
• libraries added to bundleDependencies are automatically added in the tsconfig.json when necessary to compile correctly with proper types.

decentraland-amd

• it is now generated with inline sourcemaps and not minified
• toUrl renamed to _toUrl due to definitions name clashing in .d.ts generation

kernel

• misc. type adjustments for typescript 4.0

Decentraland libraries

Due to the asynchronous nature of the Decentraland runtime and modules (i.e. @decentraland/Identity) AMD was chosen to load modules instead of synchronous bundlers (webpack, browserify).

Also, we wanted to reduce the initial learning curve as much as possible removing every non-necessary or boilerplate piece of code from the tutorials import {} from .....

The current AMD solution also enables us to have a more granular control of the order of execution of the scenes and the interaction with the runtimes, decoupling the module loading.

One of the downsides of this model, is that it has nothing to do with Node.js and mostly CommonJS, which is the module resolution of node. Thus, every module installed in node_modules is, for this runtime, only a folder. It has no semantics on name resolution of any kind. That makes integrations with existing Node.js code more difficult.

There is no evident need to use Node.js code, but the lack of a proper module system to share and load libraries or frameworks is a huge pain point for the users.

This pull request adds support for libraries leveraging the current NPM infrastructure and tools and methods. It will be possible to adapt libraries only by adding "decentralandLibrary": {} to the package.json. Then, to install the library the command npm install my-lib-name --save-bundle must be used to configure it in the scene.

It is also provided the capability to create libraries that leverage the current exposed ECS, to do so, it must be compiled using the build-ecs command instead of tsc.

A .lib file is also created, it is a JSON map containing the code of every dependency and all the files required to make the library work, including the library. The content of those files is plain JS, it should contain inline source maps and not be obfuscated in order to enable debuggability by the final developer.

That means we will have to ship non-minified libraries. Because build-ecs will ultimately remove duplicated code and minify the whole compiled file at the end.

Future: I got this debugging schema working on vscode remotely. This schema will enable better tools to debug from IDEs with inspector protocol.

How to create a decentraland library?

menduz/dcl-physics-lib (example library)

The process is very similar to the scene creation. A few extra things must be done in order to publish and use the library. (it is worth to mention that the libraries may not be published in NPM, but since they were thought as NPM packages, you can leverage local linking)

1. dcl init as usual
2. add main and typings to the package.json, matching the tsconfig.json
3. add the key decentralandLibrary to the package.json

The final package.json may be as simple as

{
  "name": "physics-lib",
  "version": "1.0.0",
  "main": "./dist/physics-lib.js",
  "typings": "./dist/physics-lib.d.ts",
  "decentralandLibrary": {},
  "files": ["dist"]
}

And the tsconfig.json is very familiar too, the only thing that changes from the template is the outFile

{
  "compilerOptions": {
    "outFile": "./dist/physics-lib.js",
    "allowJs": true
  },
  "include": ["src/**/*.ts"],
  "extends": "./node_modules/decentraland-ecs/types/tsconfig.json"
}

How to use a decentraland library?

menduz/dcl-physics-scene (example scene using library)

The easiest way to do it is by NPM, or local linking (there is also a manual way, more on that later). Either way, we leverage NPM.

I already published this library physics-lib to test the whole integration.

1. In a blank scene, run npm install physics-lib --save-bundle we use --save-bundle or -B to later tell the compiler to bundle this dependency. This is also a hack, but the intention of our usage is the intended by NPM. We are actually bundling the library.

   { // package.json
     "dependencies": {
       "physics-lib": "^1.0.0"
     },
     "bundleDependencies": [
       "physics-lib"
     ]
   }

2. Run npm run build or dcl start once. It will update the tsconfig.json to include the part that does the trick:

     "paths": {
       "*": [
         "node_modules/physics-lib/dist/physics-lib.d.ts"
       ]
     },
     "baseUrl": "."

There you go, you are now using a library, with support for the debugger!

How it works & about the manual way:

The key of the solution resides in the module resolution of typescript. We leverage a feature paths intended for complicated web projects with even more complicated build systems: we tell the compiler where to find the correct modules, in the case of the libraries, we tell it the whole path of the lib. Then typescript knows how to find the source file.

Links:

• Demo scene: https://github.com/menduz/dcl-physics-scene
• Demo library: https://github.com/menduz/dcl-physics-lib

[menduz]

@nchamo @marcosnc I have no access to add the "ready for review" label, could you?

[eordano]

Untested ACK!

[eordano]

@HPrivakos @nearnshaw can you help us test this?

[menduz]

To test:

In the kernel folder:

1. make build-essentials packages/build-ecs/index.js

In your test scene:

1. npm install as usual.
2. replace (your scene)/node_modules/decentraland-ecs/artifacts/build-ecs.js by (explorer)/kernel/packages/build-ecs/build-ecs.js
3. also run npm install terser (this won't be necessary outside this isolated test/validation)
4. in the package json, you must make sure the build-ecs you are calling is the same as the provided in the file. This is necessary due to possible global linking. To do so:
   Replace the following lines

   -    "build": "build-ecs",
   +    "build": "node_modules/decentraland-ecs/artifacts/build-ecs.js",
   -    "watch": "build-ecs --watch",
   +    "watch": "node_modules/decentraland-ecs/artifacts/build-ecs.js --watch",

[nearnshaw]

I've been testing this on several scenes, including complex scenes like Genesis Plaza and my own Game Jam project. No bugs or issues detected on those.
These scenes even include libraries that weren't created with this new process, like the utils library and the UI utils library (https://github.com/decentraland/decentraland-ui-utils)
and I can even look into the code of these libraries and add breakpoints there.

Placing breakpoints and seeing the current state of variables at that point is a beautiful thing. Really makes debugging so much easier. <3 <3 <3

IDK how possible this might be, but it would be nice if the source code was easier to find. Currently, after I pick the Sources tab I have to navigate to:
PrcelSceneWorker(previewScene)/dcl:// / scr
It's not such a long path, but there are several other branches that seem just as likely to have what I'm looking for.

[menduz]

Thanks for testing Nico, regarding the automatic navigation to the sources, nowadays, it is very out of our control. The browser does not expose any API to do it. I have a remote debugger working on another branch, to enable breakpoints directly from VSCode, but this is the first required step for that, supporting it would be a much bigger feature.

[pablitar]

LGTM! Great work!

[pablitar]

@menduz We'll be using this PR as a test to adapt our CI to be able to run forks when explicitly approved on Circle CI.

That'll require some work so this may take a while to be merged. But we are working on it :)

[menduz]

Follow up on #1344