Skip to content

Commit

Permalink
build: implement collective typecheck
Browse files Browse the repository at this point in the history
This PR replaces #2057.

Implement a collective typecheck action that can be invoked in the root of the
monorepo with `yarn typecheck` or in watch-mode with `yarn watch:typecheck`.
Include the watch-mode typecheck action as part of `yarn start` (a.k.a
`yarn watch`).

To activate collective typecheck for a package in the monorepo, its
`package.json` file should specify:

```
{
  "embark-collective": {
    "typecheck": true
  }
}
```
*-or-*
```
{
  "embark-collective": {
    "typecheck": {...}
  }
}
```

Where `{...}` above is a `tsconfig.json` fragment that will be merged into the
config generated for the package according the same rules that `tsc` applies
when merging [configs][config].

When collective typecheck begins, it generates a `tsconfig.json` for the root
of the monorepo and for each package that is activated for the action. If the
generated JSON is different than what's on disk for the respective root/package
config, or if the config is not present on disk, then it will be
written. Changes to generated `tsconfig.json` files should be committed; such
changes will arise when there are structural changes to the monorepo, e.g. a
package is added, removed, moved and/or the directory containing it is
renamed. Since the configs are only generated at the beginning of collective
typecheck, when structural changes are made in the monorepo `yarn typecheck`
(or `yarn start` or `yarn watch:typecheck`) should be restarted.

Nearly all of the packages in the monorepo (i.e. all those for which it makes
sense) have been activated for collective typecheck. Even those packages that
don't contain `.ts` sources are activated because `tsc` can make better sense
of the code base as a whole owing to the project references included in the
generated `tsconfig.json` files. Also, owing to the fully cross-referenced
`tsconfig.json` files, it's possible for `tsc` to type check the whole code
base without babel (`yarn build` or `yarn watch:build`) having been run
beforehand.

**NOTE** that a *"cold typecheck"* of the whole monorepo is resource intensive:
on this author's 2019 MacBook Pro it takes around three minutes, the fans spin
up, and `tsc` uses nearly 0.5 GB of RAM. However, once a full typecheck has
completed, the next full typecheck will complete in a few seconds or less; and
when running in watch-mode there is likewise a *big* speedup once a full
typecheck has completed, whether that full check happened before it's running
in watch-mode or when watch-mode itself resulted in a full check before
switching automatically to incremental check, as well a corresponding *big*
reduction in resource consumption. A full check will be needed any time
`yarn typecheck` (or `yarn start` or `yarn watch:typecheck`) is run in a fresh
clone plus `yarn install`, or after doing `yarn reboot[:full]` or `yarn reset`.

The combination of options in each generated package-level `tsconfig.json` and
the root `tsconfig.base.json` result in `tsc` writing `.d.ts` files (TypeScript
declaration files) into the `dist/` directory of each package. That
output is intended to live side-by-side with babel's output, and therefore the
`"rootDir"` option in each generated config is set to `"./src"`.

In projects activated for collective typecheck, `.js` may be converted to `.ts`
and/or `.ts` sources may be added without any additional changes needed in
package-level `package.json`.

---

Reorganize types in `packages/core/typings` (a.k.a `@types/embark`) into
`packages/core/core` (`embark-core`), refactor other packages' imports
accordingly, and delete `packages/core/typings` from the monorepo. This results
in some similarly named but incompatible types exported from `embark-core`
(e.g. `Events` and `EmbarkEvents`, the latter being the one from
`packages/core/typings`); future refactoring should consolidate those types. To
avoid circular dependency relationships it's also necessary to split out
`Engine` from `embark-core` into its own package (`embark-engine`) and to
introduce a bit of duplication, e.g. the `Maybe` type that's now defined in
both `embark-i18n` and `embark-core`.

In the process of the types reorg, move many dependencies spec'd in various
`package.json` to the `package.json` of the package/s that actually depend on
them, e.g. many are moved from `packages/embark/package.json` to
`packages/core/engine/package.json`. Related to those moves, fix some Node.js
`require`-logic related to bug-prone dependency resolution.

Fix all type errors that appeared as a result of activating collective
typecheck across the whole monorepo.

Reactivate `tslint` in `packages/core/core` and fix the remaining linter errors.

Tidy up and add a few items in the root `package.json` scripts.

Bump lerna from `3.16.4` to `3.19.0`.

Bumpt typescript from `3.6.3` to `3.7.2`.

Bumpt tslint from `5.16.0` to `5.20.1`.

Make various changes related to packages' `import`/`require`ing packages that
weren't spec'd in their respective `package.json`. More refactoring is needed
in this regard, but changes were made as the problems were observed in the
process of authoring this PR.

[config]: https://www.typescriptlang.org/docs/handbook/tsconfig-json.html
  • Loading branch information
michaelsbradleyjr authored and iurimatias committed Dec 13, 2019
1 parent a9c91c9 commit ee56f37
Show file tree
Hide file tree
Showing 235 changed files with 4,328 additions and 2,556 deletions.
2 changes: 2 additions & 0 deletions .gitignore
@@ -1,6 +1,7 @@
.DS_Store
.idea
.nyc_output
.tsconfig.collective.json
.vscode
@embark*.tgz
NOTES
Expand All @@ -14,6 +15,7 @@ npm-debug.log*
npm-shrinkwrap.json
package
package-lock.json
tsconfig.tsbuildinfo
yarn-debug.log*
yarn-error.log*
yarn.lock
Expand Down
3 changes: 2 additions & 1 deletion dapps/tests/app/package.json
Expand Up @@ -13,6 +13,7 @@
"react": "16.12.0",
"react-bootstrap": "0.32.4",
"react-dom": "16.12.0",
"rimraf": "3.0.0",
"zeppelin-solidity": "1.12.0"
},
"license": "MIT",
Expand All @@ -22,7 +23,7 @@
"ci": "npm run qa",
"clean": "npm run reset",
"qa": "npm run test",
"reset": "npx embark-reset",
"reset": "npx embark-reset && npx rimraf embark-*.tgz package",
"test": "npx embark test"
},
"version": "5.0.0-alpha.4"
Expand Down
5 changes: 3 additions & 2 deletions dapps/tests/contracts/package.json
Expand Up @@ -2,7 +2,8 @@
"description": "Test DApp for integration testing purposes",
"devDependencies": {
"embark": "^5.0.0-alpha.4",
"embark-reset": "^5.0.0-alpha.2"
"embark-reset": "^5.0.0-alpha.2",
"rimraf": "3.0.0"
},
"license": "MIT",
"name": "embark-dapp-test-contracts",
Expand All @@ -11,7 +12,7 @@
"ci": "npm run qa",
"clean": "npm run reset",
"qa": "npm run test",
"reset": "npx embark-reset",
"reset": "npx embark-reset && npx rimraf embark-*.tgz package",
"test": "npx embark test"
},
"version": "5.0.0-alpha.4"
Expand Down
7 changes: 7 additions & 0 deletions dapps/tests/service/package.json
Expand Up @@ -3,9 +3,16 @@
"dependencies": {
"haml": "0.4.3"
},
"devDependencies": {
"rimraf": "3.0.0"
},
"license": "MIT",
"main": "index.js",
"name": "embark-dapp-test-service",
"private": true,
"scripts": {
"clean": "npm run reset",
"reset": "npx rimraf embark-*.tgz package"
},
"version": "5.0.0-alpha.0"
}
16 changes: 8 additions & 8 deletions package.json
Expand Up @@ -7,7 +7,7 @@
"find-up": "4.1.0",
"form-data": "2.5.1",
"fs-extra": "8.1.0",
"lerna": "3.16.4",
"lerna": "3.19.0",
"minimist": "1.2.0",
"npm-run-all": "4.1.5",
"nyc": "13.1.0",
Expand All @@ -29,7 +29,7 @@
"ci:full": "npm-run-all cwtree \"ci -- --concurrency={1}\" ci:dapps cwtree -- 1",
"clean": "node scripts/monorun --stream clean",
"clean:full": "npx npm-run-all clean clean:top",
"clean:top": "npx rimraf node_modules",
"clean:top": "npm run reset:top && npx rimraf node_modules",
"coverage": "npm-run-all coverage:collect coverage:report",
"coverage:collect": "node scripts/coverage-collect",
"coverage:coveralls": "nyc report --reporter=text-lcov | coveralls || exit 0",
Expand All @@ -39,27 +39,27 @@
"deploy:site": "node site/deploy-site",
"globalize": "node scripts/globalize",
"lint": "node scripts/monorun --parallel lint",
"package": "lerna exec --stream --concurrency=1 -- npm pack",
"package": "lerna exec --concurrency=1 --no-private --stream -- npm pack",
"postclean": "npx lerna clean --yes",
"postreboot": "yarn install",
"postreboot:full": "yarn install",
"preci:full": "yarn install",
"preclean": "npm run reset:top",
"preqa:full": "yarn install",
"qa": "node scripts/monorun --ignore embark-dapp-* --stream qa",
"qa:dapps": "lerna run --concurrency=1 --scope embark-dapp-* --stream qa",
"qa:full": "npm-run-all cwtree reboot:full cwtree \"qa -- --concurrency={1}\" qa:dapps cwtree -- 1",
"reboot": "npm run clean",
"reboot:full": "npm run clean:full",
"release": "node scripts/release",
"reset": "node scripts/monorun --stream reset",
"reset:full": "npx npm-run-all reset reset:top",
"reset:top": "npx rimraf .nyc_output coverage",
"reset": "node scripts/monorun --stream reset && npm-run-all reset:*",
"reset:top": "npx rimraf .tsconfig.collective.json .nyc_output coverage",
"reset:tsbuildinfo": "npx lerna exec --parallel -- npx rimraf node_modules/.cache/tsc",
"start": "node scripts/monorun --parallel start",
"test": "node scripts/monorun --ignore embark-dapp-* --stream test",
"test:dapps": "lerna run --concurrency=1 --scope embark-dapp-* --stream test",
"test:full": "npm-run-all test test:dapps",
"typecheck": "node scripts/monorun --parallel typecheck",
"typecheck": "node scripts/monorun --stream typecheck",
"typecheck:clean": "npm run typecheck -- -- -- --clean",
"watch": "node scripts/monorun --parallel watch",
"watch:build": "node scripts/monorun --parallel watch:build",
"watch:lint": "node scripts/monorun --parallel watch:lint",
Expand Down
7 changes: 5 additions & 2 deletions packages/cockpit/api-client/package.json
Expand Up @@ -22,18 +22,21 @@
"url": "https://github.com/embark-framework/embark.git"
},
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
"files": [
"dist"
],
"embark-collective": {
"build:node": true
"build:node": true,
"typecheck": true
},
"scripts": {
"_build": "npm run solo -- build",
"_typecheck": "npm run solo -- typecheck",
"ci": "npm run qa",
"clean": "npm run reset",
"lint": "eslint src/",
"qa": "npm-run-all lint _build",
"qa": "npm-run-all lint _typecheck _build",
"reset": "npx rimraf dist embark-*.tgz package",
"solo": "embark-solo"
},
Expand Down
12 changes: 12 additions & 0 deletions packages/cockpit/api-client/tsconfig.json
@@ -0,0 +1,12 @@
{
"compilerOptions": {
"composite": true,
"declarationDir": "./dist",
"rootDir": "./src",
"tsBuildInfoFile": "./node_modules/.cache/tsc/tsconfig.embark-api-client.tsbuildinfo"
},
"extends": "../../../tsconfig.base.json",
"include": [
"src/**/*"
]
}
2 changes: 1 addition & 1 deletion packages/cockpit/ui/package.json
Expand Up @@ -98,7 +98,7 @@
"redux-saga": "1.1.3",
"rimraf": "3.0.0",
"simple-line-icons": "2.4.1",
"typescript": "3.6.3",
"typescript": "3.7.2",
"uuid": "3.3.2",
"velocity-react": "1.4.1",
"web3": "1.2.1",
Expand Down
18 changes: 9 additions & 9 deletions packages/core/code-runner/package.json
Expand Up @@ -25,28 +25,29 @@
"url": "https://github.com/embark-framework/embark.git"
},
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
"embark-collective": {
"build:node": true
"build:node": true,
"typecheck": true
},
"scripts": {
"_build": "npm run solo -- build",
"_typecheck": "npm run solo -- typecheck",
"ci": "npm run qa",
"clean": "npm run reset",
"lint": "npm-run-all lint:*",
"lint:js": "eslint src/",
"lint:ts": "tslint -c tslint.json \"src/**/*.ts\"",
"qa": "npm-run-all lint typecheck _build",
"qa": "npm-run-all lint _typecheck _build",
"reset": "npx rimraf dist embark-*.tgz package",
"solo": "embark-solo",
"typecheck": "tsc",
"watch": "run-p watch:*",
"watch:typecheck": "npm run typecheck -- --preserveWatchOutput --watch"
"solo": "embark-solo"
},
"eslintConfig": {
"extends": "../../../.eslintrc.json"
},
"dependencies": {
"@babel/runtime-corejs3": "7.7.4",
"@types/async": "3.0.3",
"async": "2.6.1",
"colors": "1.3.2",
"core-js": "3.4.3",
Expand All @@ -60,13 +61,12 @@
"web3": "1.2.1"
},
"devDependencies": {
"@types/async": "3.0.3",
"embark-solo": "^5.0.0-alpha.2",
"eslint": "5.7.0",
"npm-run-all": "4.1.5",
"rimraf": "3.0.0",
"tslint": "5.16.0",
"typescript": "3.6.3"
"tslint": "5.20.1",
"typescript": "3.7.2"
},
"engines": {
"node": ">=10.17.0 <12.0.0",
Expand Down
4 changes: 2 additions & 2 deletions packages/core/code-runner/src/index.ts
@@ -1,4 +1,4 @@
import { Callback, Embark, Events } /* supplied by @types/embark in packages/embark-typings */ from "embark";
import { Callback, Embark, EmbarkEvents } from "embark-core";
import { __ } from "embark-i18n";
import { Logger } from 'embark-logger';
import * as fs from "./fs";
Expand All @@ -8,7 +8,7 @@ export { fs, VM };

class CodeRunner {
private logger: Logger;
private events: Events;
private events: EmbarkEvents;
private vm: VM;

constructor(embark: Embark, _options: any) {
Expand Down
2 changes: 1 addition & 1 deletion packages/core/code-runner/src/vm.ts
@@ -1,5 +1,5 @@
import { each } from "async";
import { Callback } /* supplied by @types/embark in packages/embark-typings */ from "embark";
import { Callback } from "embark-core";
import { compact, dappPath, isEs6Module, recursiveMerge } from "embark-utils";
import { Logger } from 'embark-logger';
import * as path from "path";
Expand Down
26 changes: 24 additions & 2 deletions packages/core/code-runner/tsconfig.json
@@ -1,4 +1,26 @@
{
"extends": "../../../tsconfig.json",
"include": ["src/**/*"]
"compilerOptions": {
"composite": true,
"declarationDir": "./dist",
"rootDir": "./src",
"tsBuildInfoFile": "./node_modules/.cache/tsc/tsconfig.embark-code-runner.tsbuildinfo"
},
"extends": "../../../tsconfig.base.json",
"include": [
"src/**/*"
],
"references": [
{
"path": "../../embarkjs/embarkjs"
},
{
"path": "../core"
},
{
"path": "../logger"
},
{
"path": "../utils"
}
]
}
19 changes: 10 additions & 9 deletions packages/core/console/package.json
Expand Up @@ -26,39 +26,40 @@
"url": "https://github.com/embark-framework/embark.git"
},
"main": "./dist/lib/index.js",
"types": "./dist/lib/index.d.ts",
"embark-collective": {
"build:node": true
"build:node": true,
"typecheck": true
},
"scripts": {
"_build": "npm run solo -- build",
"_typecheck": "npm run solo -- typecheck",
"ci": "npm run qa",
"clean": "npm run reset",
"lint": "npm-run-all lint:*",
"lint:js": "eslint src/",
"lint:ts": "tslint -c tslint.json \"src/**/*.ts\"",
"qa": "npm-run-all lint typecheck _build test",
"qa": "npm-run-all lint _typecheck _build test",
"reset": "npx rimraf .nyc_output coverage dist embark-*.tgz package",
"solo": "embark-solo",
"test": "nyc --reporter=html --reporter=json mocha \"dist/test/**/*.js\" --exit --no-timeouts --require source-map-support/register",
"typecheck": "tsc",
"watch": "run-p watch:*",
"watch:typecheck": "npm run typecheck -- --preserveWatchOutput --watch"
"test": "nyc --reporter=html --reporter=json mocha \"dist/test/**/*.js\" --exit --no-timeouts --require source-map-support/register"
},
"eslintConfig": {
"extends": "../../../.eslintrc.json"
},
"dependencies": {
"@babel/runtime-corejs3": "7.7.4",
"@types/json-stringify-safe": "5.0.0",
"async": "2.6.1",
"chalk": "2.4.2",
"core-js": "3.4.3",
"embark-core": "^5.0.0-alpha.4",
"embark-i18n": "^5.0.0-alpha.2",
"embark-utils": "^5.0.0-alpha.4",
"fs-extra": "8.1.0",
"json-stringify-safe": "5.0.1"
},
"devDependencies": {
"@types/json-stringify-safe": "5.0.0",
"embark-logger": "^5.0.0-alpha.4",
"embark-solo": "^5.0.0-alpha.2",
"eslint": "5.7.0",
Expand All @@ -67,8 +68,8 @@
"nyc": "13.1.0",
"rimraf": "3.0.0",
"source-map-support": "0.5.13",
"tslint": "5.16.0",
"typescript": "3.6.3"
"tslint": "5.20.1",
"typescript": "3.7.2"
},
"engines": {
"node": ">=10.17.0 <12.0.0",
Expand Down
11 changes: 4 additions & 7 deletions packages/core/console/src/lib/index.ts
@@ -1,6 +1,5 @@
import { waterfall } from "async";
import chalk from "chalk";
import { Callback, Embark, Events } /* supplied by @types/embark in packages/embark-typings */ from "embark";
import { Embark, EmbarkEvents } from "embark-core";
import constants from "embark-core/constants.json";
import { __ } from "embark-i18n";
import { dappPath, escapeHtml, exit, jsonFunctionReplacer } from "embark-utils";
Expand All @@ -16,9 +15,9 @@ interface HelpDescription {
usage?: string;
}

class Console {
export default class Console {
private embark: Embark;
private events: Events;
private events: EmbarkEvents;
private plugins: any;
private version: string;
private logger: any;
Expand Down Expand Up @@ -150,7 +149,7 @@ class Console {
__("The web3 object and the interfaces for the deployed contracts and their methods are also available"));
return helpText.join("\n");
} else if (["quit", "exit", "sair", "sortir", __("quit")].indexOf(cmd) >= 0) {
exit();
exit(0);
}
return false;
}
Expand Down Expand Up @@ -285,5 +284,3 @@ class Console {
}
}
}

module.exports = Console;
9 changes: 6 additions & 3 deletions packages/core/console/src/lib/suggestions.ts
@@ -1,6 +1,9 @@
import { Contract, Embark, Events } /* supplied by @types/embark in packages/embark-typings */ from "embark";
import { Contract, Embark, EmbarkEvents } from "embark-core";
import { fuzzySearch } from "embark-utils";
import { suggestions as defaultSuggestions } from "../../suggestions.json";
import { readJsonSync } from "fs-extra";
import { join } from "path";

const { suggestions: defaultSuggestions } = readJsonSync(join(__dirname, "../../suggestions.json"));

interface ContractsManager {
[key: string]: any;
Expand All @@ -23,7 +26,7 @@ type SuggestionsList = Suggestion[];

export default class Suggestions {
private embark: Embark;
private events: Events;
private events: EmbarkEvents;
private contracts: ContractsManager;
private static readonly DEFAULT_SUGGESTIONS = defaultSuggestions;
private _suggestions: SuggestionsList = [];
Expand Down

0 comments on commit ee56f37

Please sign in to comment.