Add Broccoli 2.0 support by oligriffiths · Pull Request #7798 · ember-cli/ember-cli

oligriffiths · 2018-04-30T04:07:28Z

What

Once broccolijs/broccoli#363 merges, this PR adds support for broccoli 2.0 in ember-cli via BROCCOLI_2 environment flag. If there is a better way to do this, please LMK and I will change.

The requisite change is to require the broccoli package, and to reformat the response from the broccoli build() method, into:

{
  directory: node.outputPath,
  graph: node,
}

Test

In my testing, the output of both the 0-18x series and master is the same, I verified this with:

ember build dist1
EMBER_CLI_BROCCOLI_2=1 ember build dist2
diff -rq dist1 dist2

It should output nothing.

To test using the system temp directory, do:

EMBER_CLI_BROCCOLI_2=1 EMBER_CLI_SYSTEM_TEMP=1 ember build

Updated tests to validate the build results are the same, and support for system tmp dir.

Turbo87 · 2018-04-30T07:28:50Z

nice work! 👏

via BROCCOLI_MASTER environment flag. If there is a better way to do this, please LMK and I will change

we usually use https://github.com/ember-cli/ember-cli/blob/master/lib/experiments/index.js for these things, though that does not use env vars (yet). I would recommend migrating the feature flag to that file but instead of using the symbol() code there keep using the env var that you already have.

oligriffiths · 2018-05-01T02:03:04Z

@Turbo87 Done. What's the best way to CI test this? Is it possible to re-run all tests with and without the env var set?

Turbo87 · 2018-05-01T09:45:59Z

What's the best way to CI test this? Is it possible to re-run all tests with and without the env var set?

What we currently do is: on master all experiments are enabled, on beta and release all experiments are supposed to be disabled. That led to beta currently being broken since the tests passed with the Module Unification experiment being enabled, but when I disabled the experiment for the beta release the test suite started breaking.

What I'm meaning to say: what we currently do is not great and I think running the test suite with experiments turned off by default would be better and then adding another CI job per experiment.

@ember-cli/core thoughts?

mansona · 2018-05-01T10:58:44Z

@Turbo87 I don't know how crazy you want to get with the CI setup but I think that we should be considering running a full on/off matrix of experiments on each CI build, that way we would catch any potential interactions between experiments.

I don't think it needs to go so far as to test each node version with the full matrix of experiments but maybe we could pick one node version and then run the experiment matrix off it as a smoke test?

The current issue with the Beta branch was introduced on 1st March and would have shown on any test that disabled the MU experiment.

I'm happy to help set this up 😄

Turbo87 · 2018-05-01T13:01:29Z

My approach would be:

one job per node supported version running without experiments
one job per experiment running the fastest node version (10)
one job running with all experiments enabled on the fastest node version

Certainly can't catch all issues, but seems like a reasonable compromise. PR welcome, but we should discuss the details in the CLI call on Thursday.

rwjblue · 2018-05-01T13:21:04Z

@Turbo87 - I completely agree with the suggested path forward in your last comment (#7798 (comment)).

oligriffiths · 2018-05-01T13:24:52Z

This approach seems to make sense and provides a decent balance between every permutation of experiments thus taking longer to the suite to pass and coverage of conditions.

In my case, there’s a bug somewhere as the test suite doesn’t pass with the experiment enabled.

rwjblue · 2018-05-01T16:27:54Z

I submitted #7803 for refactoring the CI runs...

rwjblue

Very excited for this progress!

rwjblue · 2018-05-02T13:12:54Z

lib/experiments/index.js

 let experiments = {
  MODULE_UNIFICATION: symbol('module-unification'),
  DELAYED_TRANSPILATION: symbol('delayed-transpilation'),
+  BROCCOLI_MASTER: process.env.BROCCOLI_MASTER === '1', // Note: EXPERIMENTAL, DO NOT USE!


Hehe, these are all experimental 😉

Lets make the flag: process.env.EMBER_CLI_BROCCOLI_1 (I don't like using _MASTER conceptually here)...

rwjblue · 2018-05-02T13:13:34Z

lib/models/builder.js


-    const broccoli = require('broccoli-builder');
+    let broccoli;
+    if (this.broccoliMaster) {


I'd rather do the experiments check here, instead of in the build task

if (experiments.BROCCOLI_MASTER) { } else { }

rwjblue · 2018-05-02T13:14:25Z

lib/models/builder.js

      .then(() => this.builder.build(willReadStringDir))
+      .then(node => {
+        // broccoli-builder reformats the response into {directory, graph}
+        if (this.broccoliMaster) {


Lets do the experiments guard here directly also

rwjblue · 2018-05-02T13:15:46Z

lib/tasks/build.js

      outputPath: options.outputPath,
      environment: options.environment,
      project: this.project,
+      broccoliMaster: experiments.BROCCOLI_MASTER,


Should be able to revert changes to this file once the builder itself does its own experiments check...

rwjblue · 2018-05-02T13:15:50Z

lib/tasks/serve.js

      outputPath: options.outputPath,
      project: this.project,
      environment: options.environment,
+      broccoliMaster: experiments.BROCCOLI_MASTER,


Should be able to revert changes to this file once the builder itself does its own experiments check...

Turbo87 · 2018-05-17T08:19:31Z

lib/models/builder.js

+        });
+      } else {
+        this._cleanupPromise = this.builder.cleanup();
+      }


you can simplify this using:

this._cleanupPromise = Promise.resolve().then(() => this.builder.cleanup());

which should work for both Broccoli 0.x and 1.x

Turbo87 · 2018-05-17T08:20:05Z

tests/unit/models/builder-test.js


-    it('calls instrumentation.stop(build, result, resultAnnotation)', function() {
-      let mockAnnotation = 'MockAnnotation';
+    if (!experiments.BROCCOLI_MASTER) {


this should be BROCCOLI_1, right?

Turbo87 · 2018-05-17T08:20:16Z

tests/unit/models/builder-test.js

-    it('allows addons to add promises preBuild', function() {
-      let preBuild = td.replace(addon, 'preBuild', td.function());
-      td.when(preBuild(), { ignoreExtraArgs: true, times: 1 }).thenReturn(Promise.resolve());
+    if (!experiments.BROCCOLI_MASTER) {


rwjblue · 2018-06-11T23:20:34Z

FYI - I just released broccoli@2.0.0-beta.2 (updates deps, drops RSVP, drops Node 4).

oligriffiths · 2018-06-11T23:25:38Z

👌

rwjblue · 2018-06-12T00:15:02Z

@oligriffiths - I just rebased, added the bump commit to broccoli@2.0.0-beta.2, and force pushed to kick off a new CI run. I think this is very close to being landable...

rwjblue · 2018-06-12T14:51:24Z

Restarted some of the CI builds (they were failing on the yarn installation)...

oligriffiths · 2018-06-12T15:12:20Z

@rwjblue thanks, I saw that, good to know you can manually restart them, was gonna do a noop commit

stefanpenner · 2018-06-12T22:56:10Z

bin/ember

+  }).catch((err) => {
+    // should not normally reach here
+    // this handles the case where console-ui is broken
+    console.error(err);


In general we would prefer to not mutate this file. But this may be an exception, @rwjblue c/d?

Chatted with @krisselden in slack about this, I think we can probably move this to lib/cli.js and still handle the broken scenario.

rwjblue · 2018-06-13T00:27:12Z

lib/models/builder.js

  constructor(options) {
    super(options);

+    this.broccoli2 = Boolean(options.broccoli2 === undefined ? experiments.BROCCOLI_2 : options.broccoli2);


I think it would be better to remove this.broccoli2 completely, and replace it with direct experiments.BROCCOLI_2 checks...

I couldn’t see how to mock that property in tests. I want to compare with the property on and off in the same test.

Why? I think having tests around the expected behaviors should be enough. In general, from the outside, I don't think which builder we use should actually matter much, right?

Well, we want to ensure that the files are exactly the same between both versions.
I guess if we have a fixture directory and compare against that, that'd also work.

Totally agreed. I think the goal here should be that the migration is seemless and the output is identical. And we should be able to do that via fixtures (either inside the file system, or just via fixturify in the test itself)..

Cool. Are we doing this anywhere already?

rwjblue · 2018-06-13T00:29:00Z

lib/models/builder.js

    super(options);

+    this.broccoli2 = Boolean(options.broccoli2 === undefined ? experiments.BROCCOLI_2 : options.broccoli2);
+    this.systemTemp = Boolean(options.systemTemp === undefined ? experiments.SYSTEM_TEMP : options.systemTemp);


Same RE: experiment checks. In general it should not be possible to force these experiments to true without process.env.EMBER_CLI_* is set...

rwjblue · 2018-06-13T00:29:55Z

lib/models/builder.js

+   * @param node The node returned from Broccoli builder
+   */
+  compatNode(node) {
+    if (this.broccoli2) {


I was hoping that we could structure the code such that the broccoli@2 case is the "normal" case, and we do any shimming/polyfilling for the older broccoli-builder APIs...

I think we should do that once we move out of experimental mode

rwjblue · 2018-06-13T00:31:19Z

package.json

    "chai-as-promised": "^7.0.0",
    "chai-files": "^1.2.0",
    "co": "^4.6.0",
+    "dir-compare": "^1.4.0",


What are we using this for?

Comparing the directory structure and files for a Broccoli-builder vs broccoli 2.0 build to ensure they’re identical.

Hmm, I think in general I'd prefer to use something like:

expect(fixturify.readSync(oneDir)).to.deep.equal(fixturify.readSync(twoDir))

the more you know. Will test that.

Will that compare file contents/hash?

fixturify.readSync loads an entire directory (recursively) into a POJO (where the values are the string contents of the individual files).

Snippet from the readme:

var fixturify = require('fixturify') var obj = { 'foo.txt': 'foo.txt contents', 'subdir': { 'bar.txt': 'bar.txt contents' } } fixturify.writeSync('testdir', obj) // write it to disk fixturify.readSync('testdir') // => deep-equals obj

@rwjblue are there already tests that already assert the output of the build is correct?

stefanpenner · 2018-06-18T14:44:15Z

lib/models/builder.js

    } else {
      broccoli = require('broccoli-builder');
+      if (this.systemTemp) {
+        console.warn('EMBER_CLI_SYSTEM_TEMP only works in combination with EMBER_CLI_BROCCOLI_2');


we should simply error here

Will need to isolate tests in that case, because we can't do only combo environment variables (e.g. BROCCOLI_2 and SYSTEM_TEMP). Right now, the SYSTEM_TEMP flag will cause this error when being run without setting BROCCOLI_2.
Will do a little refactor.

krisselden · 2018-06-18T16:43:39Z

tests/unit/models/builder-test.js

+        if (experiments.BROCCOLI_2) {
+          expect(result.graph.__heimdall__).to.not.be.undefined;
+        } else {
+          expect(result.graph.constructor.name).to.equal('Node');


Not sure I understand the question...

rwjblue · 2018-06-27T20:06:40Z

Updated to ensure SYSTEM_TEMP experiment only runs with BROCCOLI_2.

rwjblue · 2018-06-27T20:56:16Z

Just rebased against master to get some of the CI fixes that have landed...

oligriffiths · 2018-06-29T17:15:35Z

@rwjblue tests all passing 🎉

oligriffiths · 2018-06-29T19:47:08Z

@krisselden I had to remove your changes to get tests passing. Things have changed a bit since you implemented them, and im not 100% sure what they were changing. So if you wouldn't mind testing this branch now, and see if you get the same errors, I'd appreciate it.

Ok, lots of changes since this review, gonna dismiss and re-tag

rwjblue · 2018-07-12T20:47:39Z

Rebased and updated broccoli to 2.0.0-beta.3...

oligriffiths · 2018-07-12T22:07:40Z

Whoop 🎉

oligriffiths changed the title ~~Add BROCCOLI_MASTER env var to support broccoli 1.0~~ Add Broccoli 1.0 support Apr 30, 2018

oligriffiths mentioned this pull request Apr 30, 2018

Return outputNode from build promise broccolijs/broccoli#363

Merged

rwjblue previously requested changes May 2, 2018

View reviewed changes

This was referenced May 3, 2018

Migrate to using Travis stages. #7803

Merged

Implement cleanup promise broccolijs/broccoli#365

Closed

oligriffiths force-pushed the broccoli-master branch from 2832349 to 11e4a70 Compare May 15, 2018 14:34

Turbo87 reviewed May 17, 2018

View reviewed changes

Turbo87 added the Needs Submitter Response label May 17, 2018

oligriffiths force-pushed the broccoli-master branch from c0839d4 to 5e2c13a Compare May 20, 2018 22:56

oligriffiths mentioned this pull request May 20, 2018

Port ember-cli/broccoli-builder to master broccolijs/broccoli#360

Closed

10 tasks

oligriffiths force-pushed the broccoli-master branch from 5e2c13a to b13e0bf Compare May 28, 2018 20:54

oligriffiths changed the title ~~Add Broccoli 1.0 support~~ Add Broccoli 2.0 support May 28, 2018

rwjblue force-pushed the broccoli-master branch from a0daca3 to b7eac6d Compare June 12, 2018 00:14

stefanpenner reviewed Jun 12, 2018

View reviewed changes

rwjblue reviewed Jun 13, 2018

View reviewed changes

stefanpenner approved these changes Jun 18, 2018

View reviewed changes

krisselden reviewed Jun 18, 2018

View reviewed changes

rwjblue force-pushed the broccoli-master branch from 3cad9d4 to 66d5c05 Compare June 27, 2018 20:55

oligriffiths force-pushed the broccoli-master branch 3 times, most recently from 7cd2082 to 8607dda Compare June 29, 2018 16:04

oligriffiths requested a review from rwjblue July 2, 2018 17:57

Oli Griffiths and others added 2 commits July 12, 2018 16:46

Add support for Broccoli 2

ec89ff4

Update to broccoli@2.0.0-beta.3.

b295f96

rwjblue force-pushed the broccoli-master branch from 8607dda to b295f96 Compare July 12, 2018 20:47

rwjblue merged commit 819f15e into ember-cli:master Jul 12, 2018

rwjblue removed the Needs Submitter Response label Jul 12, 2018

oligriffiths deleted the broccoli-master branch July 12, 2018 22:07

rwjblue mentioned this pull request Jul 13, 2018

The 'tmp' folder in the project root causes the typescript compiler to consume 100% cpu #7730

Closed

bertdeblock mentioned this pull request May 9, 2022

Remove unused CLI error class #9900

Merged

Conversation

oligriffiths commented Apr 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Test

Uh oh!

Turbo87 commented Apr 30, 2018

Uh oh!

oligriffiths commented May 1, 2018

Uh oh!

Turbo87 commented May 1, 2018

Uh oh!

mansona commented May 1, 2018

Uh oh!

Turbo87 commented May 1, 2018

Uh oh!

rwjblue commented May 1, 2018

Uh oh!

oligriffiths commented May 1, 2018

Uh oh!

rwjblue commented May 1, 2018

Uh oh!

rwjblue left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rwjblue commented Jun 11, 2018

Uh oh!

oligriffiths commented Jun 11, 2018

Uh oh!

rwjblue commented Jun 12, 2018

Uh oh!

rwjblue commented Jun 12, 2018

Uh oh!

oligriffiths commented Jun 12, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

oligriffiths commented Apr 30, 2018 •

edited

Loading