feat(broccoli): add incremental MergeTrees plugin #2064

caitp · 2015-05-21T16:37:54Z

caitp · 2015-05-21T16:38:17Z

TODO: add tests, fix the dart build which seems to be messed up.

caitp · 2015-05-21T17:09:36Z

There's a problem with this --- if the inputTrees come from another incremental plugin, there's no way to know if the "real" source file was removed (in which case the output should be removed), or if an input was not included in the last build process.

So what this means is, right now, the mergeTrees plugin will delete any specs because it doesn't think they're around anymore. That's not very good

illustration:

EchoBeach:angular caitp$ gulp test.unit.cjs
Dart SDK detected
[13:10:10] Using gulpfile ~/git/angular/gulpfile.js
[13:10:10] Starting 'build/clean.js'...
[13:10:10] Starting 'build/clean.tools'...
[13:10:10] Finished 'build/clean.tools' after 1.38 ms
[13:10:10] Starting 'build.tools'...
[13:10:10] Starting '!build.tools'...
[13:10:11] Finished 'build/clean.js' after 433 ms
[13:10:15] Finished '!build.tools' after 4.1 s
[13:10:15] Finished 'build.tools' after 4.11 s
[13:10:15] Starting 'test.unit.cjs'...
[13:10:15] Starting '!build.js.cjs'...
Tree diff:          funnel-output_path-cxvFz9UF.tmp, duration:    56ms,   208 changes detected (files:   208, directories:  167)
Tree diff: replace_filter-tmp_dest_dir-ZJvM0X0C.tmp, duration:    21ms,   416 changes detected (files:   416, directories:   55)
Tree diff:          funnel-output_path-9hlvAMyv.tmp, duration:     0ms,     1 changes detected (files:     1, directories:    1)
Tree diff: diffing_plugin_wrapper-output_path-DHPT9XAs.tmp, duration:    23ms,   417 changes detected (files:   417, directories:   55)
Tree diff:          funnel-output_path-YUWFgTka.tmp, duration:     0ms,     1 changes detected (files:     1, directories:    1)
Tree diff: diffing_plugin_wrapper-output_path-VAUHrzcS.tmp, duration:    21ms,   418 changes detected (files:   418, directories:   55)
Tree diff:          funnel-output_path-2Rh5EuAW.tmp, duration:     1ms,     1 changes detected (files:     1, directories:    1)
Tree diff:                   funnel-dest_UULMeC.tmp, duration:    24ms,   128 changes detected (files:   128, directories:  167)
Tree diff: diffing_plugin_wrapper-output_path-A8wXyHUC.tmp, duration:    22ms,   419 changes detected (files:   419, directories:   55)
Tree diff: diffing_plugin_wrapper-output_path-QmXT90nW.tmp, duration:    14ms,   384 changes detected (files:   384, directories:   24)
Tree diff:                   funnel-dest_1liPKx.tmp, duration:     2ms,    27 changes detected (files:    27, directories:   15)
Tree diff:    lodash_renderer-dest_dir-TUPYyHtV.tmp, duration:     0ms,     3 changes detected (files:     3, directories:    3)
Tree diff:                   funnel-dest_GqlAaQ.tmp, duration:    39ms,   833 changes detected (files:   833, directories:   87)

Slowest Trees                                 | Total               
----------------------------------------------+---------------------
DiffingTraceurCompiler                        | 6604ms              
DiffingTSCompiler                             | 4859ms              

Slowest Trees (cumulative)                    | Total (avg)         
----------------------------------------------+---------------------
DiffingTraceurCompiler (1)                    | 6604ms              
DiffingTSCompiler (1)                         | 4859ms              
MergeTrees (4)                                | 875ms (218 ms)      

creating node_modules symlink hack
creating link dist/js/cjs/node_modules/angular2 ../angular2
creating link dist/js/cjs/node_modules/benchmarks ../benchmarks
creating link dist/js/cjs/node_modules/benchmarks_external ../benchmarks_external
creating link dist/js/cjs/node_modules/benchpress ../benchpress
creating link dist/js/cjs/node_modules/examples ../examples
creating link dist/js/cjs/node_modules/rtts_assert ../rtts_assert
[13:10:28] Finished '!build.js.cjs' after 14 s
[13:10:28] Starting 'test.unit.cjs/ci'...
Started
..............................*.....................................................*.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................*.................................................................................................................................................................................................................................................................................................................................................................

Pending:

1) change detection dynamic change detection error handling should wrap exceptions into ChangeDetectionError
  No reason given


2) change detection JIT change detection error handling should wrap exceptions into ChangeDetectionError
  No reason given


3) Form Model ControlGroup optional components valueChanges should not fire an event when an excluded control is updated
  No reason given

1040 specs, 0 failures, 3 pending specs
Finished in 4.816 seconds
[13:10:35] Finished 'test.unit.cjs/ci' after 6.66 s

==================================================
 WATCH TRIGGERED BY FILE CHANGE #1
   (modules/angular2/src/facade/async.ts touched in another terminal)
 On: 5/21/2015 at 1:10:39 PM
==================================================
[13:10:39] Starting '!build.js.cjs'...
Tree diff:          funnel-output_path-cxvFz9UF.tmp, duration:    27ms,     0 changes detected (files:   208, directories:  167)
Tree diff: replace_filter-tmp_dest_dir-ZJvM0X0C.tmp, duration:     6ms,   416 changes detected (files:     0, directories:    0) /* long pointless array of changed files */
Tree diff:          funnel-output_path-9hlvAMyv.tmp, duration:     0ms,     0 changes detected (files:     1, directories:    1)
Tree diff: diffing_plugin_wrapper-output_path-DHPT9XAs.tmp, duration:     8ms,   416 changes detected (files:     1, directories:   55) /* long pointless array of changed files */
Tree diff:          funnel-output_path-YUWFgTka.tmp, duration:     0ms,     0 changes detected (files:     1, directories:    1)
Tree diff: diffing_plugin_wrapper-output_path-VAUHrzcS.tmp, duration:     8ms,   416 changes detected (files:     2, directories:   55) /* long pointless array of changed files */
Tree diff:          funnel-output_path-2Rh5EuAW.tmp, duration:     0ms,     0 changes detected (files:     1, directories:    1)
Tree diff:                   funnel-dest_UULMeC.tmp, duration:    25ms,     0 changes detected (files:   128, directories:  167)
Tree diff: diffing_plugin_wrapper-output_path-A8wXyHUC.tmp, duration:    10ms,   416 changes detected (files:     3, directories:   55) /* long pointless array of changed files */
Tree diff: diffing_plugin_wrapper-output_path-QmXT90nW.tmp, duration:    13ms,     0 changes detected (files:   384, directories:   24)
Tree diff:                   funnel-dest_1liPKx.tmp, duration:     2ms,     0 changes detected (files:    27, directories:   15)
Tree diff:    lodash_renderer-dest_dir-TUPYyHtV.tmp, duration:     0ms,     3 changes detected (files:     0, directories:    0) /* long pointless array of changed files */
Tree diff:                   funnel-dest_GqlAaQ.tmp, duration:    37ms,   419 changes detected (files:   414, directories:   87) /* long pointless array of changed files */

Slowest Trees                                 | Total               
----------------------------------------------+---------------------
ReplaceFilter                                 | 349ms               
Funnel                                        | 246ms               
MergeTrees                                    | 121ms               
ReplaceFilter                                 | 117ms               
MergeTrees                                    | 81ms                
DestCopy                                      | 74ms                
MergeTrees                                    | 72ms                
MergeTrees                                    | 71ms                

Slowest Trees (cumulative)                    | Total (avg)         
----------------------------------------------+---------------------
ReplaceFilter (2)                             | 466ms (233 ms)      
MergeTrees (4)                                | 346ms (86 ms)       
Funnel (10)                                   | 329ms (32 ms)       
DestCopy (1)                                  | 74ms                

[13:10:40] Finished '!build.js.cjs' after 1.28 s
[13:10:40] Starting 'test.unit.cjs/ci'...
Started


No specs found
Finished in 0.001 seconds
[13:10:40] Finished 'test.unit.cjs/ci' after 295 ms

caitp · 2015-05-21T17:35:03Z

interestingly, test.unit.dart works properly, and seems to rebuild very quickly

caitp · 2015-05-21T19:13:04Z

so um, any ideas there?

IgorMinar · 2015-05-22T05:28:09Z

tools/broccoli/broccoli-merge-trees.ts

+      let inputPath = this.inputPaths[index];
+      let existsLater = (relativePath) => {
+        for (let i = treeDiffs.length - 1;
+      i > index; --i) {


clang-format bug?

IgorMinar · 2015-05-22T05:42:09Z

my theory is that the reason why this is not working is that you are hitting #2051

incremental plugins producing input trees should not be a problem because the input tree is always complete even if it was incrementally updated. so you will never get an input tree just with the incremental updates, it will always be a full tree that contains incremental updates.

caitp · 2015-05-22T12:47:16Z

my theory is that the reason why this is not working is that you are hitting #2051

incremental plugins producing input trees should not be a problem because the input tree is always complete even if it was incrementally updated. so you will never get an input tree just with the incremental updates, it will always be a full tree that contains incremental updates.

Yeah, when I was looking at your patch it occurred to me that that's probably what this was. I think we should get 2053 in today

caitp · 2015-05-28T19:21:43Z

PTAL

IgorMinar · 2015-05-29T20:07:41Z

the first commit looks good.

the second commit could use some tests. can you add them please?

caitp · 2015-05-29T22:12:44Z

okay, I added "some" test coverage --- I'm not sure why travis is complaining about "toBe" on the Matchers interface, since I can see it locally quite plainly. But it will probably complain about the thing fixed by https://github.com/angular/DefinitelyTyped/pull/3 too when it gets around to it

caitp · 2015-05-30T22:56:08Z

At the moment a bunch of integration tests fail, investigating

caitp · 2015-05-30T23:07:10Z

nah, same tests fail locally on master too. I don't think this PR is breaking em, but not sure why travis is green for the master branch

IgorMinar · 2015-06-01T17:48:46Z

yeah, the original merger is conservative and throws when duplicates are detected. I think that this is a good default and avoids accidental over-writes.

I was also hacking on the funnel plugin and realized that having addedPaths in diff result is useful for plugins like funnel and mergeTrees. I implemented this in one of the commits under #2257

IgorMinar · 2015-06-04T05:55:20Z

@caitp still no update? do you need help?

caitp · 2015-06-04T22:09:48Z

https://www.diffchecker.com/fejjguqf here's a diff of the outputs from mergeTrees, with the master branch on the right

caitp · 2015-06-04T22:41:22Z

https://gist.github.com/caitp/b9251a63bba113cf4832 and here is a log of what the tree-differ is checking. It looks like merge-trees never gets notified of the existence of those other files. Are they symlinks perhaps? Would that matter?

caitp · 2015-06-05T03:25:39Z

As mentioned on slack:

The problem seems to be coming from the fact that we are mutating live inputTrees in the tree builder (

angular/tools/broccoli/trees/browser_tree.ts

Lines 105 to 125 in 7b1e928

    
           function writeScriptsForPath(relativePath, result) { 
        
             copyVendorScriptsTo(path.dirname(relativePath)); 
        
             return result.replace('@@FILENAME_NO_EXT', relativePath.replace(/\.\w+$/, '')); 
        
           } 
        
           var htmlTree = new Funnel(modulesTree, {include: ['*/src/**/*.html'], destDir: '/'}); 
        
           htmlTree = replace(htmlTree, { 
        
             files: ['examples*/**'], 
        
             patterns: [{match: /\$SCRIPTS\$/, replacement: htmlReplace('SCRIPTS')}], 
        
             replaceWithPath: writeScriptsForPath 
        
           }); 
        
           htmlTree = replace(htmlTree, { 
        
             files: ['benchmarks/**'], 
        
             patterns: [{match: /\$SCRIPTS\$/, replacement: htmlReplace('SCRIPTS_benchmarks')}], 
        
             replaceWithPath: writeScriptsForPath 
        
           }); 
        
           htmlTree = replace(htmlTree, { 
        
             files: ['benchmarks_external/**'], 
        
             patterns: [{match: /\$SCRIPTS\$/, replacement: htmlReplace('SCRIPTS_benchmarks_external')}], 
        
             replaceWithPath: writeScriptsForPath 
        
           });

,

angular/tools/broccoli/trees/browser_tree.ts

Lines 94 to 103 in 7b1e928

    
           function copyVendorScriptsTo(destDir) { 
        
             servingTrees.push(new Funnel(vendorScriptsTree, {srcDir: '/', destDir: destDir})); 
        
             if (destDir.indexOf('benchmarks') > -1) { 
        
               servingTrees.push(new Funnel(vendorScripts_benchmark, {srcDir: '/', destDir: destDir})); 
        
             } 
        
             if (destDir.indexOf('benchmarks_external') > -1) { 
        
               servingTrees.push( 
        
                   new Funnel(vendorScripts_benchmarks_external, {srcDir: '/', destDir: destDir})); 
        
             } 
        
           }

).

This is a problem for the diffing plugin wrapper, because stabilizeTrees() will replace the live array with a copy of the inital array at the time stabilization occurred, so mergeTrees() never sees all the extra inputTrees that should be there.

The right thing to do here is to just not mutate the live inputTrees, and make sure there is a workable set of inputTrees to begin with. On top of that, in the existing code on master, some duplicate work is actually happening because trees are pushed into the live inputTrees array multiple times. So we should be able to see a slight perf boost from fixing that. (this is probably the reason why overwrite: true is needed at all)

caitp · 2015-06-05T14:41:26Z

The extra {overwrite: true} seems to be needed due to the polymer url_params_for_form script for some reason.

tmp/diffing_plugin_wrapper-output_path-qmFFK91M.tmp/benchmarks_external/src/tree/polymer/url_params_to_form.js is duplicate of tmp/tree_stabilizer-output_path-Z35ehLqR.tmp/benchmarks_external/src/tree/polymer/url_params_to_form.js

caitp · 2015-06-05T19:18:59Z

tools/broccoli/trees/browser_tree.ts

+// TODO(caitp): Figure out how to make this fast, while still being able to
+// determine the trees immediately. Manually maintaining a list of paths to
+// manage seems cumbersome.
+function getServedPaths() {


Per TODO comment above, Igor wants to just manually maintain the list of paths, and we probably want that list to happen later (or maybe be a constant kServedPaths or something.

caitp · 2015-06-05T20:35:29Z

tools/broccoli/trees/browser_tree.ts

@@ -145,7 +184,8 @@ module.exports = function makeBrowserTree(options, destinationPath) {

  es5Tree = mergeTrees([es5Tree, htmlTree]);

-  var mergedTree = mergeTrees([stew.mv(es6Tree, '/es6'), stew.mv(es5Tree, '/es5')]);
+  var mergedTree =
+      stew.log(mergeTrees([stew.mv(es6Tree, '/es6'), stew.mv(es5Tree, '/es5')]), {output: 'tree'});


removed locally, won't be checked in

caitp · 2015-06-05T20:40:01Z

CI failure doesn't seem to be fault of this PR, same error in https://travis-ci.org/angular/angular/jobs/65624606

Closes #1815 Closes #2064

angular-automatic-lock-bot · 2019-09-06T13:21:24Z

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}

caitp added the in progress label May 21, 2015

googlebot added the cla: yes label May 21, 2015

caitp force-pushed the multi-tree branch from 060906a to a348893 Compare May 21, 2015 16:46

caitp added this to the Build System milestone May 21, 2015

caitp assigned IgorMinar May 21, 2015

IgorMinar reviewed May 22, 2015
View reviewed changes

caitp force-pushed the multi-tree branch from a348893 to ff2d5ea Compare May 28, 2015 19:20

caitp added the #development_infrastructure label May 29, 2015

caitp force-pushed the multi-tree branch 2 times, most recently from 7b04472 to bc8bc48 Compare May 29, 2015 00:08

mhevery added area: build & ci Related the build and CI infrastructure of the project and removed area: build & ci Related the build and CI infrastructure of the project #development_infrastructure labels May 29, 2015

mhevery removed this from the Build System milestone May 29, 2015

caitp force-pushed the multi-tree branch from bc8bc48 to 5594882 Compare May 29, 2015 21:47

caitp force-pushed the multi-tree branch 2 times, most recently from 2a8e631 to c28a57e Compare June 4, 2015 21:42

caitp mentioned this pull request Jun 5, 2015

Make inputTrees immutable broccolijs/broccoli-plugin#1

Closed

caitp force-pushed the multi-tree branch from 14bcf13 to cfc5918 Compare June 5, 2015 07:45

caitp force-pushed the multi-tree branch from cfc5918 to 59fbf49 Compare June 5, 2015 15:57

caitp reviewed Jun 5, 2015
View reviewed changes

caitp added 5 commits June 5, 2015 15:56

feat(diffing-broccoli-plugin): support multiple inputTrees

5de74d1

chore(tsd): update fs-extra

7ab3e20

chore(tsd): update jasmine

0e73a3c

feat(broccoli): add diffing MergeTrees plugin

e1bdddd

feat(broccoli): improve merge-trees plugin and add "overwrite" option

2e48136

caitp force-pushed the multi-tree branch 2 times, most recently from 2e8d36c to 7bae3f4 Compare June 5, 2015 19:59

fix(broccoli): ensure that inputTrees are stable

6c69b81

caitp force-pushed the multi-tree branch from 7bae3f4 to 6c69b81 Compare June 5, 2015 20:11

caitp reviewed Jun 5, 2015
View reviewed changes

caitp closed this in 41ae8e7 Jun 5, 2015

caitp added a commit that referenced this pull request Jun 5, 2015

feat(broccoli): add diffing MergeTrees plugin

4ee3fda

Closes #1815 Closes #2064

angular-automatic-lock-bot bot locked and limited conversation to collaborators Sep 6, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(broccoli): add incremental MergeTrees plugin #2064

feat(broccoli): add incremental MergeTrees plugin #2064

caitp commented May 21, 2015

caitp commented May 21, 2015

caitp commented May 21, 2015

caitp commented May 21, 2015

caitp commented May 21, 2015

IgorMinar May 22, 2015

IgorMinar commented May 22, 2015

caitp commented May 22, 2015

caitp commented May 28, 2015

IgorMinar commented May 29, 2015

caitp commented May 29, 2015

caitp commented May 30, 2015

caitp commented May 30, 2015

IgorMinar commented Jun 1, 2015

IgorMinar commented Jun 4, 2015

caitp commented Jun 4, 2015

caitp commented Jun 4, 2015

caitp commented Jun 5, 2015

caitp commented Jun 5, 2015

caitp Jun 5, 2015

caitp Jun 5, 2015

caitp commented Jun 5, 2015

angular-automatic-lock-bot bot commented Sep 6, 2019

feat(broccoli): add incremental MergeTrees plugin #2064

feat(broccoli): add incremental MergeTrees plugin #2064

Conversation

caitp commented May 21, 2015

caitp commented May 21, 2015

caitp commented May 21, 2015

caitp commented May 21, 2015

caitp commented May 21, 2015

IgorMinar May 22, 2015

Choose a reason for hiding this comment

IgorMinar commented May 22, 2015

caitp commented May 22, 2015

caitp commented May 28, 2015

IgorMinar commented May 29, 2015

caitp commented May 29, 2015

caitp commented May 30, 2015

caitp commented May 30, 2015

IgorMinar commented Jun 1, 2015

IgorMinar commented Jun 4, 2015

caitp commented Jun 4, 2015

caitp commented Jun 4, 2015

caitp commented Jun 5, 2015

caitp commented Jun 5, 2015

caitp Jun 5, 2015

Choose a reason for hiding this comment

caitp Jun 5, 2015

Choose a reason for hiding this comment

caitp commented Jun 5, 2015

angular-automatic-lock-bot bot commented Sep 6, 2019