-
Notifications
You must be signed in to change notification settings - Fork 25.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(broccoli): add incremental MergeTrees plugin #2064
Conversation
TODO: add tests, fix the dart build which seems to be messed up. |
There's a problem with this --- if the inputTrees come from another incremental plugin, there's no way to know if the "real" source file was removed (in which case the output should be removed), or if an input was not included in the last build process. So what this means is, right now, the mergeTrees plugin will delete any specs because it doesn't think they're around anymore. That's not very good illustration: EchoBeach:angular caitp$ gulp test.unit.cjs
Dart SDK detected
[13:10:10] Using gulpfile ~/git/angular/gulpfile.js
[13:10:10] Starting 'build/clean.js'...
[13:10:10] Starting 'build/clean.tools'...
[13:10:10] Finished 'build/clean.tools' after 1.38 ms
[13:10:10] Starting 'build.tools'...
[13:10:10] Starting '!build.tools'...
[13:10:11] Finished 'build/clean.js' after 433 ms
[13:10:15] Finished '!build.tools' after 4.1 s
[13:10:15] Finished 'build.tools' after 4.11 s
[13:10:15] Starting 'test.unit.cjs'...
[13:10:15] Starting '!build.js.cjs'...
Tree diff: funnel-output_path-cxvFz9UF.tmp, duration: 56ms, 208 changes detected (files: 208, directories: 167)
Tree diff: replace_filter-tmp_dest_dir-ZJvM0X0C.tmp, duration: 21ms, 416 changes detected (files: 416, directories: 55)
Tree diff: funnel-output_path-9hlvAMyv.tmp, duration: 0ms, 1 changes detected (files: 1, directories: 1)
Tree diff: diffing_plugin_wrapper-output_path-DHPT9XAs.tmp, duration: 23ms, 417 changes detected (files: 417, directories: 55)
Tree diff: funnel-output_path-YUWFgTka.tmp, duration: 0ms, 1 changes detected (files: 1, directories: 1)
Tree diff: diffing_plugin_wrapper-output_path-VAUHrzcS.tmp, duration: 21ms, 418 changes detected (files: 418, directories: 55)
Tree diff: funnel-output_path-2Rh5EuAW.tmp, duration: 1ms, 1 changes detected (files: 1, directories: 1)
Tree diff: funnel-dest_UULMeC.tmp, duration: 24ms, 128 changes detected (files: 128, directories: 167)
Tree diff: diffing_plugin_wrapper-output_path-A8wXyHUC.tmp, duration: 22ms, 419 changes detected (files: 419, directories: 55)
Tree diff: diffing_plugin_wrapper-output_path-QmXT90nW.tmp, duration: 14ms, 384 changes detected (files: 384, directories: 24)
Tree diff: funnel-dest_1liPKx.tmp, duration: 2ms, 27 changes detected (files: 27, directories: 15)
Tree diff: lodash_renderer-dest_dir-TUPYyHtV.tmp, duration: 0ms, 3 changes detected (files: 3, directories: 3)
Tree diff: funnel-dest_GqlAaQ.tmp, duration: 39ms, 833 changes detected (files: 833, directories: 87)
Slowest Trees | Total
----------------------------------------------+---------------------
DiffingTraceurCompiler | 6604ms
DiffingTSCompiler | 4859ms
Slowest Trees (cumulative) | Total (avg)
----------------------------------------------+---------------------
DiffingTraceurCompiler (1) | 6604ms
DiffingTSCompiler (1) | 4859ms
MergeTrees (4) | 875ms (218 ms)
creating node_modules symlink hack
creating link dist/js/cjs/node_modules/angular2 ../angular2
creating link dist/js/cjs/node_modules/benchmarks ../benchmarks
creating link dist/js/cjs/node_modules/benchmarks_external ../benchmarks_external
creating link dist/js/cjs/node_modules/benchpress ../benchpress
creating link dist/js/cjs/node_modules/examples ../examples
creating link dist/js/cjs/node_modules/rtts_assert ../rtts_assert
[13:10:28] Finished '!build.js.cjs' after 14 s
[13:10:28] Starting 'test.unit.cjs/ci'...
Started
..............................*.....................................................*.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................*.................................................................................................................................................................................................................................................................................................................................................................
Pending:
1) change detection dynamic change detection error handling should wrap exceptions into ChangeDetectionError
No reason given
2) change detection JIT change detection error handling should wrap exceptions into ChangeDetectionError
No reason given
3) Form Model ControlGroup optional components valueChanges should not fire an event when an excluded control is updated
No reason given
1040 specs, 0 failures, 3 pending specs
Finished in 4.816 seconds
[13:10:35] Finished 'test.unit.cjs/ci' after 6.66 s
==================================================
WATCH TRIGGERED BY FILE CHANGE #1
(modules/angular2/src/facade/async.ts touched in another terminal)
On: 5/21/2015 at 1:10:39 PM
==================================================
[13:10:39] Starting '!build.js.cjs'...
Tree diff: funnel-output_path-cxvFz9UF.tmp, duration: 27ms, 0 changes detected (files: 208, directories: 167)
Tree diff: replace_filter-tmp_dest_dir-ZJvM0X0C.tmp, duration: 6ms, 416 changes detected (files: 0, directories: 0) /* long pointless array of changed files */
Tree diff: funnel-output_path-9hlvAMyv.tmp, duration: 0ms, 0 changes detected (files: 1, directories: 1)
Tree diff: diffing_plugin_wrapper-output_path-DHPT9XAs.tmp, duration: 8ms, 416 changes detected (files: 1, directories: 55) /* long pointless array of changed files */
Tree diff: funnel-output_path-YUWFgTka.tmp, duration: 0ms, 0 changes detected (files: 1, directories: 1)
Tree diff: diffing_plugin_wrapper-output_path-VAUHrzcS.tmp, duration: 8ms, 416 changes detected (files: 2, directories: 55) /* long pointless array of changed files */
Tree diff: funnel-output_path-2Rh5EuAW.tmp, duration: 0ms, 0 changes detected (files: 1, directories: 1)
Tree diff: funnel-dest_UULMeC.tmp, duration: 25ms, 0 changes detected (files: 128, directories: 167)
Tree diff: diffing_plugin_wrapper-output_path-A8wXyHUC.tmp, duration: 10ms, 416 changes detected (files: 3, directories: 55) /* long pointless array of changed files */
Tree diff: diffing_plugin_wrapper-output_path-QmXT90nW.tmp, duration: 13ms, 0 changes detected (files: 384, directories: 24)
Tree diff: funnel-dest_1liPKx.tmp, duration: 2ms, 0 changes detected (files: 27, directories: 15)
Tree diff: lodash_renderer-dest_dir-TUPYyHtV.tmp, duration: 0ms, 3 changes detected (files: 0, directories: 0) /* long pointless array of changed files */
Tree diff: funnel-dest_GqlAaQ.tmp, duration: 37ms, 419 changes detected (files: 414, directories: 87) /* long pointless array of changed files */
Slowest Trees | Total
----------------------------------------------+---------------------
ReplaceFilter | 349ms
Funnel | 246ms
MergeTrees | 121ms
ReplaceFilter | 117ms
MergeTrees | 81ms
DestCopy | 74ms
MergeTrees | 72ms
MergeTrees | 71ms
Slowest Trees (cumulative) | Total (avg)
----------------------------------------------+---------------------
ReplaceFilter (2) | 466ms (233 ms)
MergeTrees (4) | 346ms (86 ms)
Funnel (10) | 329ms (32 ms)
DestCopy (1) | 74ms
[13:10:40] Finished '!build.js.cjs' after 1.28 s
[13:10:40] Starting 'test.unit.cjs/ci'...
Started
No specs found
Finished in 0.001 seconds
[13:10:40] Finished 'test.unit.cjs/ci' after 295 ms |
interestingly, |
so um, any ideas there? |
let inputPath = this.inputPaths[index]; | ||
let existsLater = (relativePath) => { | ||
for (let i = treeDiffs.length - 1; | ||
i > index; --i) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
clang-format bug?
my theory is that the reason why this is not working is that you are hitting #2051 incremental plugins producing input trees should not be a problem because the input tree is always complete even if it was incrementally updated. so you will never get an input tree just with the incremental updates, it will always be a full tree that contains incremental updates. |
Yeah, when I was looking at your patch it occurred to me that that's probably what this was. I think we should get 2053 in today |
PTAL |
7b04472
to
bc8bc48
Compare
the first commit looks good. the second commit could use some tests. can you add them please? |
okay, I added "some" test coverage --- I'm not sure why travis is complaining about "toBe" on the Matchers interface, since I can see it locally quite plainly. But it will probably complain about the thing fixed by https://github.com/angular/DefinitelyTyped/pull/3 too when it gets around to it |
At the moment a bunch of integration tests fail, investigating |
nah, same tests fail locally on master too. I don't think this PR is breaking em, but not sure why travis is green for the master branch |
yeah, the original merger is conservative and throws when duplicates are detected. I think that this is a good default and avoids accidental over-writes. I was also hacking on the funnel plugin and realized that having |
@caitp still no update? do you need help? |
2a8e631
to
c28a57e
Compare
https://www.diffchecker.com/fejjguqf here's a diff of the outputs from mergeTrees, with the master branch on the right |
https://gist.github.com/caitp/b9251a63bba113cf4832 and here is a log of what the tree-differ is checking. It looks like merge-trees never gets notified of the existence of those other files. Are they symlinks perhaps? Would that matter? |
As mentioned on slack: The problem seems to be coming from the fact that we are mutating live inputTrees in the tree builder ( angular/tools/broccoli/trees/browser_tree.ts Lines 105 to 125 in 7b1e928
angular/tools/broccoli/trees/browser_tree.ts Lines 94 to 103 in 7b1e928
This is a problem for the diffing plugin wrapper, because The right thing to do here is to just not mutate the live inputTrees, and make sure there is a workable set of inputTrees to begin with. On top of that, in the existing code on master, some duplicate work is actually happening because trees are pushed into the live inputTrees array multiple times. So we should be able to see a slight perf boost from fixing that. (this is probably the reason why |
The extra
|
// TODO(caitp): Figure out how to make this fast, while still being able to | ||
// determine the trees immediately. Manually maintaining a list of paths to | ||
// manage seems cumbersome. | ||
function getServedPaths() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Per TODO comment above, Igor wants to just manually maintain the list of paths, and we probably want that list to happen later (or maybe be a constant kServedPaths
or something.
2e8d36c
to
7bae3f4
Compare
@@ -145,7 +184,8 @@ module.exports = function makeBrowserTree(options, destinationPath) { | |||
|
|||
es5Tree = mergeTrees([es5Tree, htmlTree]); | |||
|
|||
var mergedTree = mergeTrees([stew.mv(es6Tree, '/es6'), stew.mv(es5Tree, '/es5')]); | |||
var mergedTree = | |||
stew.log(mergeTrees([stew.mv(es6Tree, '/es6'), stew.mv(es5Tree, '/es5')]), {output: 'tree'}); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removed locally, won't be checked in
CI failure doesn't seem to be fault of this PR, same error in https://travis-ci.org/angular/angular/jobs/65624606 |
This issue has been automatically locked due to inactivity. Read more about our automatic conversation locking policy. This action has been performed automatically by a bot. |
Closes #1815