Watcher semantics are incorrect for use with BROCCOLI_ENABLED_MEMOIZE

[ef4]

Places like this:

ember-cli/lib/broccoli/ember-app.js

Line 277 in 3527827

stylesTree = new UnwatchedDir(this._stylesPath);

that use UnwatchedDir are assuming that complete rebuilds will be triggered by the parent path's watcher. But that violates the assumptions that make BROCCOLI_ENABLED_MEMOIZE work. It's supposed to know which trees need to rebuild based on which files the watcher mentions. An UnwatchedTree will never rebuild, and then its downstream trees will also not rebuild.

However, it turns out this bug is cancelled out by other bugs, and apps do still get rebuilding of their styles. But it happens by accident, because

1. the app styles tree gets merged with many other trees and then and re-funneled before getting processed, which (incorrectly) causes it to depend on additional trees that are WatchedDirs and not UnwatchedDirs. This contagion causes later stages to run even though app.trees.styles itself never updates.
2. We get lucky with the broccoli directory symlinking. So even though the app.trees.styles tree never updates, the changes propagate forward via the symlinked directory.

I came upon this behavior because of the way Embroider consumes the app's styles. It depends on app.trees.styles, and that works with BROCCOLI_ENABLED_MEMOIZE off because even though that tree never updates, trees downstream from it would always run and would follow the symlinking to the updated files. But with BROCCOLI_ENABLED_MEMOIZE enabled, app CSS fails to rebuild under embroider because we really need the tree to update correctly.

[rwjblue]

FWIW, the reason this is using UnwatchedDir is that app/styles is within the app tree. The fix here would be to confirm that we are using the nested dir, and use a funnel off of the trees.app instead of making a second UnwatchedDir. This way the downstream tree would properly invalidate/rerun upon change, but still avoid indicating two files changed (one for styles/foo.scss and another for foo.scss which is what was happening waaaaaay back when).

[ef4]

That makes sense. So I think in detail it would be:

1. If the user provided their own broccoli tree as app.trees.styles, we just use that and there's nothing else to do.
2. If the user provided a string (or nothing, which we default to the string app/styles) for app.trees.styles, we need to know whether it's nested inside their app tree.
   1. If the user provided a string (or nothing, which we default to the string app) for their app.trees.app, we can compare that with their app.trees.styles string to see if we have a subdir
      1. If its a subdir, we can funnel off the app tree.
      2. Otherwise we make a new watched root.
   2. if the user provided a broccoli tree as app.trees.app, we are out of luck and can't tell whether the styles are nested inside it. We could make a watched root and just accept the possibility of double watching.

[angelayanpan]

was this resolved? @ef4 did anything change from 2020?

[ef4]

I don't think anything is changed, embroider still ships with a workaround for this. But if you are seeing a problem in your apps, please report it because the workaround in embroider is supposed to make this not matter.

[angelayanpan]

in classic ember build, we are not able to use the BROCCOLI_ENABLED_MEMOIZE flag because the re-build doesn't update scss changes.

[ef4]

Ah, so when I reported this it wasn't actually causing any problems in classic apps. So maybe something has changed.