Recommended way to ignore files in tree?

[gonvaled]

As described here broccoli has problems with emacs temporary files (on-purpose broken symlinks). It was suggested to use a brocignore file, but that seems to be deprecated (#179).

What is the recommended method to globally ignore files in brocoli?

[joliss]

There's currently no way to do that.

Can you share your use case? What specific problem are you having or trying to solve?

[stefanpenner]

for reference, in ember-cli we solved this by moving to our own watcher that does this filtering.

https://github.com/krisselden/broccoli-sane-watcher
https://github.com/krisselden/broccoli-sane-watcher/blob/master/index.js#L9-L11
https://github.com/krisselden/broccoli-sane-watcher/blob/master/index.js#L19
and finally
https://github.com/krisselden/broccoli-sane-watcher/blob/master/index.js#L105

[gonvaled]

@joliss: coming back to ember (and broccolijs) after a long time, this issue is still giving me trouble. My use case is simple: emacs generates backup files and other temporary files, for example files with suffix ~ (my_file.js~). Broccoli watches that. Regularly I clean my temporary files, and broccoli complains:

file deleted templates/_urls.hbs~
ENOENT, no such file or directory 'xxx/tmp/broccoli_merge_trees-output_path-QxtRxWFv.tmp/_urls.hbs~'

The only way to get rid of the problem is to stop and restart ember serve, which is a very annoying workflow.

I would say that a config parameter to ignore files according to a glob or a regexp would help here.

[stefanpenner]

@rwjblue / @ef4 you use emacs, why does this not affect you?

[ef4]

I customized emacs to keep backups in system temp instead, and I disabled lockfiles entirely (a feature of dubious value if you're accustomed to running a singleton emacs daemon anyway):

(setq backup-directory-alist
      `((".*" . ,temporary-file-directory)))
(setq auto-save-file-name-transforms
      `((".*" ,temporary-file-directory t)))
(setq create-lockfiles nil)

But I agree that it would be nice to make broccoli handle these cases gracefully.

I think there are two separate issues being discussed.

One is that the lockfiles are deliberately-dangling symlinks (named like .#your-file.js), where the "destination" path is actually encoding data about who's doing the locking.

Two is that backup files (named like your-file.js~) are created and destroyed frequently, and sometimes that seems to make broccoli sad (or at least it used to, I haven't experimented recently).

[stefanpenner]

Two is that backup files (named like your-file.js~) are created and destroyed frequently, and sometimes that seems to make broccoli sad (or at least it used to, I haven't experimented recently).

we should /~$/ to: https://github.com/broccolijs/broccoli-sane-watcher/blob/master/index.js#L9-L11 @gonvaled do you have some cycles to try a PR ?

The lockfiles seem also an issue, but unrelated to this issue exactly? Although I suspect once, this issue is resolve the lockfiles will become the next issue.

It seems like we would need to guard all symlink reading? Im nervous to add a check to validate all symlinks in walk-sync, i can't imagine that would be a performant change... I'm open to suggestions,

[gonvaled]

@stefanpenner I can try this, but I would need some pointers. afaik I have no broccoli-sane-watcher installed. The only broccoli related stuff in my package.json is:

    "broccoli-asset-rev": "^2.2.0",
    "broccoli-funnel": "^1.0.1",
    "broccoli-merge-trees": "^1.1.0",
    "broccoli-stew": "^1.0.4",
    "broccoli-unwatched-tree": "^0.1.1",

So I do not know how to:

• use the broccoli-sane-watcher
• use a local version for testing (as oposed to the npm version specified in the package.json)

You mention that ember-cli uses broccoli-sane-watcher, but I have an ember-cli generated project, and it is not using that (afaik):

» ember --version
version: 1.13.13
node: 0.12.2
npm: 2.14.10
os: linux x64

Am I using the right ember version? Do I need to configure something to activate broccoli-sane-watcher?

[ghempton]

The case we are struggling with is supporting a project structure that contains root-level files that are of interest. E.g:

src/
test/
index.html
Brocfile.js

Is there currently any way to have a filter apply to . without processing the entirety of the tmp/ directory that is generated by Broccoli?

E.g.

let tree = new Funnel('.', {
  files: ['index.html']
});

Creates enormous slowdowns and causes the watcher to enter infinite watch loops triggered by Broccoli's own changes to the tmp/ dir.

[stefanpenner]

Creates enormous slowdowns and causes the watcher to enter infinite watch loops triggered by Broccoli's own changes to the tmp/ dir.

This is largely an OS limitation. e.g. FSEvents only understand recursive watching...

For this use-case, i've wanted the ability to specify watcher: 'polling' on a specific input tree. That way we could get the best of both worlds (only use polling for these one-off files), and system FS watching for large trees we care about.

This shouldn't be very hard to add, but would require buy-in, and changes to the builder/watcher relationship.

Maybe the high level API would be something like:

let tree = new Funnel('.', {
  files: ['index.html'],
  watcher: { type: 'polling', interval: 1000 } // or maybe something more generic like "per file"
})

[ghempton]

That would be nice!

[stefanpenner]

We do have these broccoli-source primitives. They currently support WatchedDir and UnwatchedDir. Now that broccoli 1.0 bundles sane watcher, we could allow WatchedDir to be configurable, given it control over how broccoli (via sane) watches it e.g. polling vs events vs none ..

[noxecane]

Any progress on this. I am new to ember/broccolli so I don't understand most of what you are saying(with regards to broccolli source and co)

[ef4]

@noxecane here's a recent discussion on how to remove files from a broccoli tree. Please chime in there if it's relevant and you have more questions, or make a new topic and we can point you to the right resources.

[noxecane]

Thanks @ef4 it did work.

[noxecane]

To be clear, this is what I did

const Funnel = require('broccoli-funnel');
const EmberApp  = .....

module.exports = function(defaults) {
  let app = new EmberApp(defaults, {
    trees: {
      app: (new Funnel('app', {
        exvlude: ['**/.#*.js',  '**/.#*.hbs', '**/.#*.scss']
        }))
    }
  });