How to exclude specific files?

[jvolker]

I've got a problem excluding files when doing multiple manipulations in one call:

There is set A of images matched by a glob-pattern which I would like to manipulate in a certain way.
Then there is set B matched by a different glob-pattern, which I would like to manipulate differently, but the latter is overlapping with the first one.

How can I efficiently exclude set A from set B?
Ideally, I would just like to pass the first glob-pattern as exclude pattern in the second run, without having to merge it into a single glob pattern.

In node-glob, e.g. there is an ignore option which excepts arrays as well, which seems to be what I'm looking for:

Add a pattern or an array of glob patterns to exclude matches.

Thanks!

[axe312ger]

There is the exclamation mark pattern to ignore things in a glob string. Sth like src/**/!(banana.txt)

If you give me more details on your folder structure I might be able to create the globs u need.

Best,
Benedikt

[jvolker]

Thanks. I know about the exclamation mark pattern. Just find it a bit tedious. :)

In total, I've got images in 3 different places in the hierarchy.

1. images/*.{png,jpg,jpeg}
2. content/projects/*.{png,jpg,jpeg}
3. content/projects/*/images/*.{png,jpg,jpeg}

The first and second one contains images with four different file name endings (e.g. images/*_endingA.jpg) which all have to be treated differently (multiple sizes, cropping, names).
For each of the file name endings, I'm passing a set options/methods to metalsmith-sharp. Some of those file endings will be passed multiple times with different sizing instructions.

My code already looks quite messy. Now I need to exclude images from previous runs from the next run. Not sure if spicing up the glob patterns would make it more readable. But of course, I appreciate any suggestions.

Did you consider allowing to send arrays of sizes, glob patterns, and other commands which would unfold in the library? Would make it a lot easier to use in those situations.

[jvolker]

There is multimatch extending minimatch to support the easy way of just passing multiple (positive and/or negative) patterns.

[jvolker]

I found a way to exclude multiple file name endings in a glob pattern:
images/!(*_endingA|*_endingB).+(png|jpg|jpeg)

One of the patterns I would like to exclude is actually a path to a single file though. That leaves me with something like that I guess:

{images/!(*_endingA|*_endingB).+(png|jpg|jpeg),path/to/file.jpg}

Haven't thoroughly tested it yet but this seems to work. EDIT: Doesn't work. This includes path/to/file.jpg and doesn't exclude it. I'm stuck.

Still, I think it's a bit inconvenient to have to go this far into the details of the glob syntax and would appreciate being able to pass multiple patterns instead.

[axe312ger]

Feels like you shoud consider restructuring your folder structure to be more simple/categorized.

Otherwise, you might try:
{images/!(*_endingA|*_endingB).+(png|jpg|jpeg),!(path/to/file.jpg)}

[jvolker]

Feels like you shoud consider restructuring your folder structure to be more simple/categorized.

Yeah, you might be right I could probably simplify things slightly, though I don't think that's going make it a whole lot easier, unfortunately.

{images/!(*_endingA|*_endingB).+(png|jpg|jpeg),!(path/to/file.jpg)}

Thanks, but it doesn't work, unfortunately.

[jvolker]

I'm tempted to do the following now:

1. Run through the files array myself using multimatch by passing multiple positive and negative glob patterns and store matching files in an array.
2. Build up the options for metalsmith sharp. I would convert the array to this glob pattern {path/to/file1.jpg,path/to/file2.jpg} and use it as source.
3. Pass the options to metalsmith-sharp.

It seems a bit silly, but the easiest thing to do.

[jvolker]

I finally solved it by doing the following:

1. Run through the files array myself using multimatch by passing multiple positive and negative glob patterns and store matching files in an array.

2. I implemented passing options on a per-file basis into this plugin and use it for this purpose. So I'm not passing an array as I had planned previously but attaching options for every image file. This is the corresponding Pull Request.

A side note
I also found this implementation of multimatch in a fork of metalsmith-sharp:
emanueleperuffo@70dc648

[axe312ger]

Hey Volker,

thanks for your effort and the research. I have the feeling that a per file configuration will not be really intuitive and implements a pattern which is (at least for me) new to metalsmith.

I really like what @emanueleperuffo did in his fork. Would that fit for your use-case, too?

https://github.com/sindresorhus/multimatch is done by a serious, well known maintainer so I'd be very open to migrate to that more flexible glob pattern implementation.

Maybe Emanuele is open to create a PR from his fork to this plugin? :)

Best,
Benedikt

[jvolker]

No worries.

I have the feeling that a per file configuration will not be really intuitive and implements a pattern which is (at least for me) new to metalsmith.

To be honest, I was struggling a bit with finding simple words to explain the behaviour in the readme. Maybe rewording that text would help?

I really like what @emanueleperuffo did in his fork. Would that fit for your use-case, too?

It would definitely help with the original issue I started this thread for.

But regarding my case as discussed on Slack, the decisions on what images to process are depending on more than a glob pattern, they depend on metadata of other files. The issue I had was to first determine which images to process and later to pass the corresponding options it to metalsmith-sharp during the same Metalsmith build process.

The only other way I know of solving this would be to run Metalsmith twice for every build. First to figure out which images to process and to build the options for metalsmith-sharp. Second to pass the options to metalsmith-sharp.

What really made me going the per-file route was the opinion of @tsdorsey (on Slack):

I think the ideal metalsmith way would be to allow per file settings in the file metadata to be read by the sharp plugin. The sharp plugin isn't that large or complicated. Might be worth it to fork and add that capability. Then your addMeta step does exactly that, it adds metadata to each image for sharp to consume later.

[tsdorsey]

Hey @axe312ger you mentioned that this pattern is "(at least for you) new to metalsmith". I would just like to point out that it's actually at the core of metalsmith. It's just hidden in that we usually use yaml syntax at the top of a file to do it instead of build time code. As you can imagine, this becomes difficult with binary files so it pretty much has to be done using build code instead.

If you look at the examples on the Metalsmith.io website they use the https://github.com/segmentio/metalsmith-drafts plugin to show how this works. You put an attribute in the file using yaml syntax (in this case, draft: true). The .source method turns all the files in the source directory into objects and adds any attributes from the yaml code to the object. Then a plugin reads those per file attributes and makes a call on what to do with each file (in this case to let them pass or to drop them from the build).

Now, I'm not saying you HAVE to go this route, it's your plugin to maintain as you see fit. 😃I just wanted to point out that it's not an obscure or new pattern.

[axe312ger]

Okay guys, thanks for that clarification.

I would now even go the way to implement both. Having the possibility to negate the glob pattern is very useful (#68).

Having a per-file basis might also be needed by some users, and yes, I totally forgot about that the YAML frontmatter headers actually are per file based settings 👍 Thx @tsdorsey for the great explanation.

So let's get #67 in, too. 🚀

[jvolker]

Sounds great! Thanks a lot, @tsdorsey and @axe312ger.