More detailed passthrough copy options

[edwardhorsford]

I'm trying to use manual passthrough copy to copy some files over to my dist directory - but the outcome is not what I expected. I don't know if this is intentional, but it seems odd.

I've got a folder of files I'd like to be copied. They're at src/copy-pass-through/. I added them to the eleventy config with eleventyConfig.addPassthroughCopy("./src/copy-pass-through");

They get copied to dist, but the starting folder structure is preserved. So I get dist/src/copy-pass-through

I would expect the files to be copied relative to the starting location. So a file located in src/copy-pass-through/images/foo.jpg would get copied to dist/images/foo.jpg.

The current behavior seems odd to me - if my files are in a folder, I'm forced to have that appear in dist. This doesn't support my own organisation. It means if I want a file to appear in the root of dist I'm forced to have it in the root of the project.

[kleinfreund]

It would certainly be nice to be able to structure a project by having an input directory, an output directory and at the root level, project configuration (e.g. .eleventy.js, package.json, etc.).

[zachleat]

Interesting request! I see the benefit there.

Maybe we should add an optional second argument to addPassthroughCopy to declare a output directory target.

eleventyConfig.addPassthroughCopy("./src/copy-pass-through", "/"); would copy files to the root output directory, maintaining directory structures from that root.

OR, it may be better to create a new method instead. Maybe eleventyConfig.addPassthroughCopyFromRoot("./src/copy-pass-through"); I don’t like this name, I would welcome suggestions!

I’m leaning toward the latter, if the name is good.

[zachleat]

This repository is now using lodash style issue management for enhancements. This means enhancement issues will now be closed instead of leaving them open.

The enhancement backlog can be found here: https://github.com/11ty/eleventy/issues?utf8=%E2%9C%93&q=label%3Aneeds-votes+sort%3Areactions-%2B1-desc+

Don’t forget to upvote the top comment with 👍!

[edwardhorsford]

I'm not sure I entirely follow what you intend the second option to do. Act as the current function does, and change the existing one to match my expected behaviour? or the other way around?Changing the existing method isn't backwards compatible of course.

My instinct is that the current behavior of preserving directory structure before the path you've specified is not a common need. If someone wants a directory structure to appear in their output, they can recreate it from the point they've specified as their source. But to have a file copied through to the root of the destination currently requires it to exist in the root of the project.

I think I'd have more of a preference for the first option, as it more closely matches what I'm familiar with - such as with gulp or grunt. Both of those frequently have a source and destination directory. This gives you flexibility about how you want to structure your files.

In gulp / grunt it's often done as:

src: "copy-pass-through/**.*"
dest: "/*"

In the short term, I've bypassed this method by doing my copying with gulp rather than eleventy.

[zachleat]

Oh, no I wouldn’t break backwards compatibility.

Your instinct may be correct there. If it is, a new method might allow us to change the main path in the docs and de-emphasize the current method in order to minimize confusion moving forward.

Maybe eleventyConfig.addPassthroughCopyContents?

All of my projects rely on the current method’s behavior 🤷‍♂️

[philhawksworth]

I certainly agree with @edwardhorsford's description of how he would expect that to behave. I would also tend to want to have the easier control over the resultant file structure that this delivers.

When it comes to the implementation options suggested, I find the first idea, where an optional parameter dictates if the paths created by the pass through are relative to the root or not, a little hard to comprehend from looking at it.

While tricky to name, I think a new method with the relative path preserving behaviour, might be the most easy to comprehend down the line.

Naming it though... Tricky tricky!

eleventyConfig.addPassthroughCopyRelative ?

[jevets]

Just throwing this out there.
What about allowing a data file determine the resultant output paths, with an object?

{ "src-glob": "dest" }

src/
  copy-pass-through/
    copy-pass-through.json
    docs/
    icons/
    colors/

// copy-pass-through.json
{
  "docs/*.pdf": "/assets/pdf/",
  "icons/favicon.ico": "/",
  "icons/*": "/icons",
  "colors/*.md": false
}

And along with that, from .eleventyConfig.js:

eleventyConfig.addPassthroughCopy('string'); // gets treated as it currently does

eleventyConfig.addPassthroughCopy({ // object as (src glob): (dest)
  'docs/*.pdf': '/assets/pdf/',
  'icons/favicon.ico': '/',
  'icons/*': '/icons',
  'colors/*.md': false // don't output
})

Maybe this idea can help spark some other ideas to help us get a bit closer... This one is tricky, indeed.

[zachleat]

I like that @jevets!

[MadeByMike]

I needed something like this so I have a PR that has about half the work done: #452

Is anyone interested in collaborating on the rest of what is mentioned here?

[jgarber623]

Big 👍 to this feature request.

My preference is to update the existing method (eleventyConfig.addPassthroughCopy()) to accept an optional second parameter, which is interpreted as being relative to the configured output directory.

const inputDir = './src';
const outputDir = './public';

module.exports = eleventyConfig => {
  // copy everything in `./src/_assets` to `./public/assets`
  eleventyConfig.addPassthroughCopy(`${inputDir}/_assets`, 'assets');

  return {
    dir: {
      input: inputDir,
      output: outputDir,
      passthroughFileCopy: true
    }
  }
};

[MadeByMike]

I've been working on this and I'm struggling with this pattern:

eleventyConfig.addPassthroughCopy({ // object as (src glob): (dest)
  'docs/*.pdf': '/assets/pdf/',
  'icons/favicon.ico': '/',
  'icons/*': '/icons'
})

The problem with this is that it makes it difficult to recursively copy directories. What should be do here:

eleventyConfig.addPassthroughCopy({ // object as (src glob): (dest)
  'static/**/*': '/all/',
})

Do we flatten the directory structure and put all files, no matter how deep in the directory /all? Obviously this will override things.

Otherwise we look for the start of a regex and decide this is the base? But what if I wanted to copy all files in nested CSS directories, to a base CSS folder?

eleventyConfig.addPassthroughCopy({ // object as (src glob): (dest)
  'static/**/css/*': '/css/',
})

If I look for the start of a regex here I'm going to get the unwanted folders in the output :(

No idea how to solve. Please HALP!

[kleinfreund]

It looks like this API will be very complex; thus, it should be specified clearly. We need a set of test cases of input globs and their expected output paths that we agree on. Here are some points I think should be handled by the API.

• No leading slashes. The output paths must be relative to and inside Eleventy’s output directory. The input globs must be relative to and inside Eleventy’s input directory. If either rule is broken, throw an error.
• If an object will be used as a map of input globs to output paths, both properties and their values are always strings. Something like 'colors/*.md': false doesn’t make sense to me. Do that in .eleventyignore.

In general, I’m not sure if we should allow globs at all there. A mapping like static/**/css/* → css could mean many different things and people will almost certainly expect it to a different thing than the one that was chosen.

The core of this feature request is this: It should be able to string an input path’s prefix from the output path when passthrough-copying files. That is:

• By default, the contents of static/img end up in $outputDir/static/img.
• Optionally, the contents of static/img can end up in a different directory inside the output directory: $outputDir/$mappedOutputPath. An empty string would result in $outputDir. Setting the output path to assets would result in $outputDir/assets.

[Ryuno-Ki]

mapping like static/**/css/* → css

I can see several meanings:

• Flat everything in $outputDir/css
• Preserve the directory structure (that is static/theme/css/colours.css would become $outputDir/theme/css/colours.css etc.)

What about using some placeholders?

static/?category/css/* → css/?category/*

(not fully thought through the implications, just want to bring up ideas).

@edwardhorsford I believe you brought it up first here. What would be your expectation?

Do we have prior art? It reminds me on Gruntfiles...

[jevets]

@kleinfreund I agree with what you're saying above. The leading slash was intended to imply relativity to the root of the designated output path (per glob), but I agree it's better to avoid that ambiguity entirely.

Getting back to the original issue and @zachleat's initial solution idea, though...

The real goal of this feature seems more about keeping things organized locally but modifying how and where some files/folders end up when built. i.e. keep a static assets folder but copy its contents to _site on build. The main benefit is keeping a cleaner source during development. (The most pain I've dealt with in this limitation is when I'm migrating an existing project into Eleventy and I'd prefer not to move files and change paths.)

One solution is to use an npm package script to handle the copying, but this doesn't work well during development (eleventy --serve), if at all, unless you bring your own local dev server and run eleventy --watch alongside it.

I've just dealt with it and kept my static assets right in input.dir. Yes it feels kinda gross, but I get over it soon enough.

---

Recently I've been thinking it could be better to offer an after hook of sorts in eleventy's build step. For example, if I could:

// .eleventy.js

module.exports = eleventyConfig => {
  // this could run on --watch, --serve, and (build)
  eleventyConfig.afterBuild(() => {
    // fs.copyFileSync(...)
  })
}

This way we could handle passthrough directory mapping and anything else we may need to do.

Or if the existing Transforms could somehow be used to alter the paths after the fact. (Copy Passthroughs currently don't go through Transforms.)

Just an idea, thinking out loud.

[MadeByMike]

I've updated the pull request with what I think is a passable solution: #452

Basically old API works. That means if there is no output path it's resolved automatically. For both these the output would be resolved to static/js assuming the root inputDirectory is src/site:

config.addPassthroughCopy({"src/site/static/js": true});
config.addPassthroughCopy("src/site/static/js");

If it's a glob, it's just going to copy everything to the specified output directory. That means both these resolve all files to /js. The first one looks for any files in any js directory in static, no matter how nested. The other lookes for .js files directly in static. Key thing is, both resolve to /js without preserving the directory structure from the glob.

config.addPassthroughCopy({"src/site/static/**/js/*": "/js"});
config.addPassthroughCopy({"src/site/static/*.js": "/js"});

If it's not a glob and the key resolves to a directory or file, it will use the recursive copy method and you can rename the directory just as above but keeping the source directory structure. The only limitation here is you will get all files. Eg map static to files.

config.addPassthroughCopy({"src/site/static/": "/files"});

[MadeByMike]

@zachleat going to run this on my site for a while and see how it goes. Would love to get some feedback from you.

Also fast-glob has an option of matching directories as well. I considered turning this on which would result in being able to target a directory with and copy everything inside it but not filter by file types etc... so maybe not useful :(

If we did that, it would mean with this example any nested js directory would be mapped to /js keeping any subfolders in the js directories...

config.addPassthroughCopy({"src/site/static/**/js/": "/js"});

[edwardhorsford]

@edwardhorsford I believe you brought it up first here. What would be your expectation?

I think what's being proposed is not what I would have expected - and doesn't meet my particular needs. I'd like my folder structure preserved - I think what's being proposed flattens the structure.

I'd like to provide two locations. Point A and point B
I'd like the files and folder structure at point A to be copied to point B.

I always assumed **/* meant 'recursive' copy - perhaps it actually has a different meaning though.

---

FWIW all my images are in nested folders by date. This keeps them nicely grouped - but also means I don't need to worry about file name collisions. It's also super helpful as I use cloudinary as my imageCDN - and my assets locations there mirror my local assets.

[MadeByMike]

@edwardhorsford You can still copy folders just not with globs. So if you want the entire js directory you'd just go:

config.addPassthroughCopy({"src/site/static/js/": "/js"});

This works for a lot of use-cases. The one that is tricky is this... imagine the following folders all contain .js files:

static/component-1/js
static/component-1/js/nested-1/
static/component-1/js/nested-2/
static/component-2/js
static/component-2/js/nested-1/
static/component-2/js/nested-2/

And you want to copy all the **/js folders to a /js folder in the root, dropping the component-1 or component-2 part? This PR cannot do that and I'm not sure that the solution is simple.

It's easy to match files, but the tricky part in our situation is what do you want to remove? With all other cases we're assuming you want to remove everything before the matched filename, e.g:

config.addPassthroughCopy({"src/site/static/js/": "/js"}); 
// Removes:  src/site/static/js/ from matched paths and copies to /js

config.addPassthroughCopy({"src/site/static/js/file.js": "/my-script/"}); 
// Also removes:  src/site/static/js/ from matched paths and copies to /js

What do we do for globs? At the moment we are doing the same. Is it as simple as stripping everything before ** and removing this from matched paths before copying??

[MadeByMike]

config.addPassthroughCopy({"/static/**/js" : "/js"});
// Remove: /static/ from matched paths and copy to /js ???

config.addPassthroughCopy({"/static/**/js/*.js" : "/js"});
// Also remove: /static/ from matched paths and copy to /js ???

config.addPassthroughCopy({"/static/**/components/**/js/*.js" : "/js"});
// Also remove: /static/ from matched paths and copy to /js ???

I don't know if the above is what people expect? Maybe it is as simple as removing everything before the first ** from matched files before copying?

[zachleat]

Okay to remove this one from the enhancement queue because the PR was merged? Or are there additional things documented left open here?

[MadeByMike]

There is a suggested change for how remap paths with globs containing ** if enough people support it I'd be happy to do it. We should do it quickly as it's a breaking change from what was just merged.

[zachleat]

@MadeByMike is this a globstar thing? https://github.com/mrmlnc/fast-glob#globstar

[MadeByMike]

@zachleat the globstar option is about how we want it to match files. I assume the default of true is the most common behaviour for globs and we should leave it?

The extra discussed is about what we should do with the copy after matching a glob. The question is, given the directory structure: src/nested/file.txt and the config {'src/**/file.txt': '/dest'}. Should it copy to: /dest/file.txt or dest/nested/file.txt?

@zachleat What do you think the users expectation is?

Also what about this one...

Given the directory structure: src/nested-1/files/nested-2/file.txt and the config {'src/**/files/**/file.txt': '/dest'} should it copy to: /dest/file.txt or dest/nested-1/files/nested-2/file.txt? or something else?

[zachleat]

I think any time you’re changing the output directory, it should flatten the matches to the output directory. If they want to maintain directory structure, they should use the prior behavior with a string argument.

An example: assuming input directory of src and output directory of _site, file src/nested/file.txt:

// Pick one
eleventyConfig.addPassthroughCopy("src/**/file.txt"); // copy to _site/nested/file.txt
eleventyConfig.addPassthroughCopy("src/**/file.txt": "dest"); // copy to _site/dest/file.txt
eleventyConfig.addPassthroughCopy("src/**/file.txt": "dest/nested"); // copy to _site/dest/nested/file.txt

I think there are plenty of options here to make it flexible enough—unless anyone has any other complaints here. I don’t think we need to get too elaborate.

[zachleat]

Docs are up at https://www.11ty.io/docs/copy/#using-globs and https://www.11ty.io/docs/copy/#change-the-output-directory

[hidegh]

@MadeByMike I am using npm glob for recursive file search then fs.mkdir(path, { recursive: true }) and fs.copyFile to do the trick. See: #379 (comment)

Also when using passthrough, make sure that it works with permalink settings too!

[maddsua]

Would be really nice being able to specify addPassthroughCopy({'src/_static': '/'}) or something similar so 11ty would dump all the contents of that static directory directly into the public.

Relative paths are actually still quite broken. For an example, EleventyRenderPlugin for some reason completely ignores 11ty's input directory path and takes everything relative to where you run node. Oh wait, addPassthroughCopy does that too.

Or is that intended to be this way?

upd: Is it just me skipping over entire blocks of docs, or was this actually added recently? All good now 😂