Error: "Multiple passthrough copy files are trying to write to the same output file"

[nhoizey]

Describe the bug

Trying to upgrade from 0.12.1 to 1.0.0, I get an error during build with the TemplatePassthroughManager:

[11ty] Unhandled rejection in promise: (more in DEBUG output)
[11ty] > Having trouble copying

`TemplateWriterWriteError` was thrown
[11ty] > Having trouble copying './src/**/*.{jpg,png,gif,svg,kmz,zip,css}'

`TemplatePassthroughManagerCopyError` was thrown
[11ty] > Multiple passthrough copy files are trying to write to the same output file (_site/.well-known/favicon.jpg). src/.well-known/favicon.jpg and src/.well-known/favicon.jpg

`TemplatePassthroughManagerCopyError` was thrown:
[11ty]     TemplatePassthroughManagerCopyError: Multiple passthrough copy files are trying to write to the same output file (_site/.well-known/favicon.jpg). src/.well-known/favicon.jpg and src/.well-known/favicon.jpg
        at /Users/nhoizey/Documents/Devs/@perso/nicolas-hoizey.com/node_modules/@11ty/eleventy/src/TemplatePassthroughManager.js:120:19
        at async Promise.all (index 0)
[11ty] Problem writing Eleventy templates: (more in DEBUG output)
[11ty] > Having trouble copying

`TemplateWriterWriteError` was thrown
[11ty] > Having trouble copying './src/**/*.{jpg,png,gif,svg,kmz,zip,css}'

`TemplatePassthroughManagerCopyError` was thrown
[11ty] > Multiple passthrough copy files are trying to write to the same output file (_site/.well-known/favicon.jpg). src/.well-known/favicon.jpg and src/.well-known/favicon.jpg

`TemplatePassthroughManagerCopyError` was thrown:
[11ty]     TemplatePassthroughManagerCopyError: Multiple passthrough copy files are trying to write to the same output file (_site/.well-known/favicon.jpg). src/.well-known/favicon.jpg and src/.well-known/favicon.jpg
        at /Users/nhoizey/Documents/Devs/@perso/nicolas-hoizey.com/node_modules/@11ty/eleventy/src/TemplatePassthroughManager.js:120:19
        at async Promise.all (index 0)
[11ty] Copied 121 files / Wrote 0 files in 24.35 seconds (v1.0.0)

To Reproduce

This project runs on 0.12.1 without issue: https://github.com/nhoizey/nicolas-hoizey.com/

Upgrading Eleventy to 1.0.0 and running the build gives the error above.

.eleventy.js contains this configuration:

  eleventyConfig
    .addPassthroughCopy(
      path.join(config.dir.src, '**/*.{jpg,png,gif,svg,kmz,zip,css}')
    )
    .addPassthroughCopy(path.join(config.dir.src, 'assets'))
    .addPassthroughCopy(path.join(config.dir.src, '.well-known'))
    .addPassthroughCopy(path.join(config.dir.src, '.htaccess'))
    .addPassthroughCopy(path.join(config.dir.src, '_headers'))
    .addPassthroughCopy(path.join(config.dir.src, 'opensearch.xml'))
    .addPassthroughCopy(path.join(config.dir.src, 'manifest.webmanifest'));

Both '**/*.{jpg,png,gif,svg,kmz,zip,css}' and '.well-known' match the same file: src/.well-known/favicon.jpg.

It looks like 1.0 behaves differently than 0.12.1. Either 0.12.1 was copying the file twice, or it first got the list, and removed duplicates before actually copying.

Expected behavior

The build should run as before.

Environment:

• OS and Version: macOS 12.1
• Eleventy Version: 0.12.1 and 1.0.0

[pdehaan]

Interesting.
I can repro w/ 1.0.0-canary.2 as well (but similarly not in v0.12.1).

Sample repo at https://github.com/pdehaan/11ty-2174

> Executing task: npm run build <


> 11ty-2174@1.0.0 build
> DEBUG=Eleventy:TemplatePassthroughManager eleventy

@11ty/eleventy v1.0.1-canary.2
  Eleventy:TemplatePassthroughManager Resetting counts to 0 +0ms
  Eleventy:TemplatePassthroughManager TemplatePassthrough copy started. +9ms
  Eleventy:TemplatePassthroughManager `addPassthroughCopy` config API paths: { 'src/.well-known': true, 'src/**/*.{jpg,svg,png}': true } +0ms
  Eleventy:TemplatePassthroughManager `addPassthroughCopy` config API normalized paths: [ { inputPath: './src/.well-known', outputPath: true }, { inputPath: './src/**/*.{jpg,svg,png}', outputPath: true } ] +1ms
  Eleventy:TemplatePassthroughManager TemplatePassthrough copying from config: { inputPath: './src/.well-known', outputPath: true } +0ms
  Eleventy:TemplatePassthroughManager TemplatePassthrough copying from config: { inputPath: './src/**/*.{jpg,svg,png}', outputPath: true } +0ms
  Eleventy:TemplatePassthroughManager Copied './src/.well-known' (1 files) +53ms
[11ty] Problem writing Eleventy templates:
[11ty] Having trouble copying

`TemplateWriterWriteError` was thrown
[11ty] Having trouble copying './src/**/*.{jpg,svg,png}'

`TemplatePassthroughManagerCopyError` was thrown
[11ty] Multiple passthrough copy files are trying to write to the same output file (www/.well-known/favicon.png). src/.well-known/favicon.png and src/.well-known/favicon.png

`TemplatePassthroughManagerCopyError` was thrown:
[11ty] Copied 1 file / Wrote 2 files in 0.06 seconds (v1.0.1-canary.2)

[nhoizey]

@pdehaan thanks for the confirmation! 👍

[claudioc]

I have the same issue, in the folder containing images (and only that). No glob whatsoever. The origin commit for this feature was c3c9d7b

  eleventyConfig.addPassthroughCopy(`${input}/fonts`);
  eleventyConfig.addPassthroughCopy(`${input}/css`);
  eleventyConfig.addPassthroughCopy(`${input}/img`);
  eleventyConfig.addPassthroughCopy(`${input}/.htaccess`);

The error is the > Having trouble copying './src/img'

[claudioc]

I fixed my problem changing the return object from my .eleventy.js from templateFormats: ['md', 'njk', 'png', 'jpg'], to templateFormats: ['md', 'njk'], and leaving "/img" as part of the copy

[nhoizey]

I now have the issue on multiple projects I want to update.

Is this a bug/regression, or should I rewrite my addPassthroughCopy() calls?

[nhoizey]

On the positive side of this "issue", it means I currently copy the same files multiple times, which is not good for build time.

I guess Eleventy could warn about it, but not break like currently.

[pdehaan]

I'm definitely on "team bug/regression" since this previously worked and seems like a common workflow (where you might have overlapping globs).

/cc @zachleat: Not sure if we want to include a fix for this in v1.0.1 (currently canary.3).

npm run build

> 11ty-2174@1.0.0 build
> DEBUG=Eleventy:TemplatePassthroughManager eleventy

@11ty/eleventy v1.0.1-canary.3
  Eleventy:TemplatePassthroughManager Resetting counts to 0 +0ms
  Eleventy:TemplatePassthroughManager TemplatePassthrough copy started. +8ms
  Eleventy:TemplatePassthroughManager `addPassthroughCopy` config API paths: { 'src/.well-known': true, 'src/**/*.{jpg,svg,png}': true } +0ms
  Eleventy:TemplatePassthroughManager `addPassthroughCopy` config API normalized paths: [ { inputPath: './src/.well-known', outputPath: true }, { inputPath: './src/**/*.{jpg,svg,png}', outputPath: true } ] +1ms
  Eleventy:TemplatePassthroughManager TemplatePassthrough copying from config: { inputPath: './src/.well-known', outputPath: true } +0ms
  Eleventy:TemplatePassthroughManager TemplatePassthrough copying from config: { inputPath: './src/**/*.{jpg,svg,png}', outputPath: true } +0ms
  Eleventy:TemplatePassthroughManager Copied './src/.well-known' (1 files) +21ms
[11ty] Problem writing Eleventy templates:
[11ty] 1. Having trouble copying (via EleventyPassthroughCopyError)
[11ty] 2. Having trouble copying './src/**/*.{jpg,svg,png}' (via TemplatePassthroughManagerCopyError)
[11ty] 3. Multiple passthrough copy files are trying to write to the same output file (www/.well-known/favicon.png). src/.well-known/favicon.png and src/.well-known/favicon.png (via TemplatePassthroughManagerCopyError)
[11ty] Copied 1 file / Wrote 2 files in 0.03 seconds (v1.0.1-canary.3)


echo $? # 1

But interestingly, despite the error-ish looking message and non-zero exit code, it seemed to copy the conflicting files. 🤷

tree -a www/
www/
├── .well-known/
│   └── favicon.png
├── assets/
│   └── foo.png
├── index.html
└── posts/
    ├── index.html
    └── logo.svg

3 directories, 5 files

---

Unsatisfying workaround: Move the .well-known/ folder outside of the src/ directory to "fix" the overlapping globs. That seemed to work for my minimal use case, but obviously tedious for real world projects where that might not be feasible.

[nhoizey]

@pdehaan I did move some files and change some addPassthroughCopy() calls for a small personal project: nhoizey/nicolas-hoizey.photo@8e5a53e

But it will be more difficult on larger projects.

[zachleat]

Hmm, appreciate y’all filing this—I’m seeing some additional nuance here now.

Because we have more than one way to add passthrough copy entries to the build, it seems like it’s somewhat common to have a single input -> output map entry listed more than once in a build.

The nuance here is that the error message is currently based on output paths only. We need to de-dupe based on both the input and output first, for two reasons:

1. so that the same input/output pair doesn’t copy more than once (wasted resources)
2. so that the same input/output pair doesn’t throw an error (unnecessary error)

I’d like to experiment with keeping the error message when two different input paths attempt to copy to the same output path though—I think that’s warranted!

[nhoizey]

@zachleat I fully agree. 👍

Same input/output pair could generate a warning, at least in debug mode.

[zachleat]

Filed the de-duping work as a separate issue because it proved to be a little bit more complicated than I want to do right now #2452

But we’ll move forward with a warning for the issue in #2452 and an error when differing inputs copy to the same output.

[zachleat]

If there is a case to be made to put this into 1.0 please let me know!

[nhoizey]

This was a breaking change from 0.12.1 to 1.0.0, which is allowed when changing the major number, so I guess it's up to you.

I made changes to my projects to be able to move to 1.0.1 already.