Outputs that lay outside of the workspace folder are not restored from cache when daemon is used

[rafaeltab]

Verify canary release

• I verified that the issue exists in the latest Turborepo canary release.

Link to code that reproduces this issue

https://github.com/rafaeltab/turborepo-outside-workspace-repro

What package manager are you using / does the bug impact?

npm

What operating system are you using?

Windows & WSL2

Which canary version will you have in your reproduction?

TBD

Describe the Bug

When caching files that lay outside the workspace directory they are not restored when using the daemon.

{
  "...": "rest of the turbo file",
  "pipelines": {
    "build": {
       "...": "normal content of build pipe here",
       "outputs": ["dist/**"]
    },
    "special-package#build": {
       "...": "normal content of build pipe here",
       "outputs": ["../../dist/**"]
    }
  }
}

On the first run of turbo run build using this config a cache file is created, which contains the correct ../../dist folder. However after deleting the folder, and running turbo run build again, the folder is not restored from the cache.

Turbo gives the message: files already on disk (exact message will be added soon)

Expected Behavior

The dist folder should be restored.

To Reproduce

• Clone the repro
• Run npm install
• Run npm run build
• Delete the dist folder in the root of the repo
• Delete the dist folder in the packages/example-0 workspace
• Run npm run build again (should be FULL TURBO)

The dist folder is not populated with the cached files.
The dist folder inside the packages/example-0 workspace is populated with the cached files

_TURBO-2251

[rafaeltab]

There seems to be some really funky business going on. I am able to partially reproduce the same issue in wsl, but the cache tar file looks different. I will investigate this further on the original windows pc where this first occurred on Tuesday with the same turbo version, and update the issue accordingly.

[rafaeltab]

The issue seems to be a little different than I first thought. It is related to the properly cached files not being restored when deleted from disk when using the daemon. I think it might also be OS specific, will do some further testing tonight.

[yizixu]

I also encountered this issue today. As long as the target directory is outside the workspace, deleting it and executing build again will not restore it. I hope the author can fix this issue as soon as possible. It's urgent. Thank you

[rafaeltab]

@yizixu from my testing, as long as you run with --no-daemon flag it works. I'd also recommend checking what you put in the turbo.json pipeline. We accidentally put the folder name instead of the workspace name (name inside package.json).

There is no warning if this name is incorrect.

[yizixu]

@rafaeltab I am using pnpm exec turbo run build in jenkins. I am sure I am using workspace name. I will try it tomorrow - no-daemon. If it works, I will come back to thank you, my benefactor.

[rafaeltab]

I doubt that the no-daemon solution will work in that case, as the problem is related to the daemon thinking the files don't have to be restored as they are already on disk. This is less likely to occur in a CI environment, it is more likely to be a configuration issue, other caching issue, or a separate problem with turbo. Nevertheless give it a try tomorrow, and if it is not the solution let me know here.

You can go to the Turbo Discord server or the GitHub Discussions for further assistance with your problem.

[yizixu]

Unfortunately, --no-daemon does not take effect. My code is as follows:

const filterParams = apps.reduce((pre, cur) => {
  pre.push(`--filter=${cur}`)
  return pre
}, [])

await execa('pnpm', ['exec', 'dotenv', '--', 'turbo', 'run', 'build', ...filterParams, '--concurrency=20', '--no-daemon'], {
  stdio: 'inherit'
})

But my approach is rather special, because I need to assemble the dist folders of all packages into a special directory, so the output.path of my app is ../../dist/oms/child/smart-app1 , instead of setting outputs in turbo.json, I don’t know what the difference will be between these two approaches.

[yizixu]

@rafaeltab Now I think this problem can be ignored. I changed the mv of jenkins to cp, which will not cause the directory to be lost, and there will be no problem of unrestored cache. Finally, thank you for your reply. ,Thank you so much

[gsoltis]

Reasonably certain I've traced what was going wrong here, at least with the daemon piece. Fixed in #7327