New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error copying static files while building site with --cleanDestinationDir #4246

Closed
biodranik opened this Issue Jan 10, 2018 · 9 comments

Comments

Projects
None yet
2 participants
@biodranik
Contributor

biodranik commented Jan 10, 2018

Error: Error copying static files: remove /Users/alex/Developer/landing-hugo/docs/ru/team: directory not empty

The issue reproduces randomly (with enough calls of hugo --cleanDestinationDir it reproduces 100%, usually after 5-10 calls). Multihost setup, 2 languages site with two sections and total 43 pages. There are 15 md pages with translations (30 total) in the failed team directory.

ls /Users/alex/Developer/landing-hugo/docs/ru/team shows that there are index.html and subfolders with other index.html files inside.

Generation stops after this error and docs folder is left in the inconsistent state.

Code investigation shows that the issue is caused by calling copyStatic and buildSites in parallel (in fullBuild function). The issue does not reproduce if they are run in sequential order. As I understand it, dst and src dirs are compared and files are deleted at the end of copyStatic, but buildSites can still run at this moment.

@biodranik

This comment has been minimized.

Show comment
Hide comment
@biodranik

biodranik Jan 10, 2018

Contributor

Probably, the best solution would be to move cleanDestinationDir call out of the copyStatic and run it at the end, when copyStatic and buildSites will finish their parallel execution.

Contributor

biodranik commented Jan 10, 2018

Probably, the best solution would be to move cleanDestinationDir call out of the copyStatic and run it at the end, when copyStatic and buildSites will finish their parallel execution.

@bep bep added the Bug label Jan 10, 2018

@bep bep added this to the 0.32.4 milestone Jan 10, 2018

@bep bep closed this in 768ec5d Jan 10, 2018

@biodranik

This comment has been minimized.

Show comment
Hide comment
@biodranik

biodranik Jan 10, 2018

Contributor

cleanDestinationDir was the only working option for localhost development (hugo server), which correctly rebuilt all site pages. Content is not correctly updated in several cases without this option (not all required pages are rebuilt), and the only way to see the changes is to stop and run again hugo server.

It was also a very convenient option to avoid rm -rf docs call before building site on disk.

Contributor

biodranik commented Jan 10, 2018

cleanDestinationDir was the only working option for localhost development (hugo server), which correctly rebuilt all site pages. Content is not correctly updated in several cases without this option (not all required pages are rebuilt), and the only way to see the changes is to stop and run again hugo server.

It was also a very convenient option to avoid rm -rf docs call before building site on disk.

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Jan 10, 2018

Member

cleanDestinationDir was the only working option for localhost development (hugo server), which correctly rebuilt all site pages

I have never used that option. We may fix this properly by re-implementing this, but I don't have time for this now, esp. since I don't use it myself; so I took the quick path. If there is a bug in the hugo server code, let us fix that. If someone can come up with a clean solution, I am happy to merge it ... for Hugo 0.33.

Member

bep commented Jan 10, 2018

cleanDestinationDir was the only working option for localhost development (hugo server), which correctly rebuilt all site pages

I have never used that option. We may fix this properly by re-implementing this, but I don't have time for this now, esp. since I don't use it myself; so I took the quick path. If there is a bug in the hugo server code, let us fix that. If someone can come up with a clean solution, I am happy to merge it ... for Hugo 0.33.

@biodranik

This comment has been minimized.

Show comment
Hide comment
@biodranik

biodranik Jan 10, 2018

Contributor
Contributor

biodranik commented Jan 10, 2018

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Jan 10, 2018

Member

@biodranik see #4248 if someone can create a PR for this (i.e. revert my relevant commit and create a working version), I will consider it for 0.32.4. Else it will have to wait.

But note that moving it after the build will not work (or it does not make sense in my head).

Member

bep commented Jan 10, 2018

@biodranik see #4248 if someone can create a PR for this (i.e. revert my relevant commit and create a working version), I will consider it for 0.32.4. Else it will have to wait.

But note that moving it after the build will not work (or it does not make sense in my head).

@biodranik

This comment has been minimized.

Show comment
Hide comment
@biodranik

biodranik Jan 10, 2018

Contributor

Why it doesn’t make sense and why it will not work?
As I understood this flag, hugo does not sync src and dest without it. E.g. if I remove or rename a file in src, it’s generated or copied version will stay at dst. Only this flag will actually sync and delete these files.

Build and copy files stages are already done in parallel. The bug was caused by this sync called before the build has finished. So moving it after parallel step is an obvious and good solution.

Contributor

biodranik commented Jan 10, 2018

Why it doesn’t make sense and why it will not work?
As I understood this flag, hugo does not sync src and dest without it. E.g. if I remove or rename a file in src, it’s generated or copied version will stay at dst. Only this flag will actually sync and delete these files.

Build and copy files stages are already done in parallel. The bug was caused by this sync called before the build has finished. So moving it after parallel step is an obvious and good solution.

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Jan 10, 2018

Member

As I understood this flag, hugo does not sync src and dest without it.

That is only partly true. What you may end up with without this flag are some stale files.

But if you

  • Sync from static to /public and build to /public in parallel
  • Then remove the files in /public not present in /static

Then, in my head, you will end up lot worse than we are now.

Please enlighten me if I'm wrong.

So, the simple solution is to revert my commit + no concurrency in those two steps when that flag is set.

Member

bep commented Jan 10, 2018

As I understood this flag, hugo does not sync src and dest without it.

That is only partly true. What you may end up with without this flag are some stale files.

But if you

  • Sync from static to /public and build to /public in parallel
  • Then remove the files in /public not present in /static

Then, in my head, you will end up lot worse than we are now.

Please enlighten me if I'm wrong.

So, the simple solution is to revert my commit + no concurrency in those two steps when that flag is set.

@biodranik

This comment has been minimized.

Show comment
Hide comment
@biodranik

biodranik Jan 10, 2018

Contributor

I see it now, thanks for the explanation. I have mistakenly assumed that src is not a static folder, but some virtual file system of all source files (including the generated one).

In this case, the only good way to fix it is to avoid running static copy and build tasks in parallel if this flag was specified.

Contributor

biodranik commented Jan 10, 2018

I see it now, thanks for the explanation. I have mistakenly assumed that src is not a static folder, but some virtual file system of all source files (including the generated one).

In this case, the only good way to fix it is to avoid running static copy and build tasks in parallel if this flag was specified.

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Jan 10, 2018

Member

I have mistakenly assumed that src is not a static folder, but some virtual file system of all source files (including the generated one).

It is virtual file system; it's a union FS composed of potential many. But static + build all end up in the same destination (/public by default).

Member

bep commented Jan 10, 2018

I have mistakenly assumed that src is not a static folder, but some virtual file system of all source files (including the generated one).

It is virtual file system; it's a union FS composed of potential many. But static + build all end up in the same destination (/public by default).

bep added a commit that referenced this issue Jan 10, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment