watch with lots of files dies #168
Comments
|
Based on the FAQ comment here it seems nothing can be done: https://github.com/howeyc/fsnotify Perhaps the best solution it to add a "watch mask"? I have lots of static files that will hardly ever change. |
|
I think a reasonable workaround is to set On Mac OSX in particular, Then it doesn't crash for me. However, I don't think it should panic, either. Panicking is really bad behavior for a program such as this. It should be printing an error and continuing to try to run, or if that's unrecoverable, exiting with a nonzero code, but not panicking. |
|
I disagree that nothing can be done. I haven't tried it, but I would be surprised if you couldn't make a |
|
Yeah, with a cursory search I think this is the right place to do it: http://golang.org/pkg/syscall/#Setrlimit |
|
Thanks for digging into this. Care to make a PR? Steve Francia
|
|
Thanks!
|
|
@spf13 @xaprb I'm still seeing this issue... By the way, my theme ( One way to avoid this would be if |
|
@chibicode Are you REALLY synching node_modules? Why? You don't need an ignore feature to fix your problem. Just use Gulp to build just what you need into /static. |
|
@bep the problem is that my
And while I develop my theme, I use HugoBasicExample repo to test it, like this:
And when I try to run HugoBasicExample above with Anyway, I'm pretty sure there's a workaround, but I can't find anything simple that meets the following requirements:
I'm not saying I'd like this feature implemented. I just wish it existed. |
|
Hi @chibicode, Just double checking:
|
|
Hi @chibicode, more questions for you:
Many thanks! P.S. Strangely, I cannot reproduce this problem on my Debian GNU/Linux at all, not even after I did this: P.P.S. Edited this post to change |
|
Looking through git log:
The code in |
|
Hi @chibicode, The following appear to be one of the most comprehensive info on max open files limit on Mac OS X. Please take a look: In particular, OS X Yosemite (10.10), which came out in October 2014, eight months after the aforementioned fixes, has made changes to how this limit is set. You may also try this for a quick fix: Please let us know how it works! Thanks! |
A local `node_modules` directory can easily contain tens of thousands of files, easily exhausting the tiny default max open files limit especially on OS X Yosemite, in spite of the fact that Hugo already had code in place since February 2014 to try to raise the maxfiles ulimit. Also skip `.git` and `bower_components` directories. The file watching situation will improve when https://github.com/go-fsnotify/fsevents become ready, but until then, we will be thrifty. :-) Thanks to @chibicode for the suggestion. See #168 for continued discussions.
|
Hi @chibicode, I am being adventurous and decided to take up your suggestion, and committed b9b70fb to skip It turns out many of the mysteries can be explained by reading about
Also, according to https://github.com/go-fsnotify/fsnotify/, fsnotify uses Last but not least... should we keep the discussion here? Or should we start a new issue and continue from there? Afterall, this bug was already fixed in February/March 2014 last year. ;-) Cheers, |
|
Hi @anthonyfok, wow, thank you so much for such a thorough response. I did try Also thanks for b9b70fb - although this looks like a huge hack and I now feel bad for requesting this. Sorry I'm on a hurry, but really appreciated your response. |
Fixes gohugoio#922 add finalize command and tests to change draft status of specified content to false and set date to Now() undraft command Allow hyphens in shortcode name Fixes gohugoio#929 Keep trailing slash when baseUrl contains a sub path Before this commit, .Site.BaseUrl ended up as: http://mysite.com/ => http://mysite.com/ http://mysite.com/sub/ => http://mysite.com/sub Now it becomes: http://mysite.com/ => http://mysite.com/ http://mysite.com/sub/ => http://mysite.com/sub/ Fixed gohugoio#931 Improve error message on missing shortcode inner content Fixes gohugoio#933 Allow the same shortcode to be used with or without inline content Fixes gohugoio#934 Update Readme.md with additional contribution guides Added Gitter badge Restructure top of Readme.md Fixing image in readme Using a smaller Header Image Tidying the Readme a bit more Update github-pages-blog.md change `hugo serve` to `hugo server` Add benchmark for the shortcode lexer Apply gofmt -s Avoid panic when pagination on 0 pages Fixes gohugoio#948 Prevent 404.html from prettifying into 404/index.html Restore @realchaseadams's commit 348e123 "Force `UglyUrls` option to force `404.html` file name" which got lost after some refactoring (commit 8db3c0b). Remove the equivalent "force `UglyUrls`" code for `sitemap.xml` because the refactored code now calls `renderAndWriteXML()` which uses `WriteDestFile()` which does not prettify a filename. Fixes gohugoio#939 Fix errors reported by Go Vet Fix some Go Lint errors Apply some more Golint suggestions helpers: apply some Golint rules author: fix doc hugolib: apply some Hugolint rules page: make receiver name on Page methods consistent Apply some Golint rules on Page, esp. making the method receiver names consistent: (page *Page) ==> (p *Page) menu: make receiver name on Page methods consistent Apply some Golint rules on Menu, esp. making the method receiver names consistent. hugolib: apply some more Golint rules source: apply some Golint rules livereload: apply some Golint rules parser: apply some Golint rules There is only one s.PageTarget() - so we cannot change it, even tempoararily. We have to find another solution to this. ... Prevent 404.html from prettifying into 404/index.html Restore @realchaseadams's commit 348e123 "Force `UglyUrls` option to force `404.html` file name" which got lost after some refactoring (commit 8db3c0b). Remove the equivalent "force `UglyUrls`" code for `sitemap.xml` because the refactored code now calls `renderAndWriteXML()` which uses `WriteDestFile()` which does not prettify a filename. Fixes gohugoio#939 (reverted from commit c4c19ad) Handle 404 thread safely Replaces hack that temporarily changes a global flag. Fixes gohugoio#955 Fixes gohugoio#939 Fix UglyUrls on Windows Fixes gohugoio#958 Fix eq and ne tpl function issue `eq` and `ne` template functions don't work as expected when those are used with a raw number and a calculated value by add, sub etc. It's caused by both numbers type differences. For example, `eq 5 (add 2 3)` returns `false` because raw 5 is `int` while `add 2 3` returns 5 with `int64` This normalizes `int`, `uint` and `float` type values to `int64`, `uint64` and `float64` before comparing them. Other type of value is passed to comparing function without any changes. Fix gohugoio#961 Add test cases for Ne and Eq type normalisation See gohugoio#961 Add new min_version field to theme.toml template Switch from fsnotify.v0 to fsnotify.v1 API (watcher) Fixes gohugoio#357 See also gohugoio#761 Do not parse backup files with trailing '~' as templates Fixes gohugoio#964 absurlreplacer: write replacements directly to the byte buffer The extra step isn't needed and this makes the code simpler. And slightly faster: benchmark old ns/op new ns/op delta BenchmarkAbsUrl 19987 17498 -12.45% BenchmarkXmlAbsUrl 10606 9503 -10.40% benchmark old allocs new allocs delta BenchmarkAbsUrl 28 24 -14.29% BenchmarkXmlAbsUrl 14 12 -14.29% benchmark old bytes new bytes delta BenchmarkAbsUrl 3512 3297 -6.12% BenchmarkXmlAbsUrl 2059 1963 -4.66% parser: add some frontmatter test cases Skip directories like node_modules from the watchlist A local `node_modules` directory can easily contain tens of thousands of files, easily exhausting the tiny default max open files limit especially on OS X Yosemite, in spite of the fact that Hugo already had code in place since February 2014 to try to raise the maxfiles ulimit. Also skip `.git` and `bower_components` directories. The file watching situation will improve when https://github.com/go-fsnotify/fsevents become ready, but until then, we will be thrifty. :-) Thanks to @chibicode for the suggestion. See gohugoio#168 for continued discussions. Add some basic tests for doArithmetic We might have to take precision into account for floating point nubers ... at some point. doArithmetic: add test for division by zero Correct initialisms as suggested by golint First step to use initialisms that golint suggests, for example: Line 116: func GetHtmlRenderer should be GetHTMLRenderer as see on http://goreportcard.com/report/spf13/hugo Thanks to @bep for the idea! Note that command-line flags (cobra and pflag) as well as struct fields like .BaseUrl and .Url that are used in Go HTML templates need more work to maintain backward-compatibility, and thus are NOT yet dealt with in this commit. First step in fixing gohugoio#959. Remove trailing space from site build statistics Update press coverage: Fix URL; new tutorial in Chinese [Docs] Update and expand http://gohugo.io/overview/usage/ The `hugo help` output as shown in http://gohugo.io/overview/usage/ was not yet updated for v0.13. Thanks to @alebaffa for the heads up! Also: - Clarify that, after using `hugo server`, the bare `hugo` command need to be run before deployment. - Add a section on running `hugo` as production web server, and add links to two blog posts of two Hugo users sharing their experience. Partially fixes: gohugoio#852 and gohugoio#937 Add deprecated logger Add double checking in Deprecated To prevent possible duplicate log statements. source: add some test cases for File Do not ERROR-log missing /data dir Fixes gohugoio#930 Experimental AsciiDoc support with external helpers See gohugoio#470 * Based on existing support for reStructuredText files * Handles content files with extensions `.asciidoc` and `.ad` * Pipes content through `asciidoctor --safe -`. If `asciidoctor` is not installed, then `asciidoc --safe -`. * To make sure `asciidoctor` or `asciidoc` is found, after adding a piece of AsciiDoc content, run `hugo` with the `-v` flag and look for this message: INFO: 2015/01/23 Rendering with /usr/bin/asciidoctor ... Caveats: * The final "Last updated" timestamp is currently not stripped. * When `hugo` is run with `-v`, you may see a lot of these messages INFO: 2015/01/23 Rendering with /usr/bin/asciidoctor ... if you have lots of `*.ad`, `*.adoc` or `*.asciidoc` files. * Some versions of `asciidoc` may have trouble with its safe mode. To test if you are affected, try this: $ echo "Hello" | asciidoc --safe - asciidoc: ERROR: unsafe: ifeval invalid asciidoc: FAILED: ifeval invalid safe document If so, I recommend that you install `asciidoctor` instead. Feedback and patches welcome! Ideally, we should be using https://github.com/VonC/asciidocgo, @VonC's wonderful Go implementation of Asciidoctor. However, there is still a bit of work needed for asciidocgo to expose its API so that Hugo can actually use it. Until then, hope this "experimental AsciiDoc support through external helpers" can serve as a stopgap solution for our community. :-) 2015-01-30: Updated for the replaceShortcodeTokens() syntax change 2015-02-21: Add `.adoc` extension as suggested by @Fale Conflicts: helpers/content.go Add Seq template func Very similar to GNU's seq. Fixes gohugoio#552 Conflicts: tpl/template.go Added image support to the sitemap.xml template Conflicts: tpl/template_embedded.go Revert "Added image support to the sitemap.xml template" This reverts commit 3c147bd. Fixes gohugoio#972 removed duplicate word in readme delete finalize files
|
Hugo Static Site Generator v0.15-DEV BuildDate: 2015-10-15T10:07:58-06:00 I'm seeing a number of errors like this on OS X: I'm not sure why it's watching the public directory. |
|
That's very strange. We haven't seen these kind of errors in a while. It shouldn't be watching public ever. |
|
@nathany are you sure this isn't something else running on your computer? Because this one is way above my pay grade to understand ... |
|
You're probably right, as I also have reflex running in the same Terminal to watch sass files for sassc. I'll have to double check where the error came from. UPDATE: Yah, that was Reflex. Sorry for the false alarm. |
|
Am also getting It may be due to Apple's System Integrity Protection utility 'csrutil' ignoring any attempts to increase ulimits. This is on OS X Yosemite 10.10.5. I don't want to disable csrutil, so I guess I'll have to have to wait for |
|
Could Hugo use this?
|
|
@anthonyfok I don't see why not. Though perhaps fsnotify will use fsevents, or a new replacement for fsnotify. |
That would be awesome! :-D |
|
Regarding my May 14 comment, I found this command ( |
|
Thanks @hypertexthero. It's probably worth adding that command to the fsnotify Wiki https://github.com/fsnotify/fsnotify/wiki or even a blog post on https://fsnotify.org/ (which uses Hugo, btw:-). |
A local `node_modules` directory can easily contain tens of thousands of files, easily exhausting the tiny default max open files limit especially on OS X Yosemite, in spite of the fact that Hugo already had code in place since February 2014 to try to raise the maxfiles ulimit. Also skip `.git` and `bower_components` directories. The file watching situation will improve when https://github.com/go-fsnotify/fsevents become ready, but until then, we will be thrifty. :-) Thanks to @chibicode for the suggestion. See gohugoio#168 for continued discussions.
|
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
The text was updated successfully, but these errors were encountered: