New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix regression with "too many open files" on macOS #3582

Closed
mitchchn opened this Issue Jun 12, 2017 · 11 comments

Comments

Projects
None yet
4 participants
@mitchchn
Contributor

mitchchn commented Jun 12, 2017

rlimit tweaking was removed in #3512. But without it, the rlimit is only 256, and I can't use hugo serve to preview even a relatively modest site without seeing this error:

Error: listen tcp 127.0.0.1:1313: socket: too many open files

The ulimit command still works fine for me in Sierra (10.12.6 beta), so I don't understand why it was removed from Hugo:

ulimit -n 10000; hugo serve

Any clarification on this, @bep?

@bep bep added the Docs label Jun 12, 2017

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Jun 12, 2017

Member

Not sure why you run Sierra beta, but it started to fail for me at some point, and Googling it pointed me to some posts saying that "now you have to set the ulimit in some plist file" etc. So we're not maintaining a hack/magic that only work for some -- when the real solution is to set a more sensible default value yourself.

I will keep this open as a documentation issue, but I'm not reverting any ulimit hack.

Member

bep commented Jun 12, 2017

Not sure why you run Sierra beta, but it started to fail for me at some point, and Googling it pointed me to some posts saying that "now you have to set the ulimit in some plist file" etc. So we're not maintaining a hack/magic that only work for some -- when the real solution is to set a more sensible default value yourself.

I will keep this open as a documentation issue, but I'm not reverting any ulimit hack.

@bep bep changed the title from Restore rlimit magic (or find another solution for 'too many open files') to docs: Add configuration tip about too many open files Jun 12, 2017

@mitchchn

This comment has been minimized.

Show comment
Hide comment
@mitchchn

mitchchn Jun 12, 2017

Contributor
  1. Have you actually tested the plist approach? It doesn't appear to work for me, but I'll keep testing.

  2. Once the correct solution is known, would you consider allowing a script to be included in the repo, outside of the hugo binary, so that users don't have to set up a launch daemon by themselves?

Contributor

mitchchn commented Jun 12, 2017

  1. Have you actually tested the plist approach? It doesn't appear to work for me, but I'll keep testing.

  2. Once the correct solution is known, would you consider allowing a script to be included in the repo, outside of the hugo binary, so that users don't have to set up a launch daemon by themselves?

@budparr

This comment has been minimized.

Show comment
Hide comment
@budparr

budparr commented Jun 12, 2017

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Jun 12, 2017

Member

Have you actually tested the plist approach?

Yes and it worked for me. But my main point is that I cannot have a hack in Hugo that fails, i.e. creates ERROR in the log for me; that makes Hugo near hopeless to work with and if it fails on my macOS (I don't run Sierra beta), then others will soon start to complain as well.

Member

bep commented Jun 12, 2017

Have you actually tested the plist approach?

Yes and it worked for me. But my main point is that I cannot have a hack in Hugo that fails, i.e. creates ERROR in the log for me; that makes Hugo near hopeless to work with and if it fails on my macOS (I don't run Sierra beta), then others will soon start to complain as well.

@bep

This comment has been minimized.

Show comment
Hide comment
@bep
Member

bep commented Jun 12, 2017

@mitchchn

This comment has been minimized.

Show comment
Hide comment
@mitchchn

mitchchn Jun 12, 2017

Contributor

OK, it worked after following the steps in that post. I'd still like to have a shell script which sets this up, even if that script is only linked to from the documentation and not included in Hugo. It's a lot to ask writers (not developers!) to set up a launch daemon manually just so they can use the preview server.

Contributor

mitchchn commented Jun 12, 2017

OK, it worked after following the steps in that post. I'd still like to have a shell script which sets this up, even if that script is only linked to from the documentation and not included in Hugo. It's a lot to ask writers (not developers!) to set up a launch daemon manually just so they can use the preview server.

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Jun 12, 2017

Member

Just test-reverted the commit in question:

ERROR 2017/06/12 20:00:00 Unable to increase number of open files limit operation not permitted

I could start my hugo server with sudo, but that doesn't have a nice ring to it. I'm on macOS 10.12.5. I'm open to suggestions for working workarounds.

Member

bep commented Jun 12, 2017

Just test-reverted the commit in question:

ERROR 2017/06/12 20:00:00 Unable to increase number of open files limit operation not permitted

I could start my hugo server with sudo, but that doesn't have a nice ring to it. I'm on macOS 10.12.5. I'm open to suggestions for working workarounds.

@bep bep changed the title from docs: Add configuration tip about too many open files to Handle "too many open files" on macOS Jun 12, 2017

@bep bep removed the Docs label Jun 12, 2017

@bep bep added this to the v0.23 milestone Jun 12, 2017

@mitchchn

This comment has been minimized.

Show comment
Hide comment
@mitchchn

mitchchn Jun 12, 2017

Contributor

I also reverted the commit and got the same error, but then I changed the rlimit to a smaller value (still much larger than the default of 256):

		rLimit.Max = 10000
		rLimit.Cur = 10000

Rebuilt and had no issues. Can you confirm that, @bep?

Contributor

mitchchn commented Jun 12, 2017

I also reverted the commit and got the same error, but then I changed the rlimit to a smaller value (still much larger than the default of 256):

		rLimit.Max = 10000
		rLimit.Cur = 10000

Rebuilt and had no issues. Can you confirm that, @bep?

bep added a commit that referenced this issue Jun 12, 2017

@bep bep closed this in 629e143 Jun 12, 2017

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Jun 12, 2017

Member

Rebuilt and had no issues. Can you confirm that, @bep?

Yes. I have at least one other issue that I should get out there and will do a patch release in a day or so, hopefully before people start shouting. Thanks for the heads up.

Member

bep commented Jun 12, 2017

Rebuilt and had no issues. Can you confirm that, @bep?

Yes. I have at least one other issue that I should get out there and will do a patch release in a day or so, hopefully before people start shouting. Thanks for the heads up.

@mitchchn

This comment has been minimized.

Show comment
Hide comment
@mitchchn

mitchchn Jun 12, 2017

Contributor

Thanks. For what it's worth, the max allowable value seems to be 64000.

Contributor

mitchchn commented Jun 12, 2017

Thanks. For what it's worth, the max allowable value seems to be 64000.

bep added a commit that referenced this issue Jun 12, 2017

@bep bep changed the title from Handle "too many open files" on macOS to Fix regression with "too many open files" on macOS Jun 12, 2017

@bep bep modified the milestones: v0.22.1, v0.23 Jun 12, 2017

@ghost ghost referenced this issue Jun 13, 2017

Open

spf13/hugo v0.22.1 released #12

@binarymist

This comment has been minimized.

Show comment
Hide comment
@binarymist

binarymist Jun 23, 2018

ulimit -n
65536

hugo server

                   | EN   
+------------------+-----+
  Pages            | 513  
  Paginator pages  | 139  
  Non-page files   |   1  
  Static files     | 139  
  Processed images |   0  
  Aliases          | 331  
  Sitemaps         |   1  
  Cleaned          |   0  

On Linux Mint, and hugo server still won't run. Anyone know why hugo server won't run, or have any other tips?
hugo version
v0.37.1 linux/amd64 BuildDate: 2018-03-07T17:56:25Z

lsof reports about half a million open files, but only a handfull are within my static site root directory.

I get the feeling hugo is reporting on open files outside of the static website root directory?

binarymist commented Jun 23, 2018

ulimit -n
65536

hugo server

                   | EN   
+------------------+-----+
  Pages            | 513  
  Paginator pages  | 139  
  Non-page files   |   1  
  Static files     | 139  
  Processed images |   0  
  Aliases          | 331  
  Sitemaps         |   1  
  Cleaned          |   0  

On Linux Mint, and hugo server still won't run. Anyone know why hugo server won't run, or have any other tips?
hugo version
v0.37.1 linux/amd64 BuildDate: 2018-03-07T17:56:25Z

lsof reports about half a million open files, but only a handfull are within my static site root directory.

I get the feeling hugo is reporting on open files outside of the static website root directory?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment