Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Concurrent map read and write error in template handling #2224

Closed
bep opened this issue Jun 17, 2016 · 15 comments
Closed

Concurrent map read and write error in template handling #2224

bep opened this issue Jun 17, 2016 · 15 comments
Labels
Bug

Comments

@bep bep added the Bug label Jun 17, 2016
@bep bep changed the title Concurrent map read and write in taxonomy handling Concurrent map read and write Jun 17, 2016
@bep bep changed the title Concurrent map read and write Concurrent map read and write error in template handling Jun 17, 2016
@bep

This comment has been minimized.

Copy link
Member Author

@bep bep commented Jun 18, 2016

@christophermancini I created an iissue over at the Go stdlib repo (and they seem to take it seriously as they have put the go1.7 milestone on it, which is in beta by now).

But is your failing repository available on GitHub (or whatever) for other to test?

@christophermancini

This comment has been minimized.

Copy link
Contributor

@christophermancini christophermancini commented Jun 18, 2016

@bep I have the code in Bitbucket, unfortunately, I cannot legally make it open to the public. If there are a few people who want access to attempt to reproduce, I can do that. Let me know how I can help.

@moorereason

This comment has been minimized.

Copy link
Contributor

@moorereason moorereason commented Jun 18, 2016

@christophermancini, can you make a simple, proof-of-concept site that triggers the bug?

@christophermancini

This comment has been minimized.

Copy link
Contributor

@christophermancini christophermancini commented Jun 18, 2016

I am not sure I can do that because it doesn't do it 100% of the time, more so like 9 out of 10 builds of the site will fail. I think part of it has to do with the magnitude of content and the complexity of my sites data files, for instance it takes over 40 seconds to build the site.

I can give you access to the site so that you can pull it down and attempt to build it, just let me know what your bitbucket user is.

@bep

This comment has been minimized.

Copy link
Member Author

@bep bep commented Jun 18, 2016

My bitbucket account is bjornerik

@christophermancini

This comment has been minimized.

Copy link
Contributor

@christophermancini christophermancini commented Jun 18, 2016

So oddly enough, when I first encountered the issue, I was about 40% through upgrading the site to use the template block feature. Now that I have completed it, I cannot reproduce it. I have also improved many partial templates to make sure they are Node / Page safe. Of course I have changed so many files, I am not sure what the state of the code was when it was broken, but when it was, it was easy to reproduce.

@marcojakob

This comment has been minimized.

Copy link

@marcojakob marcojakob commented Oct 9, 2016

I got this error building my site on netlify.com. Then I manually triggered a new build and it worked. I also get this error when running hugo server on my Windows machine (64bit), after working for a while.

Is there some kind of race condition that only sometimes leads to problems?

Here is my error on Netlify, if that helps: https://gist.github.com/marcojakob/fd575644598a6007da61da60a986d7b5

@bep

This comment has been minimized.

Copy link
Member Author

@bep bep commented Oct 9, 2016

Is there some kind of race condition that only sometimes leads to problems?

Yes, and I suspect the issue lies inside Go stdlib:

golang/go#16101

It would be cool if you could ask the Netlify people to upgrade to Go 1.7.

@marcojakob

This comment has been minimized.

Copy link

@marcojakob marcojakob commented Oct 9, 2016

@bep I've asked the Netlify people to upgrade to Hugo 0.17 and Go 1.7.

@moorereason

This comment has been minimized.

Copy link
Contributor

@moorereason moorereason commented Oct 10, 2016

@marcojakob,
What we really need is a sample site that triggers the panic.

@marcojakob

This comment has been minimized.

Copy link

@marcojakob marcojakob commented Oct 10, 2016

@moorereason I've given you access to our repository. But note that the problem only occurs sometimes.

@moorereason

This comment has been minimized.

Copy link
Contributor

@moorereason moorereason commented Oct 11, 2016

@marcojakob, thank you.

I'm able to reproduce it with Hugo master and Go 1.7.1. I just run the following on Marco's site until it triggers.

#bash
i=0; while [ "$?" -eq "0" ]; do echo $i; ((i=i+1)); hugo --renderToMemory; done

@bep is much more familiar with the template rendering code that I am. If you wouldn't mind giving him access to your project, it would help.

Marco is using a _default/baseof.html template with several block definitions, which are overloaded extensively in the other templates. He has things like title, meta descriptions, main, header include, and body include blocks. I suspect that the problem lies in the block/define interaction. If I delete the blocks from baseof.html, the panic goes away.

My hunch is that it's a bug in the stdlib. All of the stack traces I've seen so far are hitting in the stdlib at text/template/exec.go#L369. It's accessing this map, which is apparently not protected from race conditions.

@marcojakob

This comment has been minimized.

Copy link

@marcojakob marcojakob commented Oct 11, 2016

@moorereason that's great news that you're able to reproduce it.

@bep: I've given you access as well.

BTW: I don't mind you extracting parts of the site to make a smaller, reproducable example. I just didn't want to make the whole project's source with all its revisions (and possibly future blog posts) open for every one.

@bep

This comment has been minimized.

Copy link
Member Author

@bep bep commented Oct 11, 2016

Close this as duplicate for #2549 -- that has got a reproduceable site.

@bep bep closed this Oct 11, 2016
@bep

This comment has been minimized.

Copy link
Member Author

@bep bep commented Oct 14, 2016

Note, #2549 should now be fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.