Concurrent map read and write error in template handling

[bep]

See https://gist.github.com/christophermancini/81401bbf2fac5847d9a256a9ceb2a20c

[bep]

@christophermancini I created an iissue over at the Go stdlib repo (and they seem to take it seriously as they have put the go1.7 milestone on it, which is in beta by now).

But is your failing repository available on GitHub (or whatever) for other to test?

[christophermancini]

@bep I have the code in Bitbucket, unfortunately, I cannot legally make it open to the public. If there are a few people who want access to attempt to reproduce, I can do that. Let me know how I can help.

[moorereason]

@christophermancini, can you make a simple, proof-of-concept site that triggers the bug?

[christophermancini]

I am not sure I can do that because it doesn't do it 100% of the time, more so like 9 out of 10 builds of the site will fail. I think part of it has to do with the magnitude of content and the complexity of my sites data files, for instance it takes over 40 seconds to build the site.

I can give you access to the site so that you can pull it down and attempt to build it, just let me know what your bitbucket user is.

[bep]

My bitbucket account is bjornerik

[christophermancini]

So oddly enough, when I first encountered the issue, I was about 40% through upgrading the site to use the template block feature. Now that I have completed it, I cannot reproduce it. I have also improved many partial templates to make sure they are Node / Page safe. Of course I have changed so many files, I am not sure what the state of the code was when it was broken, but when it was, it was easy to reproduce.

[marcojakob]

I got this error building my site on netlify.com. Then I manually triggered a new build and it worked. I also get this error when running hugo server on my Windows machine (64bit), after working for a while.

Is there some kind of race condition that only sometimes leads to problems?

Here is my error on Netlify, if that helps: https://gist.github.com/marcojakob/fd575644598a6007da61da60a986d7b5

[bep]

Is there some kind of race condition that only sometimes leads to problems?

Yes, and I suspect the issue lies inside Go stdlib:

golang/go#16101

It would be cool if you could ask the Netlify people to upgrade to Go 1.7.

[marcojakob]

@bep I've asked the Netlify people to upgrade to Hugo 0.17 and Go 1.7.

[moorereason]

@marcojakob,
What we really need is a sample site that triggers the panic.

[marcojakob]

@moorereason I've given you access to our repository. But note that the problem only occurs sometimes.

[moorereason]

@marcojakob, thank you.

I'm able to reproduce it with Hugo master and Go 1.7.1. I just run the following on Marco's site until it triggers.

#bash
i=0; while [ "$?" -eq "0" ]; do echo $i; ((i=i+1)); hugo --renderToMemory; done

@bep is much more familiar with the template rendering code that I am. If you wouldn't mind giving him access to your project, it would help.

Marco is using a _default/baseof.html template with several block definitions, which are overloaded extensively in the other templates. He has things like title, meta descriptions, main, header include, and body include blocks. I suspect that the problem lies in the block/define interaction. If I delete the blocks from baseof.html, the panic goes away.

My hunch is that it's a bug in the stdlib. All of the stack traces I've seen so far are hitting in the stdlib at text/template/exec.go#L369. It's accessing this map, which is apparently not protected from race conditions.

[marcojakob]

@moorereason that's great news that you're able to reproduce it.

@bep: I've given you access as well.

BTW: I don't mind you extracting parts of the site to make a smaller, reproducable example. I just didn't want to make the whole project's source with all its revisions (and possibly future blog posts) open for every one.

[bep]

Close this as duplicate for #2549 -- that has got a reproduceable site.

[bep]

Note, #2549 should now be fixed.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.