New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panic on zero-padded taxonomy weights #4628

Closed
clorenz opened this Issue Apr 15, 2018 · 13 comments

Comments

Projects
None yet
5 participants
@clorenz

clorenz commented Apr 15, 2018

$ hugo server -D --debug

[...]
DEBUG 2018/04/15 18:49:36 Load Data from 1 source(s)
INFO 2018/04/15 18:49:36 found taxonomies: map[string]string{"category":"categories", "manufacturerletter":"manufacturerletters", "manufacturer":"manufacturers", "featured":"featured", "tag":"tags"}
panic: interface conversion: interface {} is float64, not int

goroutine 50 [running]:

github.com/gohugoio/hugo/hugolib.(*Site).assembleTaxonomies(0xc4204ce2c0)
	/go/src/github.com/gohugoio/hugo/hugolib/site.go:1545 +0xee2
github.com/gohugoio/hugo/hugolib.(*Site).buildSiteMeta(0xc4204ce2c0, 0x0, 0x0)
	/go/src/github.com/gohugoio/hugo/hugolib/site.go:1367 +0x83
github.com/gohugoio/hugo/hugolib.(*HugoSites).assemble(0xc4205dab80, 0xc421824300, 0x0, 0x0)
	/go/src/github.com/gohugoio/hugo/hugolib/hugo_sites_build.go:177 +0x19a
github.com/gohugoio/hugo/hugolib.(*HugoSites).Build(0xc4205dab80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/gohugoio/hugo/hugolib/hugo_sites_build.go:60 +0x144
github.com/gohugoio/hugo/commands.(*commandeer).buildSites(0xc42013a0c0, 0x0, 0x0)
	/go/src/github.com/gohugoio/hugo/commands/hugo.go:827 +0x8b
github.com/gohugoio/hugo/commands.(*commandeer).fullBuild.func3(0x8, 0xed1560)
	/go/src/github.com/gohugoio/hugo/commands/hugo.go:423 +0x2e
github.com/gohugoio/hugo/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc4205da580, 0xc4206603d0)
	/go/src/github.com/gohugoio/hugo/vendor/golang.org/x/sync/errgroup/errgroup.go:58 +0x57
created by github.com/gohugoio/hugo/vendor/golang.org/x/sync/errgroup.(*Group).Go
	/go/src/github.com/gohugoio/hugo/vendor/golang.org/x/sync/errgroup/errgroup.go:55 +0x66

I have no clue, what might be the reason for this panic. My site currently has got 1545 .md files, maybe it has surpassed an internal limit? Any idea, how to get rid of it?

Thank you very much in advance
Christoph

@kaushalmodi

This comment has been minimized.

Member

kaushalmodi commented Apr 16, 2018

Can you post your site source?

@anthonyfok

This comment has been minimized.

Contributor

anthonyfok commented Apr 17, 2018

And please tell us the Hugo version you are using, and from where you obtain it, and on which OS you are running, etc. Many thanks!

@clorenz

This comment has been minimized.

clorenz commented Apr 17, 2018

I'm using the official version, downloaded from github,

Hugo Static Site Generator v0.38.2 linux/amd64 BuildDate: 2018-04-09T08:17:17Z

on an Ubuntu 16.04 LTS.

The content (~1250 pages) is attached, as well as my config.toml

content.tar.gz

config.toml.gz

Thank you very much in advance

@anthonyfok anthonyfok changed the title from Hugo dies with panic to Hugo 0.38.2 (and 0.39) dies with panic: interface conversion: interface {} is float64, not int Apr 17, 2018

@anthonyfok

This comment has been minimized.

Contributor

anthonyfok commented Apr 17, 2018

Thank you @clorenz.

I was able to reproduce the panic on my Debian sid (amd64) running hugo 0.39-1 compiled on Debian.
(will be available via apt install hugo later today or tomorrow). Not sure exactly why though. Will need to investigate.

@moorereason

This comment has been minimized.

Contributor

moorereason commented Apr 17, 2018

I'm sure the panic is the result of two unprotected typecasts here and here. I'm not sure what's causing it, though. I don't see where the float64 is coming from.

@kaushalmodi

This comment has been minimized.

Member

kaushalmodi commented Apr 17, 2018

Is this a problem:

km²~/temp/:content.1> rg ':\s[0-9]{7,}(\.[0-9]+)*$'
movements/r/rego/rego-2125-216.de.md
15:manufacturers_weight: 2125216

movements/r/rego/rego-2125-216.en.md
15:manufacturers_weight: 2125216

movements/f/fe/fe-4611.en.md
15:manufacturers_weight: 4611000

movements/f/fe/fe-4611.de.md
15:manufacturers_weight: 4611000

I don't know when Go assigns values as float64 instead of int.

I see some absurdly high taxonomy weights there.. but they are not as high as 2^31 = 2147483648. So not sure..

@kaushalmodi

This comment has been minimized.

Member

kaushalmodi commented Apr 17, 2018

@clorenz It would be wayy easier to reproduce your issue if you simply archived everything needed to build the site.. Only content and config is not enough, you need layouts or theme to build too.


Building sites … ERROR 2018/04/17 18:23:27 Unable to locate template for shortcode "movementlist" in page "movements/u/urofa/_index.en.md"
ERROR 2018/04/17 18:23:27 Unable to locate template for shortcode "movementlist" in page "movements/u/urofa/_index.de.md"
@kaushalmodi

This comment has been minimized.

Member

kaushalmodi commented Apr 17, 2018

OK, so my theory is correct.. there's a problem with one of the values of manufacturers_weight.. because if I delete all manufacturers_weight lines, the site builds fine:

km²~/temp/:hugo_temp/content> \find . -name "*.md" -print0 | xargs -0 \sed -i '/manufacturers_weight:.*/d'                      04/17 6:34pm
km²~/temp/:hugo_temp/content> ..                                                                                                04/17 6:36pm
km²~/temp/:hugo_temp> srvh -t bare-min -p 5656 
                   |  DE  |  EN
+------------------+------+------+
  Pages            | 1498 | 1490
  Paginator pages  |   63 |   62
  Non-page files   |    0 |    0
  Static files     |    2 |    2
  Processed images |    0 |    0
  Aliases          |    1 |    1
  Sitemaps         |    1 |    1
  Cleaned          |    0 |    0

Total in 12105 ms
Watching for changes in /home/kmodi/temp/hugo_temp/{content,layouts,themes}
Watching for config changes in /home/kmodi/temp/hugo_temp/config.toml
Serving pages from memory
Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
Web Server is available at http://localhost:5656/ (bind address 127.0.0.1)
Web Server is available at http://localhost:5657/ (bind address 127.0.0.1)
Press Ctrl+C to stop
@kaushalmodi

This comment has been minimized.

Member

kaushalmodi commented Apr 17, 2018

Not at my PC now.. but I just realized.. do any of those weight values have a dot or a comma or an exponent sign like e or E?

@kaushalmodi

This comment has been minimized.

Member

kaushalmodi commented Apr 18, 2018

Got bit by this again.. some genius in the Go lang team thought that it's a good idea to consider string representations of numbers starting with 0 as octals!

So the temporary user-facing fix is:

find . -name "*.md" -exec grep -P 'manufacturers_weight: 0[0-9]+' -l {} \; -exec \sed -r -i 's/(manufacturers_weight: )0([0-9]+)/\1\2/' {} \;
  • Removes preceding zeros from all values of manufacturers_weight.

You will end up with these 16 modified files.


Though it's not clear why strings considered as octal representations are seen as float64.

And the fix on Hugo side would be catch this exception and point to the file/line containing the offending uncastable int-string.

@moorereason moorereason added Bug and removed NeedsInvestigation labels Apr 18, 2018

@moorereason moorereason added this to the v0.39.1 milestone Apr 18, 2018

@moorereason moorereason changed the title from Hugo 0.38.2 (and 0.39) dies with panic: interface conversion: interface {} is float64, not int to Panic on zero-padded taxonomy weights Apr 18, 2018

@clorenz

This comment has been minimized.

clorenz commented Apr 18, 2018

After changing the manufacturers_weight items to contain no leading zeroes, the panic is gone.

Thank you very much for the great support!

@kaushalmodi

This comment has been minimized.

Member

kaushalmodi commented Apr 19, 2018

@bep @moorereason Does the suggestion made here make sense in this case.. to use strconv.Atoi() or explicitly setting the base to 10 for strconv.ParseInt().. because we don't expect the weights to be octal/hex/..?

@bep bep modified the milestones: v0.39.1, v0.40, v0.41 Apr 19, 2018

@bep bep modified the milestones: v0.41, v0.42 May 4, 2018

@bep bep modified the milestones: v0.42, v0.43 Jun 5, 2018

@bep bep modified the milestones: v0.43, v0.44 Jun 30, 2018

@bep bep modified the milestones: v0.44, v0.45, v0.46 Jul 10, 2018

@bep bep modified the milestones: v0.46, v0.47, v0.48 Aug 3, 2018

@bep bep modified the milestones: v0.48, v0.49 Aug 22, 2018

@bep bep modified the milestones: v0.49, v0.50 Sep 13, 2018

@moorereason

This comment has been minimized.

Contributor

moorereason commented Sep 22, 2018

@kaushalmodi,
My understanding of the situation is that we're retrieving a page parameter, and the result is a float64. The type of value we get back will depend on the library we're using for the front matter parsing (YAML vs TOML vs JSON).

I would fix the two places I linked to above with cast.ToIntE to force the value into an int.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment