New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[armel/mips port] TestPageBundlerSiteRegular fails with v0.40 and v0.40.1 #4672

Closed
anthonyfok opened this Issue Apr 26, 2018 · 6 comments

Comments

Projects
None yet
2 participants
@anthonyfok
Contributor

anthonyfok commented Apr 26, 2018

Hugo v0.40 and v0.40.1 fails to build on armel (ARM EABI with soft float) and only on armel on Debian build daemon:

Note: The same test for mips build took over 10 minutes and go test timed out, but increasing the go test timeout value should fix it.

With the help of git bisect run on a Debian armel test machine, it was found that the first commit that causes TestPageBundlerSiteRegular to fail on armel is 4d26ab3:

(sid_armel-dchroot)foka@abel:~/go/src/github.com/gohugoio/hugo$ time go test -v -run TestPageBundlerSiteRegular ./hugolib/ 
=== RUN   TestPageBundlerSiteRegular
=== PAUSE TestPageBundlerSiteRegular
=== CONT  TestPageBundlerSiteRegular
=== RUN   TestPageBundlerSiteRegular/ugly=false
ERROR 2018/04/26 14:41:26 Resources's GetByPrefix is deprecated and will be removed in Hugo 0.42. We have added the more flexible Resources.GetMatch (find one) and Resources.Match (many) to replace the "prefix" methods. 

These matches by a given globbing pattern, e.g. "*.jpg".

Some examples:

* To find all resources by its prefix in the root dir of the bundle: .Match image*
* To find one resource by its prefix in the root dir of the bundle: .GetMatch image*
* To find all JPEG images anywhere in the bundle: .Match **.jpg
=== RUN   TestPageBundlerSiteRegular/ugly=true
--- FAIL: TestPageBundlerSiteRegular (356.79s)
    --- FAIL: TestPageBundlerSiteRegular/ugly=false (179.46s)
        Error Trace:    testhelpers_test.go:440
    	            	page_bundler_test.go:194
    	Error:      	Should be true
    	Test:       	TestPageBundlerSiteRegular/ugly=false
    	Messages:   	File no match for
    	            	"TheContent" in
    	            	"/work/public/2017/pageslug/index.html":
    	            	
    	            	Single Title: Bundle Galore
    	            	Content: 
    	            	
    	            	
    	            	Sunset RelPermalink: /2017/pageslug/sunset1.jpg
    	            	
    	            	Thumb Width: 123
    	            	Thumb Name: my-sunset-1
    	            	Thumb Title: Sunset Galore 1
    	            	Thumb RelPermalink: /2017/pageslug/sunset1_hu59e56ffff1bc1d8d122b1403d34e039f_90587_123x123_fill_q75_box_smart1.jpg
    	            	
    	            	
    	            	0: Image Title: c/logo.png
    	            	0: Image Name: c/logo.png
    	            	0: Image Params: map[]
    	            	0: Image myParam: Lower:  Caps: 
    	            	
    	            	1: Image Title: Sunset Galore 1
    	            	1: Image Name: my-sunset-1
    	            	1: Image Params: map[myparam:My Sunny Param]
    	            	1: Image myParam: Lower: My Sunny Param Caps: My Sunny Param
    	            	
    	            	2: Image Title: Sunset Galore 2
    	            	2: Image Name: my-sunset-2
    	            	2: Image Params: map[myparam:My Sunny Param]
    	            	2: Image myParam: Lower: My Sunny Param Caps: My Sunny Param
    	            	
    --- FAIL: TestPageBundlerSiteRegular/ugly=true (177.33s)
        Error Trace:    testhelpers_test.go:440
    	            	page_bundler_test.go:170
    	Error:      	Should be true
    	Test:       	TestPageBundlerSiteRegular/ugly=true
    	Messages:   	File no match for
    	            	"TheContent" in
    	            	"/work/public/2017/pageslug.html":
    	            	
    	            	Single Title: Bundle Galore
    	            	Content: 
    	            	
    	            	
    	            	Sunset RelPermalink: /2017/pageslug/sunset1.jpg
    	            	
    	            	Thumb Width: 123
    	            	Thumb Name: my-sunset-1
    	            	Thumb Title: Sunset Galore 1
    	            	Thumb RelPermalink: /2017/pageslug/sunset1_hu59e56ffff1bc1d8d122b1403d34e039f_90587_123x123_fill_q75_box_smart1.jpg
    	            	
    	            	
    	            	0: Image Title: c/logo.png
    	            	0: Image Name: c/logo.png
    	            	0: Image Params: map[]
    	            	0: Image myParam: Lower:  Caps: 
    	            	
    	            	1: Image Title: Sunset Galore 1
    	            	1: Image Name: my-sunset-1
    	            	1: Image Params: map[myparam:My Sunny Param]
    	            	1: Image myParam: Lower: My Sunny Param Caps: My Sunny Param
    	            	
    	            	2: Image Title: Sunset Galore 2
    	            	2: Image Name: my-sunset-2
    	            	2: Image Params: map[myparam:My Sunny Param]
    	            	2: Image myParam: Lower: My Sunny Param Caps: My Sunny Param
    	            	
FAIL
FAIL	github.com/gohugoio/hugo/hugolib	356.897s

real	6m5.887s
user	9m50.685s
sys	0m1.795s

I have no idea what is causing this, but I will try to find out soon.

@anthonyfok anthonyfok self-assigned this Apr 26, 2018

@anthonyfok anthonyfok changed the title from TestPageBundlerSiteRegular fails on armel with v0.40 and v0.40.1 to [armel port] TestPageBundlerSiteRegular fails with v0.40 and v0.40.1 Apr 26, 2018

@bep

This comment has been minimized.

Member

bep commented Apr 26, 2018

My guess it is a timing issue. There is a "timer" config parameter that is default 10 seconds (10000 milliseconds). It is there to bail out of potential circular shortcode loops, but my guess that this can be too short in some low-cpu environments when all the tests are run in parallel.

You can add a cfg.Set("timer", 30000) or something to this test to verify.

@anthonyfok anthonyfok removed their assignment Apr 26, 2018

@anthonyfok

This comment has been minimized.

Contributor

anthonyfok commented Apr 26, 2018

My guess it is a timing issue. ... You can add a cfg.Set("timer", 30000) or something to this test to verify.

That is exactly it! Thank you so much! TestPageBundlerSiteRegular runs correctly on armel after adding cfg.Set("timer", 30000) cfg.Set("timeout", 120000) in that test.

I noticed that you have indeed written a warning in case the timeout happens:

WARNING: Timed out creating content for page %q (.Content will be empty). This is most likely a circular shortcode content loop that should be fixed. If this is just a shortcode calling a slow remote service, try to set "timeout=20000" (or higher, value is in milliseconds) in config.toml.

But that warning does not show up when I tried to run go test -v -run TestPageBundlerSiteRegular/ugly=false ./hugolib, even after I tried adding cfg.Set("verbose", true) and cfg.Set("debug", true). Was I doing it the wrong way? Or are there ways to expose these warnings during the test? Many thanks!

@bep

This comment has been minimized.

Member

bep commented Apr 26, 2018

Or are there ways to expose these warnings during the test? Many thanks!

We should improve on this, but,something ala:

s := buildSingleSite(t, deps.DepsCfg{Logger: newWarningLogger(), Fs: fs, Cfg: cfg}, BuildCfg{})

It is default on ERROR level ... I hate to look at chatty logs when I develop.

@bep

This comment has been minimized.

Member

bep commented Apr 26, 2018

As to the reason why that test takes time, is probably because of some image processing ...

@anthonyfok

This comment has been minimized.

Contributor

anthonyfok commented Apr 26, 2018

As to the reason why that test takes time, is probably because of some image processing ...

Indeed, soft float can be really slow, and according to this https://www.raspberrypi.org/forums/viewtopic.php?t=49970, soft float can be ~100 times slower than hard float. Yikes!

@anthonyfok

This comment has been minimized.

Contributor

anthonyfok commented Apr 27, 2018

Yay! https://buildd.debian.org/status/package.php?p=hugo is all green again in Hugo 0.40.1-3!

Here is the actual patch that went into the Debian package:

https://salsa.debian.org/go-team/packages/hugo/blob/master/debian/patches/004-increase-timeout-in-TestPageBundlerSiteRegular.patch

Special thanks to @ bep for providing the solution!

@anthonyfok anthonyfok closed this Apr 27, 2018

@anthonyfok anthonyfok changed the title from [armel port] TestPageBundlerSiteRegular fails with v0.40 and v0.40.1 to [armel/mips port] TestPageBundlerSiteRegular fails with v0.40 and v0.40.1 Apr 27, 2018

anthonyfok added a commit to anthonyfok/hugo that referenced this issue Apr 27, 2018

Show site build warning in TestPageBundlerSiteRegular
So that a timeout warning does appear if it does happen
especially on a slow machine with soft floating-point CPU.

Special thanks to @bep for the solution.

See gohugoio#4672

anthonyfok added a commit to anthonyfok/hugo that referenced this issue May 14, 2018

Show site build warning in TestPageBundlerSiteRegular
So that a timeout warning does appear if it does happen
especially on a slow machine with soft floating-point CPU.

Special thanks to @bep for the solution.

See gohugoio#4672
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment