[arm64] "hugo -s docs" fails with "function not implemented" #1772

Closed
anthonyfok opened this Issue Jan 13, 2016 · 5 comments

Comments

Projects
None yet
3 participants
@anthonyfok
Member

anthonyfok commented Jan 13, 2016

According to the Debian arm64 build daemon build log, a hugo -s docs would fail with "function not implemented" error:

# TODO: Add "hugo-docs" package in the future
obj-aarch64-linux-gnu/bin/hugo -s docs -d public/html --uglyURLs
2016/01/12 00:33:24 function not implemented
debian/rules:9: recipe for target 'override_dh_auto_build' failed

Initial code search on "function not implemented" led me to golang.org/x/sys/unix, Errno 38 or 0x26, i.e., ENOSYS.

I was able to reproduce this on Debian arm64 developer machine asachi.debian.org, and it is not one single "function not implemented" errors, but random amount of it, from 9 errors to 94812 errors. I'll need more time to investigate.

Note that this problem does not affect armhf and armel platforms.

@anthonyfok anthonyfok changed the title from On arm64, "hugo -s docs -d public/html --uglyURLs" fails with "function not implemented" to On arm64, "hugo -s docs" fails with "function not implemented" Jan 13, 2016

@anthonyfok

This comment has been minimized.

Show comment
Hide comment
@anthonyfok

anthonyfok Jan 13, 2016

Member

After adding

log.SetFlags(log.LstdFlags | log.Lshortfile)

to Hugo’s main.go (credit: http://stackoverflow.com/questions/24809287/how-do-you-get-a-go-golang-program-to-print-the-line-number-of-the-error-it-ju), the source of the error became more apparent:

2016/01/13 04:17:32 viper.go:260: error: function not implemented
2016/01/13 04:17:32 viper.go:260: error: function not implemented
2016/01/13 04:17:32 viper.go:260: error: function not implemented
2016/01/13 04:17:33 viper.go:260: error: function not implemented
2016/01/13 04:17:33 viper.go:260: error: function not implemented
2016/01/13 04:17:33 viper.go:260: error: function not implemented
2016/01/13 04:17:33 viper.go:260: error: function not implemented
2016/01/13 04:17:33 viper.go:260: error: function not implemented
0 draft content
0 future content
165 pages created
0 non-page files copied
0 paginator pages created
41 tags created
0 groups created
in 1100 ms

The error messages were printed by log.Println("error:", err) on Line 260 inside a goroutine, hence the multiple and random numbers of errors. However, it does continue and finish rendering the site.

And, in the case of the current Debian Hugo package, which was compiled with an older viper (commit dated 20151224) and older fsnotify (1.2.1 vs the current 1.2.8), the error was printed a bit earlier, by log.Fatal(error) on line 236, hence the one-shot error message and immediate exit:

2016/01/13 04:21:53 viper.go:236: function not implemented

Nevertheless, they both happen inside viper.WatchConfig() (hence fsnotify), which is called from either commands/hugo.go or commands/server.go, depending on which Hugo command is run.

I will look inside fsnotify next, as I suspect fsnotify could be using a syscall function that is not available on Linux/arm64 for one reason or another, like in golang/go#11918.

Anyhow, this does bring up a question:
@spf13, should commands/hugo.go be running watchConfig() even when it is not running in watch mode?

Member

anthonyfok commented Jan 13, 2016

After adding

log.SetFlags(log.LstdFlags | log.Lshortfile)

to Hugo’s main.go (credit: http://stackoverflow.com/questions/24809287/how-do-you-get-a-go-golang-program-to-print-the-line-number-of-the-error-it-ju), the source of the error became more apparent:

2016/01/13 04:17:32 viper.go:260: error: function not implemented
2016/01/13 04:17:32 viper.go:260: error: function not implemented
2016/01/13 04:17:32 viper.go:260: error: function not implemented
2016/01/13 04:17:33 viper.go:260: error: function not implemented
2016/01/13 04:17:33 viper.go:260: error: function not implemented
2016/01/13 04:17:33 viper.go:260: error: function not implemented
2016/01/13 04:17:33 viper.go:260: error: function not implemented
2016/01/13 04:17:33 viper.go:260: error: function not implemented
0 draft content
0 future content
165 pages created
0 non-page files copied
0 paginator pages created
41 tags created
0 groups created
in 1100 ms

The error messages were printed by log.Println("error:", err) on Line 260 inside a goroutine, hence the multiple and random numbers of errors. However, it does continue and finish rendering the site.

And, in the case of the current Debian Hugo package, which was compiled with an older viper (commit dated 20151224) and older fsnotify (1.2.1 vs the current 1.2.8), the error was printed a bit earlier, by log.Fatal(error) on line 236, hence the one-shot error message and immediate exit:

2016/01/13 04:21:53 viper.go:236: function not implemented

Nevertheless, they both happen inside viper.WatchConfig() (hence fsnotify), which is called from either commands/hugo.go or commands/server.go, depending on which Hugo command is run.

I will look inside fsnotify next, as I suspect fsnotify could be using a syscall function that is not available on Linux/arm64 for one reason or another, like in golang/go#11918.

Anyhow, this does bring up a question:
@spf13, should commands/hugo.go be running watchConfig() even when it is not running in watch mode?

anthonyfok added a commit to anthonyfok/hugo that referenced this issue Jan 13, 2016

Do not call watchConfig() when not in watch mode
See #1772

Also, force DisableLiveReload to true when running "hugo --watch"
(build-only non-server mode) to prevent livereload.ForceRefresh(),
which would end up blocking watchConfig() forever, from being called
because livereload.Initialize() is never called in this case.

This fixes the bug where "hugo --watch" could only reload config.toml
once before it gets stuck for good at livereload.ForceRefresh().

This is also consistent with Hugo's existing behaviour:
Non-server "hugo --watch" has never injected livereload.js
since the inception of the "watch" feature in Hugo v0.12.
@anthonyfok

This comment has been minimized.

Show comment
Hide comment
@anthonyfok

anthonyfok Jan 13, 2016

Member

It turns out the "function not implemented" (ENOSYS) errors originated from Line 85 in inotify_poller.go from fsnotify:

    n, errno := syscall.EpollWait(poller.epfd, events, -1)

Why? Because epoll_wait is deprecated according to a slideshow by Linaro who ported Linux to ARM64, and therefore, only the newer epoll_pwait is provided on Linux/arm64.

Problem is, Go has not yet implemented the corresponding EpollPwait, see the "Unimplemented" section of go/src/syscall/syscall_linux.go:948 and golang.org/x/sys/unix/syscall_linux.go:971.

It seems that the Go developers are aware of the issue, see golang/go#10235, though the milestone there changed from Go1.5 to Go1.6Early, and now Unplanned... Oh well... We'll see what happens next.

In the meantime, if anyone were really using Hugo on ARM64, we can happily report that all features work except --watch. 😉

Member

anthonyfok commented Jan 13, 2016

It turns out the "function not implemented" (ENOSYS) errors originated from Line 85 in inotify_poller.go from fsnotify:

    n, errno := syscall.EpollWait(poller.epfd, events, -1)

Why? Because epoll_wait is deprecated according to a slideshow by Linaro who ported Linux to ARM64, and therefore, only the newer epoll_pwait is provided on Linux/arm64.

Problem is, Go has not yet implemented the corresponding EpollPwait, see the "Unimplemented" section of go/src/syscall/syscall_linux.go:948 and golang.org/x/sys/unix/syscall_linux.go:971.

It seems that the Go developers are aware of the issue, see golang/go#10235, though the milestone there changed from Go1.5 to Go1.6Early, and now Unplanned... Oh well... We'll see what happens next.

In the meantime, if anyone were really using Hugo on ARM64, we can happily report that all features work except --watch. 😉

spf13 added a commit that referenced this issue Jan 29, 2016

Do not call watchConfig() when not in watch mode
See #1772

Also, force DisableLiveReload to true when running "hugo --watch"
(build-only non-server mode) to prevent livereload.ForceRefresh(),
which would end up blocking watchConfig() forever, from being called
because livereload.Initialize() is never called in this case.

This fixes the bug where "hugo --watch" could only reload config.toml
once before it gets stuck for good at livereload.ForceRefresh().

This is also consistent with Hugo's existing behaviour:
Non-server "hugo --watch" has never injected livereload.js
since the inception of the "watch" feature in Hugo v0.12.
@anthonyfok

This comment has been minimized.

Show comment
Hide comment
@anthonyfok

anthonyfok Apr 18, 2016

Member

Looks like it has been fixed in golang.org/x/sys/unix, thanks to Riku Voipio (@suihkulokki) in commit golang/sys@324e137, and the upstream golang/go#10235 has closed accordingly.

I would like to refresh relevant Go packages in Debian and make sure it is all working properly on arm64 test machine before coming back and close this issue. :-)

Member

anthonyfok commented Apr 18, 2016

Looks like it has been fixed in golang.org/x/sys/unix, thanks to Riku Voipio (@suihkulokki) in commit golang/sys@324e137, and the upstream golang/go#10235 has closed accordingly.

I would like to refresh relevant Go packages in Debian and make sure it is all working properly on arm64 test machine before coming back and close this issue. :-)

@anthonyfok anthonyfok referenced this issue in fsnotify/fsnotify Apr 19, 2016

Closed

global: switch to x/sys/unix #135

@anthonyfok

This comment has been minimized.

Show comment
Hide comment
@anthonyfok

anthonyfok Apr 20, 2016

Member

Fixed by Riku Voipio in golang/sys@324e137 and fsnotify/fsnotify#135 (committed by Nathan Youngman).
When compiled with the latest fsnotify v1.3.0, hugo serve (with --watch) works great on Linux/arm64 (aarch64). 👍

Thank you all!

Member

anthonyfok commented Apr 20, 2016

Fixed by Riku Voipio in golang/sys@324e137 and fsnotify/fsnotify#135 (committed by Nathan Youngman).
When compiled with the latest fsnotify v1.3.0, hugo serve (with --watch) works great on Linux/arm64 (aarch64). 👍

Thank you all!

@anthonyfok anthonyfok closed this Apr 20, 2016

@anthonyfok anthonyfok added this to the v0.16 milestone Apr 20, 2016

@spf13

This comment has been minimized.

Show comment
Hide comment
@spf13

spf13 Apr 22, 2016

Contributor

@anthonyfok we should add this to the release notes for 0.16

On Wed, Apr 20, 2016 at 2:53 PM Anthony Fok notifications@github.com
wrote:

Fixed by Riku Voipio in golang/sys@324e137
golang/sys@324e137 and fsnotify/fsnotify#135
fsnotify/fsnotify#135 (committed by Nathan
Youngman).
When compiled with the latest fsnotify v1.3.0, hugo serve (with --watch)
works great on Linux/arm64 (aarch64). [image: 👍]

Thank you all!


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#1772 (comment)

Contributor

spf13 commented Apr 22, 2016

@anthonyfok we should add this to the release notes for 0.16

On Wed, Apr 20, 2016 at 2:53 PM Anthony Fok notifications@github.com
wrote:

Fixed by Riku Voipio in golang/sys@324e137
golang/sys@324e137 and fsnotify/fsnotify#135
fsnotify/fsnotify#135 (committed by Nathan
Youngman).
When compiled with the latest fsnotify v1.3.0, hugo serve (with --watch)
works great on Linux/arm64 (aarch64). [image: 👍]

Thank you all!


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#1772 (comment)

@anthonyfok anthonyfok changed the title from On arm64, "hugo -s docs" fails with "function not implemented" to [arm64] "hugo -s docs" fails with "function not implemented" Jun 13, 2016

tychoish added a commit to tychoish/hugo that referenced this issue Aug 13, 2017

Do not call watchConfig() when not in watch mode
See #1772

Also, force DisableLiveReload to true when running "hugo --watch"
(build-only non-server mode) to prevent livereload.ForceRefresh(),
which would end up blocking watchConfig() forever, from being called
because livereload.Initialize() is never called in this case.

This fixes the bug where "hugo --watch" could only reload config.toml
once before it gets stuck for good at livereload.ForceRefresh().

This is also consistent with Hugo's existing behaviour:
Non-server "hugo --watch" has never injected livereload.js
since the inception of the "watch" feature in Hugo v0.12.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment