Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build hanging #1414

Closed
juliohm opened this issue Sep 12, 2020 · 49 comments
Closed

Build hanging #1414

juliohm opened this issue Sep 12, 2020 · 49 comments

Comments

@juliohm
Copy link
Contributor

juliohm commented Sep 12, 2020

I tried to build the docs of GeoStats.jl today, and for some reason it is hanging. Can you please try to reproduce?

Is there any recent major change to Documenter.jl v0.25.2 and DocumenterTools.jl v0.1.7?

@KristofferC
Copy link
Member

What happens if you try to Ctrl+C it. Does the stacktrace give any information?

@juliohm
Copy link
Contributor Author

juliohm commented Sep 13, 2020 via email

@fredrikekre
Copy link
Member

I tried multiple times to Ctrl C to get info but couldn't figure out myself.

Can you post the stacktrace at least? Any instructions on how to reproduce?

@juliohm
Copy link
Contributor Author

juliohm commented Sep 13, 2020

The basic doc build workflow should reproduce the issue:

] dev GeoStats
; cd .julia/dev/GeoStats/docs
] activate .
] instantiate
include("make.jl")

@juliohm
Copy link
Contributor Author

juliohm commented Sep 13, 2020

Here is the stack trace:

julia> include("make.jl")
[ Info: SetupBuildDirectory: setting up build directory.
[ Info: Doctest: running doctests.
[ Info: ExpandTemplates: expanding markdown templates.
GKS: Possible loss of precision in routine SET_WINDOW
GKS: Rectangle definition is invalid in routine SET_WINDOW
GKS: Rectangle definition is invalid in routine CELLARRAY
invalid range
GKS: Rectangle definition is invalid in routine SET_WINDOW
GKS: Rectangle definition is invalid in routine CELLARRAY
invalid range
GKS: Rectangle definition is invalid in routine SET_WINDOW
GKS: Rectangle definition is invalid in routine CELLARRAY
invalid range
^CERROR: LoadError: InterruptException:
Stacktrace:
 [1] _shrink(::Function, ::Base.OneTo{Int64}, ::Tuple{Array{Int64,1}}) at ./array.jl:2562
 [2] intersect(::Base.OneTo{Int64}, ::Array{Int64,1}) at ./array.jl:2566
 [3] gr_draw_markers(::Plots.Series, ::Array{Float64,1}, ::Array{Float64,1}, ::Tuple{Float64,Float64}, ::Float64, ::Int64) at /home/juliohm/.julia/packages/Plots/M1wcx/src/backends/gr.jl:352
 [4] gr_draw_markers(::Plots.Series, ::Array{Float64,1}, ::Array{Float64,1}, ::Tuple{Float64,Float64}) at /home/juliohm/.julia/packages/Plots/M1wcx/src/backends/gr.jl:346
 [5] gr_display(::Plots.Subplot{Plots.GRBackend}, ::Measures.Length{:mm,Float64}, ::Measures.Length{:mm,Float64}, ::Array{Float64,1}) at /home/juliohm/.julia/packages/Plots/M1wcx/src/backends/gr.jl:1672
 [6] gr_display(::Plots.Plot{Plots.GRBackend}, ::String) at /home/juliohm/.julia/packages/Plots/M1wcx/src/backends/gr.jl:680
 [7] _show(::Base.GenericIOBuffer{Array{UInt8,1}}, ::MIME{Symbol("image/svg+xml")}, ::Plots.Plot{Plots.GRBackend}) at /home/juliohm/.julia/packages/Plots/M1wcx/src/backends/gr.jl:1966
 [8] show at /home/juliohm/.julia/packages/Plots/M1wcx/src/output.jl:215 [inlined]
 [9] __binrepr at ./multimedia.jl:159 [inlined]
 [10] _textrepr at ./multimedia.jl:151 [inlined]
 [11] #stringmime#6 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Base64/src/Base64.jl:43 [inlined]
 [12] stringmime(::MIME{Symbol("image/svg+xml")}, ::Plots.Plot{Plots.GRBackend}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Base64/src/Base64.jl:43
 [13] display_dict(::Plots.Plot{Plots.GRBackend}) at /home/juliohm/.julia/packages/Documenter/pjwqp/src/Utilities/Utilities.jl:667
 [14] #invokelatest#1 at ./essentials.jl:710 [inlined]
 [15] invokelatest at ./essentials.jl:709 [inlined]
 [16] runner(::Type{Documenter.Expanders.ExampleBlocks}, ::Markdown.Code, ::Documenter.Documents.Page, ::Documenter.Documents.Document) at /home/juliohm/.julia/packages/Documenter/pjwqp/src/Expanders.jl:582
 [17] dispatch(::Type{Documenter.Expanders.ExpanderPipeline}, ::Markdown.Code, ::Vararg{Any,N} where N) at /home/juliohm/.julia/packages/Documenter/pjwqp/src/Utilities/Selectors.jl:167
 [18] expand(::Documenter.Documents.Document) at /home/juliohm/.julia/packages/Documenter/pjwqp/src/Expanders.jl:42
 [19] runner(::Type{Documenter.Builder.ExpandTemplates}, ::Documenter.Documents.Document) at /home/juliohm/.julia/packages/Documenter/pjwqp/src/Builder.jl:227
 [20] dispatch(::Type{Documenter.Builder.DocumentPipeline}, ::Documenter.Documents.Document) at /home/juliohm/.julia/packages/Documenter/pjwqp/src/Utilities/Selectors.jl:167
 [21] #2 at /home/juliohm/.julia/packages/Documenter/pjwqp/src/Documenter.jl:241 [inlined]
 [22] cd(::Documenter.var"#2#3"{Documenter.Documents.Document}, ::String) at ./file.jl:104
 [23] #makedocs#1 at /home/juliohm/.julia/packages/Documenter/pjwqp/src/Documenter.jl:240 [inlined]
 [24] top-level scope at /home/juliohm/.julia/dev/GeoStats/docs/make.jl:12
 [25] include(::String) at ./client.jl:457
 [26] top-level scope at REPL[1]:1
in expression starting at /home/juliohm/.julia/dev/GeoStats/docs/make.jl:12

Maybe it is an issue in Plots/GR?

@KristofferC
Copy link
Member

Looks like it is trying to show an SVG.

@juliohm
Copy link
Contributor Author

juliohm commented Sep 13, 2020

The build worked before with SVG plots. I will try to set all to PNG to see if the build finishes.

@juliohm
Copy link
Contributor Author

juliohm commented Sep 13, 2020

I tried commenting all pages in make.jl to only include index.md. The build still hangs even when I force the format to be png with gr(format=:png) in the little example.

@KristofferC
Copy link
Member

Does it hang for any figure you include? Seems like all docs that uses Plots would hit this then?

@juliohm
Copy link
Contributor Author

juliohm commented Sep 13, 2020

Now I removed the entire example from the index.md page and the build still hangs. Could you reproduce the issue or it is only happening with me?

@juliohm
Copy link
Contributor Author

juliohm commented Sep 13, 2020

It still has to do with GR:

julia> include("make.jl")
[ Info: SetupBuildDirectory: setting up build directory.
[ Info: Doctest: running doctests.
[ Info: ExpandTemplates: expanding markdown templates.
^CERROR: LoadError: InterruptException:
Stacktrace:
 [1] Array at ./boot.jl:406 [inlined]
 [2] Array at ./boot.jl:415 [inlined]
 [3] zeros at ./array.jl:525 [inlined]
 [4] zeros at ./array.jl:521 [inlined]
 [5] rehash!(::Dict{Int64,Nothing}, ::Int64) at ./dict.jl:191
 [6] _setindex! at ./dict.jl:367 [inlined]
 [7] setindex!(::Dict{Int64,Nothing}, ::Nothing, ::Int64) at ./dict.jl:388
 [8] push! at ./set.jl:57 [inlined]
 [9] union!(::Set{Int64}, ::Base.OneTo{Int64}) at ./abstractset.jl:91
 [10] Set at ./set.jl:10 [inlined]
 [11] _Set at ./set.jl:23 [inlined]
 [12] Set at ./set.jl:21 [inlined]
 [13] _shrink(::Function, ::Base.OneTo{Int64}, ::Tuple{Array{Int64,1}}) at ./array.jl:2562
 [14] intersect(::Base.OneTo{Int64}, ::Array{Int64,1}) at ./array.jl:2566
 [15] gr_draw_markers(::Plots.Series, ::Array{Float64,1}, ::Array{Float64,1}, ::Tuple{Float64,Float64}, ::Float64, ::Int64) at /home/juliohm/.julia/packages/Plots/M1wcx/src/backends/gr.jl:352
 [16] gr_draw_markers(::Plots.Series, ::Array{Float64,1}, ::Array{Float64,1}, ::Tuple{Float64,Float64}) at /home/juliohm/.julia/packages/Plots/M1wcx/src/backends/gr.jl:346
 [17] gr_display(::Plots.Subplot{Plots.GRBackend}, ::Measures.Length{:mm,Float64}, ::Measures.Length{:mm,Float64}, ::Array{Float64,1}) at /home/juliohm/.julia/packages/Plots/M1wcx/src/backends/gr.jl:1672
 [18] gr_display(::Plots.Plot{Plots.GRBackend}, ::String) at /home/juliohm/.julia/packages/Plots/M1wcx/src/backends/gr.jl:680
 [19] _show(::Base64.Base64EncodePipe, ::MIME{Symbol("image/png")}, ::Plots.Plot{Plots.GRBackend}) at /home/juliohm/.julia/packages/Plots/M1wcx/src/backends/gr.jl:1966
 [20] show(::Base64.Base64EncodePipe, ::MIME{Symbol("image/png")}, ::Plots.Plot{Plots.GRBackend}) at /home/juliohm/.julia/packages/Plots/M1wcx/src/output.jl:215
 [21] base64encode(::Function, ::MIME{Symbol("image/png")}, ::Vararg{Any,N} where N; context::Nothing) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Base64/src/encode.jl:206
 [22] _binstringmime at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Base64/src/Base64.jl:46 [inlined]
 [23] stringmime(::MIME{Symbol("image/png")}, ::Plots.Plot{Plots.GRBackend}; context::Nothing) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Base64/src/Base64.jl:43
 [24] stringmime(::MIME{Symbol("image/png")}, ::Plots.Plot{Plots.GRBackend}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Base64/src/Base64.jl:43
 [25] display_dict(::Plots.Plot{Plots.GRBackend}) at /home/juliohm/.julia/packages/Documenter/pjwqp/src/Utilities/Utilities.jl:667
 [26] #invokelatest#1 at ./essentials.jl:710 [inlined]
 [27] invokelatest at ./essentials.jl:709 [inlined]
 [28] runner(::Type{Documenter.Expanders.ExampleBlocks}, ::Markdown.Code, ::Documenter.Documents.Page, ::Documenter.Documents.Document) at /home/juliohm/.julia/packages/Documenter/pjwqp/src/Expanders.jl:582
 [29] dispatch(::Type{Documenter.Expanders.ExpanderPipeline}, ::Markdown.Code, ::Vararg{Any,N} where N) at /home/juliohm/.julia/packages/Documenter/pjwqp/src/Utilities/Selectors.jl:167
 [30] expand(::Documenter.Documents.Document) at /home/juliohm/.julia/packages/Documenter/pjwqp/src/Expanders.jl:42
 [31] runner(::Type{Documenter.Builder.ExpandTemplates}, ::Documenter.Documents.Document) at /home/juliohm/.julia/packages/Documenter/pjwqp/src/Builder.jl:227
 [32] dispatch(::Type{Documenter.Builder.DocumentPipeline}, ::Documenter.Documents.Document) at /home/juliohm/.julia/packages/Documenter/pjwqp/src/Utilities/Selectors.jl:167
 [33] #2 at /home/juliohm/.julia/packages/Documenter/pjwqp/src/Documenter.jl:241 [inlined]
 [34] cd(::Documenter.var"#2#3"{Documenter.Documents.Document}, ::String) at ./file.jl:104
 [35] #makedocs#1 at /home/juliohm/.julia/packages/Documenter/pjwqp/src/Documenter.jl:240 [inlined]
 [36] top-level scope at /home/juliohm/.julia/dev/GeoStats/docs/make.jl:12
 [37] include(::String) at ./client.jl:457
 [38] top-level scope at REPL[1]:1
in expression starting at /home/juliohm/.julia/dev/GeoStats/docs/make.jl:12

@KristofferC
Copy link
Member

Does manually making ​this call

 stringmime​(​::​MIME{Symbol("image/png")}​, ​::​Plots.Plot{Plots.GRBackend...

also repro it?

@juliohm
Copy link
Contributor Author

juliohm commented Sep 13, 2020

So where I should try to call this line? I erased all example code in index.md, and don't understand how Plots.jl is still involved in this error message.

@KristofferC
Copy link
Member

Generate your plot you use in the docs and then call showmime on it with that mime type as the first argument.

@juliohm
Copy link
Contributor Author

juliohm commented Sep 13, 2020 via email

@KristofferC
Copy link
Member

Maybe it's a docstring?

@juliohm
Copy link
Contributor Author

juliohm commented Sep 13, 2020 via email

@mortenpi
Copy link
Member

It looks like I can reproduce this, but would need a smaller MWE to figure out where it might be going wrong.

You could try running it with Documenter 0.25.1, 0.25.0 and 0.24.11, to see if it is something that changed in Documenter, although I don't think we've changed anything related to that recently (you can see the CHANGELOG).

I literally commented all doc pages, left index.md only in make.jl and erased all code snippets from index.md

Documenter will build the .md pages even if they are not listed in make.jl, so you'd actually have to remove the .md files.

@juliohm
Copy link
Contributor Author

juliohm commented Sep 16, 2020

@mortenpi I've tried with Documenter 0.25.1 and 0.25.0 and the build still hangs. I remember it was working just fine with Documenter 0.25 compat bound in the docs/Project.toml. I've erased all markdown files with plot commands and the build passed. The hanging must be related to some change in Plots.jl/GR.jl.

@jheinen are you aware of any recent change to the GR backend that may be causing this issue? Looking at the release notes of Plots.jl I found the following changes that could potentially be related?

@jheinen
Copy link

jheinen commented Sep 16, 2020

I tried a lot of combinations of different GeoStats, Plots, GR and even Julia versions. In all cases the problem is in this section. The for (i, rng) in enumerate(iter_segments(series, :scatter)) code seems to produce an endless loop, but i could not find out, why?

[Q] @juliohm : What is the last known combination of the above mentioned packages which worked?

@juliohm
Copy link
Contributor Author

juliohm commented Sep 16, 2020

The latest stable release of GeoStats.jl built successfully including the docs: https://travis-ci.org/github/JuliaEarth/GeoStats.jl/jobs/725449742

@jheinen
Copy link

jheinen commented Sep 16, 2020

That was a long time ago. Of course I have not tested such old Plots versions.

@juliohm
Copy link
Contributor Author

juliohm commented Sep 16, 2020

The build above was triggered 8 days ago. How is it a long time ago? So probably some recent compat bump in the Plots ecosystem introduced the bug?

@juliohm
Copy link
Contributor Author

juliohm commented Sep 16, 2020

My real wish is to never have to rely on Plots.jl in the future. If GR.jl had a recipe system developed just for GR, I would adopt it and forget about Plots.jl entirely. It is too much headache to depend on a project that is constantly breaking, and this is the most consistent feature of Plots.jl, unfortunately.

@jheinen
Copy link

jheinen commented Sep 16, 2020

The build above was triggered 8 days ago. How is it a long time ago? So probably some recent compat bump in the Plots ecosystem introduced the bug?

May be I'm wrong. But wasn't this build based on Plots 0.29.9 (tagged in March)?

@juliohm
Copy link
Contributor Author

juliohm commented Sep 16, 2020

Yes, and apparently some of my indirect dependencies were holding on Plots.jl. During this last week some package must have been updated somewhere, and consequently Plots.jl was bumped to version 1.6 in the docs build.

@KristofferC
Copy link
Member

I tend to check in a Manifest for the docs.

@juliohm
Copy link
Contributor Author

juliohm commented Sep 17, 2020

I used to do the same, but people recommended not checking it to guarantee that things were still working with the latest version of the dependencies. Like an additional layer of "tests".

@juliohm
Copy link
Contributor Author

juliohm commented Sep 22, 2020

Ok, the build is finishing after hours waiting. So it is not hanging but is extremely low to generate the plots.

@mortenpi
Copy link
Member

I used to do the same, but people recommended not checking it to guarantee that things were still working with the latest version of the dependencies. Like an additional layer of "tests".

I guess the middle ground would be to have compat bounds in docs/Project.toml?

@juliohm
Copy link
Contributor Author

juliohm commented Sep 27, 2020 via email

@juliohm
Copy link
Contributor Author

juliohm commented Jan 11, 2021

I am coming back to this issue again. I need an updated version of Plots.jl in the docs build, and the build is hanging when I drop the compat entry in the docs Project.toml. Can you still reproduce the issue with the master branch of GeoStats.jl?

@mortenpi
Copy link
Member

@juliohm I tried building the docs for GeoStats master -- it completed, but it did take 80 minutes.

@juliohm
Copy link
Contributor Author

juliohm commented Jan 17, 2021

@mortenpi yes, so the issue persists with the latest stable release of Plots.jl, and this only happens in the documentation build. If you copy/paste all the scripts in the REPL they should execute instantaneously.

@jheinen
Copy link

jheinen commented Jan 17, 2021

Is the documentation at least complete after 80 minutes? Are there any timeout or error messages related to Plots/GR?

@mortenpi
Copy link
Member

It would really be good to have an MWE, like a single at-example block that is much slower in Documenter compared to the REPL.

@jheinen I didn't check it too thoroughly, but I didn't see any error messages and the plots appear to be there.

@juliohm
Copy link
Contributor Author

juliohm commented Jan 17, 2021

@jheinen perhaps we should try to play with different GKS output formats? I remember that the test suite and the docs set this environment variable for GR on head-less nodes:

ENV["GKSwstype"] = "100"

@juliohm
Copy link
Contributor Author

juliohm commented Jan 17, 2021

Nevermind, the tests are executing just fine, so this shouldn't be the issue.

@juliohm
Copy link
Contributor Author

juliohm commented Mar 2, 2021

It seems that other people are experiencing similar issues with Documenter builds where code that runs quickly on the REPL takes hours on the documentation build. See above the linked issue.

Also, I noticed that in my specific case the option marker_z=rand(10000) in a scatter plot is causing a lot of delay in the plot. @jheinen anything that may have recently changed in the GR backend regarding scatter plots with colors? Something that may have affected the performance of the plot considerably?

I am always coming back to this issue when I need to release a major release of the project. :(

@jheinen
Copy link

jheinen commented Mar 2, 2021

using GR
scatter(randn(10_000), marker_z=randn(10_000))

works, but it is orders of magnitude slower than

using GR
scatter(1:10000, randn(10_000), ones(10_000) .* 200, randn(10_000) .* 255)

whatever backend I use. E.g, with 100_000 points, Plots already seems to hang - I didn't wait for the plot being completed.

There must be a problem with the scatter() implementation in the Plots backend for GR.

I will make further tests, but it would helpful to have a concrete MWE in plain Plots or GR.

@juliohm
Copy link
Contributor Author

juliohm commented Mar 2, 2021

I see some hope in this direction. Maybe the entire issue is just the marker_z option that was compromised by someone in a PR in Plots.jl.

@juliohm
Copy link
Contributor Author

juliohm commented Mar 2, 2021

@jheinen I was able to reproduce the issue with a MWE in this Gist: https://gist.github.com/juliohm/de0ceeb47b9076b626d9a0a793c658d6

It is indeed the option marker_z that is causing the very long waits.

@jheinen
Copy link

jheinen commented Mar 2, 2021

Ok, I'll try to find the cause ASAP. Unfortunately, the backend has been modified considerably in the meantime and some low-level functions have been implemented in Julia.

@KristofferC
Copy link
Member

Can this be closed? It seems like an issue with Plots being slow and has nothing to do with Documenter?

@juliohm
Copy link
Contributor Author

juliohm commented Mar 2, 2021

It seems that other people had similar hangs (see linked issue above in Turing.jl) unrelated to Plots.jl but with progress bars. It would good idea to double check first that after fixing the plot issue the hang is gone for my specific case, and then check if the Turing.jl people are still experiencing the issue.

@KristofferC
Copy link
Member

Regarding Turing.jl, there is already an issue open about that. Since this turned out to just be a Plots slowdown (with a non Documenter MWE), I'll close this.

@juliohm
Copy link
Contributor Author

juliohm commented Mar 2, 2021

Again, you are assuming that after fixing the Plots.jl issue the docs hang will be gone. I don't understand this urge to close the issue.

@KristofferC
Copy link
Member

There has been zero evidence that shows this has anything at all to do with Documenter so I don't see your urge in keeping an issue open on the Documenter.jl repo about it. Closing issues helps focus on real issues. I'd suggest opening an issue with the Plots MWE on the Plots repo.

ziotom78 added a commit to ziotom78/Healpix.jl that referenced this issue Dec 9, 2021
* Add tests for Julia 1.7

* Remove Julia 1.6 from CI test matrix

* Fix errors in Documenter.jl

* Avoid using `gr()` in documentation, as it hangs Documenter in CI builds (see [issue #1414](JuliaDocs/Documenter.jl#1414)), and switch to `pyplot()`
@MattWillFlood
Copy link

This problem seems to be common enough as this issue has been reported several times on different message boards.
I'm experiencing the same problem where the build hangs on info: ExpandTemplates

I'm no expert in Julia, but I don't think this problem is with Plots.jl...
When I run make.jl in the Julia REPL GUI (instead of VScode where I normally code), I get this issue:

julia> include("make.jl")
ERROR: LoadError: MethodError: no method matching Documenter.Writers.HTMLWriter.HTML(; prettyurls=false, canonical="https://mattwillflood.github.io/EntropyHub.jl", assets=String[], collapselevel=1)
Closest candidates are:
Documenter.Writers.HTMLWriter.HTML(; prettyurls, disable_git, edit_branch, canonical, assets, analytics) at C:\Users\MyPC.julia\packages\Documenter\IGqbY\src\Writers\HTMLWriter.jl:136 got unsupported keyword argument "collapselevel"
Stacktrace:

When I remove the collapselevel keyword (which worked in previous versions), the GUI crashes.

alexandrebouchard added a commit to Julia-Tempering/Pigeons.jl that referenced this issue Apr 1, 2023
- update doc to workaround the issue in
  JuliaDocs/Documenter.jl#1414
  (for some reason the problem does not come up with the VS REPL)
- remove Pigeons from docs/Project.toml (subtle, nasty bug, see comments
  there)
- remove a @provides that creates some mysterious problems when
  generating the informal interface pages
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants