use doc environment for servedoc

[Roger-luo]

It seems currently servedoc uses the global environment instead of the doc environment? so it only works when all the doc dependencies are installed in global environment.

[tlienart]

Yes that's correct, it should be easy to fix, I'll try to suggest a patch soon, thanks for the heads up

[tlienart]

@Roger-luo would you mind pulling the branch usedocenv and doing

julia> using LiveServer, Yao;
julia> servedocs(doc_env=true)

I was able to deploy the docs locally but could do with someone else reporting on this, cheers! Provided this works for you I'll merge and do a patch release.

Edit oh wait I think I see the issue with Weave and the looping, let me get back to you on this.

[tlienart]

Ok, so now you can test it (the branch usedocenv) and it works with:

servedocs(doc_env=true, skip_dir=joinpath("docs/src/examples"))

Please tell me if it works for you and I'll merge and release, thank you

A word of explanation + RFC

The issue with the infinite loop was due to Weave taking your .jmd files and generating .md files in the same folder (not in docs/build/...) this meant that new .md files appeared and LiveServer was considering this as modification to your website and therefore triggering a full pass again.
Now you can specify a directory that you can skip so that this doesn't happen:

servedocs(doc_env=true, skip_dir=joinpath("docs/src/examples"))

(use joinpath("docs","src","examples") if you're on windows)

Note that I would recommend re-directing the output of Weave to docs/build/... folder instead but that's up to you of course.

Side note

Your doc build is quite slow due in part to quite a lot of Weave being used (btw, I'd suggest using JuDoc which is much faster than Documenter+Weave but that's beside the point 😋 (*)) this means that the use of LiveServer doesn't really help much in your situation. I tried modifying the markdown a bit and of course it takes like 20s before this change is propagated to your browser because a full pass triggers the whole of Weave generation.

What I would suggest is that in your for each in Examples ... weave you test first to see whether the .jmd file has changed recently and if not, just skip it. Here's how I'd do it:

for each in Examples
    file_path = joinpath(@__DIR__, "src", "examples", join([each, ".jmd"]))
    # check how long ago it was last modified, if over 15 min, skip it (assume an up to date markdown exists)
    (time() - stat(file_path).mtime) < 900 || continue
    @info "expanding $file_path to markdown"
    weave(file_path)
end

That already divides the run time by a factor 2 (you could add a test to check if you're on CI in which case you wouldn't do this for instance); the run time for Documenter is also enormous and could no doubt be simplified a lot by using the appropriate keywords (at least when the docs are built locally and you want stuff to be fast) but I don't know Documenter enough to tell you exactly what should be used here.

In particular it may/should be possible to:

• suppress the link checking
• suppress the docstring checking

and that would speed things up a lot

[Roger-luo]

Oh thanks for your quick and detailed respond! I just tried it seems to work!

Note that I would recommend re-directing the output of Weave to docs/build/... folder instead but that's up to you of course.

I generate the markdown files to let it get compiled with Documenter tho. But I think you are right this should be a better idea.

Your doc build is quite slow due in part to quite a lot of Weave being used (btw, I'd suggest using JuDoc which is much faster than Documenter+Weave but that's beside the point 😋 (*)) this means that the use of LiveServer doesn't really help much in your situation. I tried modifying the markdown a bit and of course it takes like 20s before this change is propagated to your browser because a full pass triggers the whole of Weave generation.

Yes, we have been trying different solutions on this, but I think the reason we are still using Weave is mainly because we still need to convert things to jupyter notebooks when necessary (e.g doing tutorial etc.) It seems JuDocs doesn't support this (yet)? will it generate jupyter notebooks for this use case in the future?

What I would suggest is that in your for each in Examples ... weave you test first to see whether the .jmd file has changed recently and if not, just skip it. Here's how I'd do it:

nice! I'm also using LiveServer for my new book (still work in progress) https://github.com/Roger-luo/Brochure.jl with a custom serve function, I guess I could do the same thing for weave files?

I don't know Documenter enough to tell you exactly what should be used here.

Yeah, I also hope Documenter could provide an API to build only a single markdown file instead of the entire project, but I posted an issue here: JuliaDocs/Documenter.jl#1188

[fredrikekre]

IMO LiveServer should just use the env the user have set up. I don't understand why it has to be integrated in LiveServer.

[Roger-luo]

@fredrikekre but the servedoc function is already integrated with LiveServer, if one want to start a live hosted documentation with LiveServer with one line I don't think why do I need to take care of whether it is using docs project or not. and an example could be developing under a test environment that do not want dependencies in doc, I'll need to keep switching environment if servedoc can't do this itself. The current solution that enabling this by keyword doesn't change any previous behaviour either.

[fredrikekre]

julia --project=docs -e 'using LiveServer; servedocs()'

doesn't work?

[Roger-luo]

no, you will need to use the thing in .travis CI first

julia --project=docs/ -e 'using Pkg; Pkg.develop(PackageSpec(path=pwd())); Pkg.instantiate()'

or it won't build (with the current package source)

[fredrikekre]

But you will have to initialize the env with the proposed changed too.

I just don't think it is a good idea for a package to mess with the users environment.

[Roger-luo]

I think usually if one just initialize the package env, and won't touch the doc env unless we are going to change things in doc build. And this is enabled explicitly by a keyword, I don't see why not provide this convenience, it makes everyone happy. I can just ask other collaborator (especially who don't know much Julia) to modify the documentation ( a lot docs doesn't need any code) by simply type servedoc(doc_env=true) without mess the environment themselves.

[fredrikekre]

I don't really understand what you mean; you will still have to set up the documentation environment yourself, the proposed solution does not do that.

Why do you prefer

julia -e 'using LiveServer; servedocs(doc_env=true)'

over using the built-in support for modifying the environment:

julia --project=docs -e 'using LiveServer; servedocs()'

?

[Roger-luo]

oh I see, yeah. I'll close this.

[tlienart]

Just a few side points:

• @fredrikekre thanks for the input, I think it's a fair point though I'll leave the opportunity to do this with the fair warnings; if you start servedocs as its own terminal process (which is what you suggest with the command line) then what you suggest is definitely best; however if you start servedocs while in a REPL session there may be circumstances in which it makes sense to do switch the doc env.
• @Roger-luo for your Brochure.jl I think you could benefit from using JuDoc; I don't want to oversell but basically if you're not using Documenter specific stuff like scraping code to get docstrings (which you don't seem to), then I think JuDoc will work better and faster. And yes it can generate notebooks (via Literate incidentally) and eval code just like Weave. For instance have a look at https://alan-turing-institute.github.io/MLJTutorials/ which is generated with JuDoc (note that the tutorials can be downloaded as jupyter notebook etc, all of this automatic). If you're interested in this, ping me on Slack or on JuDoc repo and I'm happy to help you (though I'd definitely understand if you have a working setup for your book and don't want to mess with yet another package).

[fredrikekre]

however if you start servedocs while in a REPL session there may be circumstances in which it makes sense to do switch the doc env.

Yea, but this it is very tricky to switch the environment during runtime like that. For example, packages that have already been important might be on wrong versions etc.

[tlienart]

Yes definitely, so I'll add a fair warning and not recommend it and recommend your approach as more robust/fool-proof