Switch examples to use Literate.jl and update to 1.0

[ericphanson]

Closes #335, closes #268, closes #201 (which I'm not sure is relevant now anyway, but with the examples updated it surely can be closed). Supersedes #261.

I converted the examples to use Literate.jl. The notebook files can be generated by examples/build.jl, which is automatically done in docs/make.jl so that we can serve the built notebooks in the docs. There may be a better way to do the docs building; currently, Documenter emits warnings for the links to the notebooks (edit: just had some bad paths).

I also would like to render an HTML version so the user doesn't need to open the Jupyter notebooks necessarily. That requires the notebooks to run though! Hence I've updated all the notebooks to 1.0, and standardized plotting on Plots.jl as suggested in #338. I use a mix of ECOS and SCS since ECOS is better suited for some problems. The docs/make.jl file now uses Literate to build markdown versions of all the examples which it includes in the docs.

To do:

• Provide a way for the user to download Jupyter notebooks. Currently I generate them during the docs build step, but don't link to them anywhere. The catch is that some examples need auxiliary files.
• Check for more latex / render errors.
• Check that they all run on 1.0. I've been using 1.3-rc3, so possibly I've used features that aren't available in 1.0.
• Update the examples/build.jl script so it's easy to work with one example at a time (if you're updating or adding a new one).
• Update readme to no longer say the examples are out of date

---

Code for converting examples:

This was run in the examples directory.

import JSON

function unliterate(path, output)
    @assert !isdir(path)
    @assert !isdir(output)
    endswith(path, ".ipynb") || throw(ArgumentError("only Jupyter notebooks are supported"))
    nb = open(JSON.parse, path, "r")
    out = open(output, "w")
    prev_type = ""
    if !haskey(nb, "cells")
        apath = abspath(path)
		  # need to update notebook format
        run(`python -m nbconvert --to notebook --nbformat 4 $apath --output $apath`)
        @warn "Upgrading to nb4 " path 
        nb = open(JSON.parse, path, "r")

    end
    for cell in nb["cells"]
        type = cell["cell_type"]
        type == prev_type && print(out, "#-\n\n")
        prev_type = type
        
        if type == "markdown"
            for line in cell["source"]
                print(out, "#")
                if !isempty(strip(line))
                    print(out, " ")
                    print(out, line)
                else
                    print(out, "\n")
                end
            end
            print(out, "\n\n")
        elseif type == "code"
            for line in cell["source"]
                startswith(line, "# ") && print(out, "#")
                print(out, line)
            end
            print(out, "\n\n")
        else
            error("unknown cell type $type")
        end
    end
    close(out)
    return
end

function literate_folder(dir, output)
    !isdir(output) && mkdir(output)
    for file in readdir(dir)
        file = joinpath(dir, file)
        if endswith(file, ".ipynb")
            file_out = joinpath(output, split(basename(file), ".")[1] * "_literate.jl")
            @info "Unliterating" file file_out
            unliterate(file, file_out)
        elseif !isdir(file)
            file_out = joinpath(output, basename(file))
            @info "Copying" file file_out
            cp(file, file_out; force=true)
        end
    end
end

examples_path = "."
literate_path = "../literate"
!isdir(literate_path) && mkdir(literate_path)
for dir in filter(isdir, readdir(examples_path))
    dir == literate_path && continue
    output = joinpath(literate_path, dir)
    literate_folder(dir, output)
end

dir = "."
output = joinpath(literate_path, dir)
literate_folder(dir, output)

which was modified from https://github.com/JuliaComputing/JuliaAcademyMaterials/blob/master/unliterate.jl.

[ericphanson]

Some reasoning for decisions I made in this PR

• Why move examples to a subdirectory of docs? I initially had each folder of examples have its own project. That was kind of nice so you didn't install Dataframes and RDatasets if you were working just with examples that don't need them. But then I needed to activate each folder in order to run the code when generating the documentation. It was pointed out to me on #pkg-usage on the Julia slack that this was a bad idea, loading packages then activating new environments and loading more packages. I didn't want to spawn new Julia processes to run the code (it's slow enough already), so I decided to make a single environment for all the examples. Then we're down to two environments: one for the docs folder and one for the examples, and we have the same problem (want to activate two environments). So instead I realized it was simpler to have a nested structure of docs/examples/ instead of separate docs/ and examples/ and try to copy files out of examples/ when generating the docs. Since the examples are embedded in the docs now, I think the subdirectory structure makes sense.
• Why provide the Jupyter notebooks as one zip file? I was initially thinking that for each example I would generate a link to the Jupyter-notebook version of that example. The problem is that some examples need auxiliary files, so then I'd need a way to provide a link to a bundle of files. A zip of files was a natural way to do that, but at that point it seems reasonable to just bundle all the examples instead of requiring someone to download each one individually. This also has the advantage that we can provide a README and a Project.toml to try to make it easier to run them.
• Why generate the Jupyter notebooks with execute=false? Docs build time is already 20 minutes on Travis (it's only a few minutes on my laptop though; I think part of the difference in speed is that it has to install all the packages each time on Travis), so it would just be longer to execute all the notebooks. The advantage would be if one of the notebooks doesn't run, we would actually get an error in that case, and CI would fail, which would be ideal. Currently, if there is a problem executing the code in the notebooks, we just get warnings from Documenter but CI will still pass. So maybe this should be revisited.

Followup things to do

• Have CI fail if one of the examples doesn't run; possibly execute=true on the notebooks as mentioned above. Another possibility is strict=true on makedocs; however, we currently generate a few warnings which will then fail CI.
• Have a way to preview the docs from a PR. This has been done here: test preview push of docs fredrikekre/Literate.jl#75 and is tracked here: Add feature to make deploying preview docs for a PR easier JuliaDocs/Documenter.jl#1131. This way docs PRs can be visually inspected to see if they look right (examples ran, formatting, etc).
• Unify data formats? Currently we use DelimitedFiles, MAT, and hardcoded data in .jl files. It would be nice to stick to one.
• Have a way to render single examples to html without having to call makedocs (which generates all the examples and hence is slow). Documenter.jl doesn't seem to provide a method to do this; I tried also rendering the markdown files from Literate using Weave, but I ran into a problem with the paths to auxiliary data (@__DIR__ pointing to the wrong place). It also doesn't look the same as it does coming from Documenter (different CSS etc). However, since the Literate .jl files are directly runnable and one can just use an external markdown previewer, the formatting + code of single examples can be each checked separately without needing to call makedocs, so the fact that we are lacking this point doesn't seem so troublesome.

PRs welcome :)

[ericphanson]

Travis passes; I think the coveralls decrease is spurious since I haven't changed anything in /src or /test.

I'll go ahead and merge this-- it would be nice to preview to make sure everything works (it does locally at least), but I'll just fix it up after if there are problems.

edit: I decided to rebase first, so I'll just merge tomorrow if CI looks good still.

[fredrikekre]

Nice LOC net deletion here: +1,866 −41,342 :)