From 70aabb9dde005ca68910499e1fe8a6f09f128869 Mon Sep 17 00:00:00 2001 From: Fredrik Ekre Date: Wed, 2 May 2018 14:27:40 +0200 Subject: [PATCH] change #' to # and require ## for regular julia comments, fix #3 --- docs/src/customprocessing.md | 4 +- docs/src/fileformat.md | 30 ++++----- docs/src/outputformats.md | 18 +++--- docs/src/pipeline.md | 48 +++++++-------- examples/example.jl | 116 +++++++++++++++++------------------ src/Literate.jl | 34 +++++----- test/runtests.jl | 98 ++++++++++++++--------------- 7 files changed, 178 insertions(+), 170 deletions(-) diff --git a/docs/src/customprocessing.md b/docs/src/customprocessing.md index b4321b47..27906abe 100644 --- a/docs/src/customprocessing.md +++ b/docs/src/customprocessing.md @@ -27,8 +27,8 @@ We could of course update our source file before generating the docs, but we cou instead use a `preprocess` function that splices the date into the source for us. Consider the following source file: ```julia -#' # Example -#' This example was generated DATEOFTODAY +# # Example +# This example was generated DATEOFTODAY x = 1 // 3 ``` diff --git a/docs/src/fileformat.md b/docs/src/fileformat.md index 5716d946..b25fc408 100644 --- a/docs/src/fileformat.md +++ b/docs/src/fileformat.md @@ -8,25 +8,27 @@ stay up do date with other changes in your package. ## [**2.1.** Syntax](@id Syntax) The basic syntax is simple: -- lines starting with `#'` is treated as markdown, +- lines starting with `# ` are treated as markdown, - all other lines are treated as julia code. -The reason for using `#'` instead of `#` is that we want to be able to use `#` as comments, -just as in a regular script. Lets look at a simple example: +!!! note + If you want regular julia comments in the source file use `## ` instead of `# `. + +Lets look at a simple example: ```julia -#' # Rational numbers -#' -#' In julia rational numbers can be constructed with the `//` operator. -#' Lets define two rational numbers, `x` and `y`: +# # Rational numbers +# +# In julia rational numbers can be constructed with the `//` operator. +# Lets define two rational numbers, `x` and `y`: x = 1//3 y = 2//5 -#' When adding `x` and `y` together we obtain a new rational number: +# When adding `x` and `y` together we obtain a new rational number: z = x + y ``` -In the lines `#'` we can use regular markdown syntax, for example the `#` +In the lines starting with `#` we can use regular markdown syntax, for example the `#` used for the heading and the backticks for formatting code. The other lines are regular julia code. We note a couple of things: - The script is valid julia, which means that we can `include` it and the example will run @@ -60,11 +62,11 @@ using Documenter. Obviously we don't want to include this in the notebook, since `@docs` is Documenter syntax that the notebook will not understand. This is a case where we can prepend `#md` to those lines: ````julia -#md #' ```@docs -#md #' Literate.markdown -#md #' Literate.notebook -#md #' Literate.markdown -#md #' ``` +#md # ```@docs +#md # Literate.markdown +#md # Literate.notebook +#md # Literate.markdown +#md # ``` ```` The lines in the example above would be filtered out in the preprocessing step, unless we are generating a markdown file. When generating a markdown file we would simple remove diff --git a/docs/src/outputformats.md b/docs/src/outputformats.md index 6b04aff3..2a47c0fb 100644 --- a/docs/src/outputformats.md +++ b/docs/src/outputformats.md @@ -4,16 +4,16 @@ When the source is parsed, and have been processed it is time to render the outp We will consider the following source snippet: ```julia -#' # Rational numbers -#' -#' In julia rational numbers can be constructed with the `//` operator. -#' Lets define two rational numbers, `x` and `y`: +# # Rational numbers +# +# In julia rational numbers can be constructed with the `//` operator. +# Lets define two rational numbers, `x` and `y`: x = 1//3 #- y = 2//5 -#' When adding `x` and `y` together we obtain a new rational number: +# When adding `x` and `y` together we obtain a new rational number: z = x + y ``` @@ -45,7 +45,7 @@ z = x + y ``` ```` -We note that lines starting with `#'` is printed as regular markdown, +We note that lines starting with `# ` are printed as regular markdown, and the code lines have been wrapped in `@example` blocks. Some of the output rendering can be controlled with keyword arguments to @@ -77,8 +77,8 @@ In[3]: │ z = x + y Out[3]: │ 11/15 ``` -We note that lines starting with `#'` is put in markdown cells, -and the code lines have been put in code cells. By default the notebook +We note that lines starting with `# ` are placed in markdown cells, +and the code lines have been placed in code cells. By default the notebook is also executed and output cells populated. The current working directory is set to the specified output directory the notebook is executed. Some of the output rendering can be controlled with keyword @@ -101,7 +101,7 @@ y = 2//5 z = x + y ``` -We note that lines starting with `#'` are removed and only the +We note that lines starting with `# ` are removed and only the code lines have been kept. Some of the output rendering can be controlled with keyword arguments to [`Literate.script`](@ref): diff --git a/docs/src/pipeline.md b/docs/src/pipeline.md index 4bd530ff..647b49f7 100644 --- a/docs/src/pipeline.md +++ b/docs/src/pipeline.md @@ -30,38 +30,38 @@ and mark them as either markdown or code according to the rules described in the [Syntax](@ref Syntax) section. Lets consider the example from the previous section with each line categorized: ``` -#' # Rational numbers <- markdown -#' <- markdown -#' In julia rational numbers can be constructed with the `//` operator. <- markdown -#' Lets define two rational numbers, `x` and `y`: <- markdown - <- code -x = 1 // 3 <- code -y = 2 // 5 <- code - <- code -#' When adding `x` and `y` together we obtain a new rational number: <- markdown - <- code -z = x + y <- code +# # Rational numbers <- markdown +# <- markdown +# In julia rational numbers can be constructed with the `//` operator. <- markdown +# Lets define two rational numbers, `x` and `y`: <- markdown + <- code +x = 1 // 3 <- code +y = 2 // 5 <- code + <- code +# When adding `x` and `y` together we obtain a new rational number: <- markdown + <- code +z = x + y <- code ``` In the next step the lines are grouped into "chunks" of markdown and code. This is done by simply collecting adjacent lines of the same "type" into chunks: ``` -#' # Rational numbers ┐ -#' │ -#' In julia rational numbers can be constructed with the `//` operator. │ markdown -#' Lets define two rational numbers, `x` and `y`: ┘ - ┐ -x = 1 // 3 │ -y = 2 // 5 │ code - ┘ -#' When adding `x` and `y` together we obtain a new rational number: ] markdown - ┐ -z = x + y ┘ code +# # Rational numbers ┐ +# │ +# In julia rational numbers can be constructed with the `//` operator. │ markdown +# Lets define two rational numbers, `x` and `y`: ┘ + ┐ +x = 1 // 3 │ +y = 2 // 5 │ code + ┘ +# When adding `x` and `y` together we obtain a new rational number: ] markdown + ┐ +z = x + y ┘ code ``` In the last parsing step all empty leading and trailing lines for each chunk -are removed, but empty lines *within the same* block are kept. The leading `#' ` +are removed, but empty lines *within the same* block are kept. The leading `# ` tokens are also removed from the markdown chunks. Finally we would end up with the following 4 chunks: @@ -112,7 +112,7 @@ The example above would result in two consecutive code-chunks. After the parsing it is time to generate the output. What is done in this step is very different depending on the output target, and it is describe in more detail in the Output format sections: [Markdown Output](@ref), [Notebook Output](@ref) and -[Script Output](@ref). In short, the following is happening: +[Script Output](@ref). Using the default settings, the following is happening: * Markdown output: markdown chunks are printed as-is, code chunks are put inside a code fence (defaults to `@example`-blocks), diff --git a/examples/example.jl b/examples/example.jl index 4cf2abb2..5f526e59 100644 --- a/examples/example.jl +++ b/examples/example.jl @@ -1,43 +1,43 @@ -#' # **7.** Example -#' -#' This is an example generated with Literate based on this -#' source file: [`example.jl`](@__REPO_ROOT_URL__examples/example.jl). -#' You are seeing the -#md #' html-output which Documenter have generated based on a markdown -#md #' file generated with Literate. The corresponding notebook -#md #' can be found here: [`example.ipynb`](@__NBVIEWER_ROOT_URL__generated/example.ipynb), -#nb #' generated notebook output. The corresponding markdown (html) output -#nb #' can be found here: [`example.html`](https://fredrikekre.github.io/Literate.jl/latest/generated/example.html), -#' and the plain script output can be found here: [`example.jl`](./example.jl). +# # **7.** Example +# +# This is an example generated with Literate based on this +# source file: [`example.jl`](@__REPO_ROOT_URL__examples/example.jl). +# You are seeing the +#md # html-output which Documenter have generated based on a markdown +#md # file generated with Literate. The corresponding notebook +#md # can be found here: [`example.ipynb`](@__NBVIEWER_ROOT_URL__generated/example.ipynb), +#nb # generated notebook output. The corresponding markdown (html) output +#nb # can be found here: [`example.html`](https://fredrikekre.github.io/Literate.jl/latest/generated/example.html), +# and the plain script output can be found here: [`example.jl`](./example.jl). -#' It is recommended to have the [source file](@__REPO_ROOT_URL__examples/example.jl) -#' available when reading this, to better understand how the syntax in the source file -#' corresponds to the output you are seeing. +# It is recommended to have the [source file](@__REPO_ROOT_URL__examples/example.jl) +# available when reading this, to better understand how the syntax in the source file +# corresponds to the output you are seeing. -#' ### Basic syntax -#' The basic syntax for Literate is simple, lines starting with `#'` is interpreted -#' as markdown, and all the other lines are interpreted as code. Here is some code: +# ### Basic syntax +# The basic syntax for Literate is simple, lines starting with `# ` is interpreted +# as markdown, and all the other lines are interpreted as code. Here is some code: x = 1//3 y = 2//5 -#' In markdown sections we can use markdown syntax. For example, we can -#' write *text in italic font*, **text in bold font** and use -#' [links](https://www.youtube.com/watch?v=dQw4w9WgXcQ). +# In markdown sections we can use markdown syntax. For example, we can +# write *text in italic font*, **text in bold font** and use +# [links](https://www.youtube.com/watch?v=dQw4w9WgXcQ). -#' It is possible to filter out lines depending on the output using the -#' `#md`, `#nb`, `#jl` and `#src` tags (see [Filtering Lines](@ref)): -#md #' - This line starts with `#md` and is thus only visible in the markdown output. -#nb #' - This line starts with `#nb` and is thus only visible in the notebook output. -#jl #' - This line starts with `#jl` and is thus only visible in the notebook output. -#src #' - This line starts with `#src` and is thus only visible in the source file. +# It is possible to filter out lines depending on the output using the +# `#md`, `#nb`, `#jl` and `#src` tags (see [Filtering Lines](@ref)): +#md # - This line starts with `#md` and is thus only visible in the markdown output. +#nb # - This line starts with `#nb` and is thus only visible in the notebook output. +#jl # - This line starts with `#jl` and is thus only visible in the notebook output. +#src # - This line starts with `#src` and is thus only visible in the source file. -#' The source file is parsed in chunks of markdown and code. Starting a line -#' with `#-` manually inserts a chunk break. For example, if we want to -#' display the output of the following operations we may insert `#-` in -#' between. These two code blocks will now end up in different -#' `@example`-blocks in the markdown output, and two different notebook cells -#' in the notebook output. +# The source file is parsed in chunks of markdown and code. Starting a line +# with `#-` manually inserts a chunk break. For example, if we want to +# display the output of the following operations we may insert `#-` in +# between. These two code blocks will now end up in different +# `@example`-blocks in the markdown output, and two different notebook cells +# in the notebook output. x + y @@ -45,12 +45,12 @@ x + y x * y -#' ### Output Capturing -#' Code chunks are by default placed in Documenter `@example` blocks in the generated -#' markdown. This means that the output will be captured in a block when Documenter is -#' building the docs. In notebooks the output is captured in output cells, if the -#' `execute` keyword argument is set to true. Output to `stdout`/`stderr` is also -#' captured. +# ### Output Capturing +# Code chunks are by default placed in Documenter `@example` blocks in the generated +# markdown. This means that the output will be captured in a block when Documenter is +# building the docs. In notebooks the output is captured in output cells, if the +# `execute` keyword argument is set to true. Output to `stdout`/`stderr` is also +# captured. function foo() println("This string is printed to stdout.") @@ -59,26 +59,26 @@ end foo() -#' ### Custom processing -#' -#' It is possible to give Literate custom pre- and post-processing functions. -#' For example, here we insert two placeholders, which we will replace with -#' something else at time of generation. We have here replaced our placeholders -#' with `z` and `1.0 + 2.0im`: +# ### Custom processing +# +# It is possible to give Literate custom pre- and post-processing functions. +# For example, here we insert two placeholders, which we will replace with +# something else at time of generation. We have here replaced our placeholders +# with `z` and `1.0 + 2.0im`: MYVARIABLE = MYVALUE -#' ### [Documenter.jl interaction](@id documenter-interaction) -#' -#' In the source file it is possible to use Documenter.jl style references, -#' such as `@ref` and `@id`. These will be filtered out in the notebook output. -#' For example, [here is a link](@ref documenter-interaction), but it is only -#' visible as a link if you are reading the markdown output. We can also -#' use equations: -#' -#' ```math -#' \int_\Omega \nabla v \cdot \nabla u\ \mathrm{d}\Omega = \int_\Omega v f\ \mathrm{d}\Omega -#' ``` -#' -#' using Documenters math syntax. Documenters syntax is automatically changed to -#' `\begin{equation} ... \end{equation}` in the notebook output to display correctly. +# ### [Documenter.jl interaction](@id documenter-interaction) +# +# In the source file it is possible to use Documenter.jl style references, +# such as `@ref` and `@id`. These will be filtered out in the notebook output. +# For example, [here is a link](@ref documenter-interaction), but it is only +# visible as a link if you are reading the markdown output. We can also +# use equations: +# +# ```math +# \int_\Omega \nabla v \cdot \nabla u\ \mathrm{d}\Omega = \int_\Omega v f\ \mathrm{d}\Omega +# ``` +# +# using Documenters math syntax. Documenters syntax is automatically changed to +# `\begin{equation} ... \end{equation}` in the notebook output to display correctly. diff --git a/src/Literate.jl b/src/Literate.jl index 179ebfc3..e7a0943e 100644 --- a/src/Literate.jl +++ b/src/Literate.jl @@ -11,7 +11,7 @@ import .Documenter # # Some simple rules: # -# * All lines starting with `#'` are considered markdown, everything else is considered code +# * All lines starting with `# ` are considered markdown, everything else is considered code # * The file is parsed in "chunks" of code and markdown. A new chunk is created when the # lines switch context from markdown to code and vice versa. # * Lines starting with `#-` can be used to start a new chunk. @@ -37,25 +37,31 @@ function parse(content; allow_continued = true) lines = collect(eachline(IOBuffer(content))) chunks = Chunk[] - push!(chunks, startswith(lines[1], "#'") ? MDChunk() : CodeChunk()) + push!(chunks, ismdline(rstrip(lines[1])) ? MDChunk() : CodeChunk()) for line in lines line = rstrip(line) + # print("line = `$line`: ") if startswith(line, "#-") # new chunk # assume same as last chunk, will be cleaned up otherwise push!(chunks, typeof(chunks[end])()) - elseif startswith(line, "#'") # markdown + # println("new chunk") + elseif (line == "#" || startswith(line, "# ")) && !startswith(line, "##") # markdown if !(chunks[end] isa MDChunk) push!(chunks, MDChunk()) end - # remove "#' " and "#'\n" - line = replace(replace(line, r"^#' " => ""), r"^#'$" => "") + # remove "# " and "#\n" + line = replace(replace(line, r"^# " => ""), r"^#$" => "") push!(chunks[end].lines, line) + # println("markdown") else # code if !(chunks[end] isa CodeChunk) push!(chunks, CodeChunk()) end + # remove "## " and "##\n" + line = replace(replace(line, r"^## " => "# "), r"^##$" => "#") push!(chunks[end].lines, line) + # println("code") end end @@ -95,9 +101,9 @@ function parse(content; allow_continued = true) @assert !isempty(merged_chunks) if isa(chunk, CodeChunk) append!(merged_chunks[end].lines, chunk.lines) - else # need to put back #' + else # need to put back "#" for line in chunk.lines - push!(merged_chunks[end].lines, rstrip("#' " * line)) + push!(merged_chunks[end].lines, rstrip("# " * line)) end end else @@ -127,12 +133,12 @@ function replace_default(content, sym; if sym === :jl content *= """ #- - # This file was generated using Literate.jl, https://github.com/fredrikekre/Literate.jl + ## This file was generated using Literate.jl, https://github.com/fredrikekre/Literate.jl """ else content *= """ #- - #' *This $(sym === :md ? "page" : "notebook") was generated using [Literate.jl](https://github.com/fredrikekre/Literate.jl).* + # *This $(sym === :md ? "page" : "notebook") was generated using [Literate.jl](https://github.com/fredrikekre/Literate.jl).* """ end end @@ -217,7 +223,7 @@ Keyword arguments: - `documenter`: boolean that says if the source contains Documenter.jl specific things to filter out during script generation. Defaults to `true`. See the the manual section on [Interaction with Documenter](@ref Interaction-with-Documenter). -- `keep_comments`: boolean that, if set to `true`, keeps markdown lines (`#'`) +- `keep_comments`: boolean that, if set to `true`, keeps markdown lines as comments in the output script. Defaults to `false`. """ function script(inputfile, outputdir; preprocess = identity, postprocess = identity, @@ -248,7 +254,7 @@ function script(inputfile, outputdir; preprocess = identity, postprocess = ident write(ioscript, '\n') # add a newline between each chunk elseif isa(chunk, MDChunk) && keep_comments for line in chunk.lines - write(ioscript, "#' ", line, '\n') + write(ioscript, rstrip("# " * line * '\n')) end write(ioscript, '\n') # add a newline between each chunk end @@ -314,9 +320,9 @@ function markdown(inputfile, outputdir; preprocess = identity, postprocess = ide repo = get(ENV, "TRAVIS_REPO_SLUG", "") pkg = first(split(last(split(repo, '/')), '.')) content = """ - #' ```@meta - #' EditURL = "@__REPO_ROOT_URL__$(replace(relpath(inputfile, Pkg.dir(pkg)), "\\" => "/"))" - #' ``` + # ```@meta + # EditURL = "@__REPO_ROOT_URL__$(replace(relpath(inputfile, Pkg.dir(pkg)), "\\" => "/"))" + # ``` """ * content end diff --git a/test/runtests.jl b/test/runtests.jl index e57049b0..e6028ae0 100644 --- a/test/runtests.jl +++ b/test/runtests.jl @@ -26,27 +26,27 @@ end @testset "Literate.parse" begin content = """ - #' Line 1 + # Line 1 Line 2 - #' Line 3 - #' - #' Line 5 + # Line 3 + # + # Line 5 Line 6 Line 8 - #' Line 9 + # Line 9 #- - #' Line 11 + # Line 11 Line 12 #- Line 14 - #' Line 15 + # Line 15 #----------------- - #' Line 17 + # Line 17 Line 18 #----------------- Line 20 - #' Line 21 + # Line 21 Line 22 Line 23 Line 24 @@ -58,7 +58,7 @@ end #- Line 31 Line 32 - #' Line 33 + # Line 33 Line 34 #- Line 36 @@ -69,18 +69,18 @@ end #- Line 42 Line 43 - #' Line 44 + # Line 44 Line 45 - #' Line 46 + # Line 46 Line 47 - #' Line 48 + # Line 48 #Line 49 Line 50 - #' - #' - #' Line 53 - #' - #' + # + # + # Line 53 + # + # """ expected_chunks = Chunk[ MDChunk(["Line 1"]), @@ -125,8 +125,8 @@ end foreach(x-> println(io, x), c.lines) foreach(x-> println(iows, x, " "), c.lines) else - foreach(x -> println(io, "#' ", x), c.lines) - foreach(x -> println(iows, "#' ", x, " "), c.lines) + foreach(x -> println(io, "# ", x), c.lines) + foreach(x -> println(iows, "# ", x, " "), c.lines) end println(io, "#-") println(iows, "#-") @@ -138,40 +138,40 @@ end end # testset parser content = """ - #' # [Example](@id example-id) - #' [foo](@ref), [bar](@ref bbaarr) + # # [Example](@id example-id) + # [foo](@ref), [bar](@ref bbaarr) x = 1 - #md #' Only markdown + #md # Only markdown #md x + 1 - #nb #' Only notebook + #nb # Only notebook #nb x + 2 - #jl #' Only script + #jl # Only script #jl x + 3 - #src #' Source code only + #src # Source code only Source code only #src - # #' Comment - # another comment + ## # Comment + ## another comment #- for i in 1:10 print(i) - #' some markdown in a code block + # some markdown in a code block end - #' name: @__NAME__ - #' Link to repo root: @__REPO_ROOT_URL__ - #' Link to nbviewer: @__NBVIEWER_ROOT_URL__ # name: @__NAME__ # Link to repo root: @__REPO_ROOT_URL__ # Link to nbviewer: @__NBVIEWER_ROOT_URL__ - - #' PLACEHOLDER1 - #' PLACEHOLDER2 - # PLACEHOLDER3 - # PLACEHOLDER4 - - #' Some math: - #' ```math - #' \\int f(x) dx - #' ``` + ## name: @__NAME__ + ## Link to repo root: @__REPO_ROOT_URL__ + ## Link to nbviewer: @__NBVIEWER_ROOT_URL__ + + # PLACEHOLDER1 + # PLACEHOLDER2 + ## PLACEHOLDER3 + ## PLACEHOLDER4 + + # Some math: + # ```math + # \\int f(x) dx + # ``` """ @testset "Literate.script" begin @@ -192,7 +192,7 @@ content = """ x = 1 x + 3 - # #' Comment + # # Comment # another comment for i in 1:10 @@ -244,9 +244,9 @@ content = """ # keep_comments Literate.script(inputfile, outdir, keep_comments = true) script = read(joinpath(outdir, "inputfile.jl"), String) - @test occursin("#' # Example", script) - @test occursin("#' foo, bar", script) - @test occursin("#' \\int f(x) dx", script) + @test occursin("# # Example", script) + @test occursin("# foo, bar", script) + @test occursin("# \\int f(x) dx", script) end end end @@ -281,7 +281,7 @@ end ```@example inputfile x + 1 - # #' Comment + # # Comment # another comment ``` @@ -416,7 +416,7 @@ end """ "source": [ "x + 2\\n", - "# #' Comment\\n", + "# # Comment\\n", "# another comment" ] """, @@ -425,7 +425,7 @@ end "source": [ "for i in 1:10\\n", " print(i)\\n", - "#' some markdown in a code block\\n", + "# some markdown in a code block\\n", "end" ] """,