Initial checkin of sample problems and parsing script

[pstaabp]

This branch provides sample problems originally in the OPL with embedded comments in a particular format that can be parsed with a provided script. This was originally discussed in openwebwork/webwork2#1936

The sample problems are in doc/sample-problems and there is README.md file with more information especially on the format of the the coding documentation. These problems originally came from the OPL in the directory FortLewis/Authoring/Templates. I only brought over the PGML versions and have updated the code and the comments that were originally on the WeBWorK wiki.

There is a script bin/parse-prob-doc.pl that currently parses the file and outputs the resulting HTML file. This uses a Mojo::Template to do the formatting as HTML--I used this mainly because it lightweight, fast and is already an installed module. The template is bin/prob-template.mt.

This currently only takes a directory of pg files, parses and converts to HTML--outputting to a local directory. I plan to make this recursive to generate the files in an identical directory structure. I think it's quite crude now--perhaps just a proof of concept. Other things that are needed

• in each html file, include the full source of the pg file with documentation comments stripped (the script does the, but the template does not have this). I was thinking either have a link to the source or perhaps a collapsible element in the html page.
• nicer styling with css in common file
• an index page with links to each file (perhaps searchable)
• it may be better as a bash script for the file io with corresponding perl module for the parsing.

Lastly, I think we should have the documentation automatically be moved to another repository (perhaps pg-docs) and use GitHub pages to serve the files. The POD could also live here. We should be able to have a GitHub action run to do this last step (perhaps not automatically, but somehow triggerable).

[somiaj]

@pstaabp Here is my suggestion for the file format/parsing, somiaj@2f66491. The patch is very crude, I was mostly just testing out the fileParser, which to me is working like I envisioned. Also I think we need to remove all of those block comments from the code. I only updated a single problem for this, and won't spend more time on it unless you would like to go this direction.

[pstaabp]

@somiaj I decided to have a marker that documentation blocks start with a #:%name and then options.
I used your code as a base and I think simplified and adapted it. Now the parsing code can just walk through the code knowing where it is without the state variable.

Let me know what you think about this? All of the files in doc/Misc now use this new format and running:

perl /opt/webwork/pg/bin/parse-prob-doc.pl --problem_dir=/opt/webwork/pg/doc/sample-problems/Misc/ --out_dir=some_directory

creates a folder of html files.

[somiaj]

I can see a possible issue (I think) with your changes in which wasn't in my version. Your #: blocks no longer have to be continuous (since they wait for the next #:% line to start the next block), so you could have markdown paragraphs coming from discontinuous blocks being put together without a new line between them, thus two things that you may want to be separate paragraphs get put together into a single one.

Another thought I have is a bit more tricky. I don't like the way the markdown blocks push against the #:. If I were writing these blocks to me the following just looks better.

#:%preamble
#: The following modules need to be loaded:
#:
#:   + PGML.pl
#:   + contextLimtedVector.pl

To me having that space at the start of #: does make things a bit easier to read/write. The trick party is to allow both #: and #: , and to deal with when that extra space should be removed before sending the code to the markdown processor (unsure how badly one space can mess up the parser).

[pstaabp]

I can see a possible issue (I think) with your changes in which wasn't in my version. Your #: blocks no longer have to be continuous (since they wait for the next #:% line to start the next block), so you could have markdown paragraphs coming from discontinuous blocks being put together without a new line between them, thus two things that you may want to be separate paragraphs get put together into a single one.

Well if they don't have a #: then the line will be added to the code block instead. This would probably just add another that line to the code part.

Another thought I have is a bit more tricky. I don't like the way the markdown blocks push against the #:.

I don't think everything needs to be bumped up again the #:. I just tested your example, and it works fine. Any non #:% line would be interpreted as markdown with #: stripped from the beginning of the line (Line 59 of the script) and should preserve space.

[somiaj]

Well if they don't have a #: then the line will be added to the code block instead. This would probably just add another that line to the code part.

What I mean is something like this:

#%section
#: description of some stuff
code
code
#: more description
more code
more code

The second #: line just gets tacked onto the previous description without a new line between them, thus markdown will see this as a single paragraph, and it should probably be seen as two paragraphs.

I don't think everything needs to be bumped up again the #:. I just tested your example, and it works fine. Any non #:% line would be interpreted as markdown with #: stripped from the beginning of the line (Line 59 of the script) and should preserve space.

The space being persevered is the issue that may arise (since spaces mean things to markdown). If you just remove #: from the start of the lines, then if someone adds a space for readabilty reasons the markdown block after parsed would look something like:

 line1
 line2

     Indented line

 line3

So every line has a space in front if it. I think these should be stripped (But maybe the markdown already nicely deals with this so this is a non-issue)

[somiaj]

Note, the first issue about non-continuous markdown blocks is not really an issue if we just make sure the files are formatted correctly to begin with, just something I noticed is all.

[pstaabp]

Added more problems and updated the script to crawl through the doc/sample-problems directory and builds an html file for each problem. There is also now a categories directory (in the output directory) with an index and a list of problems for each category.

Added a list styling with bootstrap and code highlighting with highlight.js.

Next step is to get this script working as a github action that will post to openwebwork.github.io/pg-docs

Another goal is to automatically link POD documentation for non-standard macros.

[pstaabp]

I have a working version of the sample problems in a GitHub page. TLDR: go to https://pstaabp.github.io/pg-docs/.

Currently, I'm testing this on my repository, but the plan will be for this to go to openwebwork.github.io/pg-docs.

I got this working by two GitHub actions in my pg-docs repository (right now in the test-workflow branch).

• first action called "Create Sample Problems Docs"
  • copies the pg repository (currently the sample-problems-doc branch on my repository)
  • runs the bin/parse-prob-doc.pl script in PG/bin
  • commits the generated html code.
• second action called "Deploy static files" copies the contents to the GitHub pages site.

Additionally, I added a copy to clipboard for each of the code blocks. I will create a copy entire field to clipboard action as well.

[pstaabp]

Added some more features. TLDR: https://pstaabp.github.io/pg-docs/

• the pod is now created and posted on the GitHub.io page.
• the script parses each sample problem and automatically links to macro files (except PGStandard, PGML, ...)

If this is looking reasonable, I'll move it to: https://openwebwork.github.io/pg-docs/

[drgrice1]

There are a couple of things I notice that should be changed.

The .mt extension is not a standard extension for Mojolicious template files. Please change those to .html.ep which is the standard for html Mojolicious template files. Otherwise syntax highlighters for Mojolicious template files are not active by default.

Please make the parse-prob-doc.pl file executable.

[drgrice1]

Also, clean up category_layout.mt (and rename it to category_layout.html.ep). The formatting in that file is inconsistent with the formatting of the other template files.

[drgrice1]

I just noticed that the actual filename you used is catagory_layout.mt. Also fix the misspelled "catagory".

[drgrice1]

I see where you got the .mt extension now. I am accustomed to using Mojo::Templates via a Mojolicious app which uses .ep templates. If used directly, they use the .mt extension. So you can stick with that extension.

[somiaj]

@pstaabp looked around a bit, and looking good. Two things I noticed, first is I don't really like the syntax highlighting, it really isn't catered towards pg and PGML files. I wonder if there is a way we could use the PGCodeMirror syntax highlighting to match the webwork2 editor. Second lots of TEXT(beginproblem()); lines that could be removed from the example files.

[drgrice1]

Even before considering pg and PGML syntax, highlight.js does not seem to do well even for syntax highlighting of Perl. It doesn't even have a different color for variables.

Not to mention that the color scheme that highlight.js uses has low foreground/background contrast, and is far from accessible.

[pstaabp]

@drgrice1 Will cleanup the templates. And remove TEXT(beginproblem()); and unnecessary Context('Numeric') calls from examples.

I was wondering if codeMirror might be too heavyweight for this since we are not doing editing, but the advantage is that we have the PGML formatting for this. I'll try switching over to that.

[somiaj]

@pstaabp Seems CodeMirror5 has a runmode addon, https://codemirror.net/5/doc/manual.html#addon_runmode, which can use CodeMirror to use its tools to syntax highlight a text box, but not invoke any of the editor functionality.

[somiaj]

Looking closer, also check the option right under that one, runmode/colorize.js, which is described as:

By default, it'll get all pre tags. Will read the data-lang attribute of these nodes to figure out their language, and syntax-color their content using the relevant CodeMirror mode

[drgrice1]

@pstaabp: I added a pull request to your branch with some improvements to the layout. The main change is instead of generating an index and another copy of that index for each of the categories, a single index file is generated in the root output location with all of it. Bootstrap tabs are used for the categories instead of separate pages.

[somiaj]

One issue that occurred to me with using CodeMirror is the PG.js and all of the codemirror stuff is in the webwork2 repo, not the pg one.

[drgrice1]

@pstaabp: My pull request now also includes a switch from using highlight.js to using CodeMirror and its standalone runmode. For now I just copied the PG.js file from webwork2 into docs/js.

[pstaabp]

Updated with suggestions above and @drgrice1, I updated the exponential problem that uses CSS styling to format instead of niceTables.pl.

[drgrice1]

@pstaabp: I added another pull request to this branch that updates one problem and fixes another.

[drgrice1]

I think this pull request is looking pretty good now. I have not had a chance to look over and test the latest sample problems that were added, but I see that they aren't giving any errors at least when the parser-problem-doc.pl script is run. So I will for now accept them as they are.