Parse markdown descriptions #1106

xsebek · 2023-02-13T14:22:37Z

use CommonMark to parse simple markdown AST parametrised on inline/block code
validate swarm code (Document Text -> Document Syntax)
update descriptions to use markdown with following conventions:
- move - valid swarm code (the easy to write default)
- wedge{=entity} - for swarm entities
- unit{=type} - for swarm types
- require <a> <b>{=snippet} - raw snippets for invalid code
- Alt-G - bold for keyboard shortcuts

highlight code in brick widgets
closes Check code examples in entity descriptions as part of CI #309
closes Highlight commands in descriptions #545
precedes Parse, validate, and highlight code blocks #574
precedes Parse and validate swarm types in descriptions #1406
precedes Check entities in descriptions #1407

xsebek

@byorgey its only in a MVP stage right now, but it already caught this inaccuracy:

    parse scenario "data/scenarios/Tutorials/requireinv.yaml":  FAIL
      test/integration/Main.hs:115:
      Aeson exception:
      Error in $.objectives[0].goal[0]: 1:8:
        |
      1 | require
        |        ^
      reserved word 'require' cannot be used as variable name

Because require is not a command we can not use it on its own. 😄

Thinking about this a bit, maybe I could handle reserved words. It would require (pun intended) a bit more work to highlight them properly later though. 🤔

I think we should not use them for entities though and only use code for valid code snippets.

xsebek · 2023-02-13T14:27:54Z

src/Data/Text/Markdown.hs

+    readTerm txt & \case
+      Left e -> Left (T.unpack e)
+      Right Nothing -> Left "empty code"
+      Right (Just s) -> case processParsedTerm s of


We currently use inline code for types like dir -> cmd unit.

I think I can solve the type signatures as an alternative parse.

Yeah, we can make a parallel to readTerm for types, like readType = runParser (fully parseType). If readTerm fails you can just try running readType too.

I turned types to "raw inlines" for now:

`cmd a`{=type}

src/Data/Text/Markdown.hs

src/Swarm/Game/Scenario/Objective.hs

xsebek · 2023-02-13T14:55:05Z

src/Swarm/TUI/View/Objective.hs

    titleWidget = maybe (txt "?") withEllipsis textSource

 singleGoalDetails :: GoalEntry -> Widget Name
 singleGoalDetails = \case
-  Goal _gs obj -> displayParagraphs $ obj ^. objectiveGoal
+  Goal _gs obj -> displayParagraphs . map Markdown.toText $ obj ^. objectiveGoal


This needs to change to properly support the whole markdown text (with code) instead of a list of paragraphs.

I suppose it might make sense to use [[StreamNode]] as paragraphs so that we can add padding.

src/Data/Text/Markdown.hs

byorgey · 2023-02-14T16:43:02Z

src/Data/Text/Markdown.hs

+fromGeneralNode (CMark.Node p t cs) = case t of
+  CMark.DOCUMENT -> Node Document <$> mapM fromGeneralNode cs
+  CMark.PARAGRAPH -> Node Paragraph <$> mapM fromGeneralNode cs
+  CMark.SOFTBREAK -> leaf $ LeafText "\n" -- ?


According to https://github.github.com/gfm/#soft-line-breaks it seems like a SOFTBREAK is what we get at the end of a physical line of parsed text which does not end with two or more spaces. Typically we ignore such breaks and just reflow the text, so I would think we should not generate a LeafText "\n" here, unless I am misunderstanding something.

I think we might need to use a space instead of newline here.

Note to self: check that spacing is preserved.

byorgey · 2023-02-14T16:46:17Z

src/Data/Text/Markdown.hs

+    readTerm txt & \case
+      Left e -> Left (T.unpack e)
+      Right Nothing -> Left "empty code"
+      Right (Just s) -> case processParsedTerm s of


Yeah, we can make a parallel to readTerm for types, like readType = runParser (fully parseType). If readTerm fails you can just try running readType too.

src/Swarm/Game/Scenario/Objective.hs

May be able to reduce likelihood of #1086 by having a autogenerated table of the commands and the tutorials they are introduced in. This work is related to #1106 in that robot commands should be given distinct markup (backticks). ## Demo scripts/play.sh generate pedagogy **Updated:** See [rendered output](https://gist.github.com/kostmo/d4af3a6814e65fc6d455fd34b41552d3). ## New tests scripts/run-tests.sh --test-arguments '--pattern "Pedagogical soundness"' Note that the test for `crash.yaml` is currently failing because the `Give` command (and some others) are utilized in this tutorial without having been mentioned in the description or in earlier tutorials. Additionally, the "bug" described in [this comment](#1089 (comment)) **should** be caught by these tests (i.e. fail the tests) since the embedded `solution` code in the YAML file utilizes the `give` command. That is, if it currently weren't for [this issue](https://github.com/swarm-game/swarm/pull/1186/files#r1152789153).

- closes #309 - part of #545 and #574 Restyled by fourmolu (#1107) Co-authored-by: Restyled.io <commits@restyled.io>

Co-authored-by: Restyled.io <commits@restyled.io>

xsebek · 2023-08-03T01:14:11Z

Huzzah! 🥳

Turns out CommonMark has a neat extension that allows me to annotate inline code as entities/types/invalid code snippets.
Took me a while to rewrite, but at least I don't have to study a C library to figure out how it works. 😅

@kostmo If it is OK, I would switch from CMark-GFM to CommonMark. AFAIK it was only used in one place and the change was simple one-liner.

src/Swarm/Doc/Pedagogy.hs

xsebek · 2023-08-03T11:27:23Z

@byorgey @kostmo I rewrote the implementation from CMark-GFM to CommonMark and while it is a bit of a DIY project, I hope it will make reading descriptions a little easier. 🙂

For reference, the markdown highlighting conventions I went with are:

move - valid swarm code (the easy to write default)
wedge{=entity} - for swarm entities (I generated a regex to catch all standard ones, but they are not validated)
unit{=type} - for swarm types (there are not so many and I do not validate them yet)
- EDIT: this would make it easier to validate types as parsing alternatives would only report the last failure
require <a> <b>{=snippet} - raw snippets for invalid code
Alt-G - bold for keyboard shortcuts

If we want to do something else with any of those cases, they should be easy to grep and replace.

data/entities.yaml

src/Data/Text/Markdown.hs

xsebek · 2023-08-03T11:42:40Z

src/Swarm/Doc/Pedagogy.hs

-  txtLookups = M.fromList $ map (syntax . constInfo &&& id) commandConsts
+  allCode = concatMap findCode goalTextParagraphs
+  filterConst :: Syntax -> [Const]
+  filterConst sx = mapMaybe toConst $ universe (sx ^. sTerm)


@kostmo this is one of the little nice things a parsed code gives us. 🤩

Hopefully universe works just as well as your original implementation, but please check that I did not break it. 😅

src/Swarm/Game/Failure.hs

src/Swarm/TUI/View/Objective.hs

swarm.cabal

byorgey · 2023-08-04T21:54:46Z

Nice! I'm working on a detailed review, hopefully finished later today.

data/entities.yaml

data/scenarios/Challenges/Sliding Puzzles/3x3.yaml

data/scenarios/Tutorials/requireinv.yaml

src/Swarm/Game/Failure.hs

src/Data/Text/Markdown.hs

data/scenarios/Tutorials/craft.yaml

byorgey · 2023-08-05T03:21:12Z

src/Data/Text/Markdown.hs

+-- | Get chunks of nodes not exceeding length and broken at word boundary.
+--
+-- The split will end when no more nodes (then words) can fit or on 'ParagraphBreak'.
+chunksOf :: Int -> [StreamNode] -> [[StreamNode]]


It is a shame we seem to be reimplementing some of the functionality of https://hackage.haskell.org/package/word%2Dwrap ...

We are doing slightly more - we have annotated chunks of text, which we break while keeping the annotation.

Maybe this could be generalised as some kind of traversal?

Co-Authored-By: Brent Yorgey <byorgey@gmail.com> Co-authored-by: Brent Yorgey <byorgey@gmail.com>

Co-authored-by: Restyled.io <commits@restyled.io>

byorgey · 2023-08-05T17:08:04Z

🚀

byorgey · 2023-08-05T17:12:27Z

Hmm, it doesn't seem to be working for me. After pulling main and rebuilding, I get this:

xsebek · 2023-08-05T17:16:42Z

@byorgey odd, it looks like if you used old exe with new data. 😕 It worked fir me on the move tutorial, but I did not check alk of them.

I will test it with autoplay and add a script to go through all of them.

byorgey · 2023-08-05T17:55:43Z

Oh, I see, the goal dialog is formatted properly, but the formatting doesn't show up when you look at the description of an entity.

xsebek requested a review from byorgey February 13, 2023 14:22

restyled-io bot mentioned this pull request Feb 13, 2023

Restyle Parse markdown goal description #1107

Merged

xsebek commented Feb 13, 2023

View reviewed changes

xsebek changed the title ~~Parse markdown goal description~~ Parse markdown descriptions Feb 13, 2023

byorgey reviewed Feb 14, 2023

View reviewed changes

kostmo mentioned this pull request Apr 2, 2023

Dump a list of tutorials and commands they first introduce #1186

Merged

byorgey mentioned this pull request May 15, 2023

Highlight commands in descriptions #545

Closed

xsebek and others added 5 commits August 2, 2023 12:00

Parse markdown goal description

b396220

- closes #309 - part of #545 and #574 Restyled by fourmolu (#1107) Co-authored-by: Restyled.io <commits@restyled.io>

fixup

a42a219

fixup descriptions

dc04cfd

Move to commonmark

b8242ce

Fix tutorials using raw attributes

85c7d88

xsebek force-pushed the markdown-code branch from 961d67e to 85c7d88 Compare August 2, 2023 22:39

restyled-io bot mentioned this pull request Aug 2, 2023

Restyle Parse markdown descriptions #1402

Merged

xsebek and others added 3 commits August 3, 2023 03:05

Render code

23848b0

Fixup more scenario descriptions

2532a12

Restyled by fourmolu (#1402)

67b0309

Co-authored-by: Restyled.io <commits@restyled.io>

kostmo reviewed Aug 3, 2023

View reviewed changes

src/Swarm/Doc/Pedagogy.hs Outdated Show resolved Hide resolved

xsebek added 4 commits August 3, 2023 10:22

Lint

e6f12be

Restyle again

f678b0b

Fixup crash tutorial

ed7d03a

Put errors to stderr

95cb9e7

xsebek mentioned this pull request Aug 3, 2023

Parse, validate, and highlight code blocks #574

Open

xsebek added 2 commits August 3, 2023 13:15

Add docs

abd488e

Remove List.singleton for GHC8

6694171

xsebek marked this pull request as ready for review August 3, 2023 11:18

xsebek requested review from kostmo and byorgey August 3, 2023 11:18

xsebek commented Aug 3, 2023

View reviewed changes

xsebek added 4 commits August 4, 2023 21:23

Make shortcuts bold

49aa7ed

Mark code in tutorials

e53fc0d

Unify paragraph display

ff391d8

Add bounds

dc3a242

byorgey approved these changes Aug 5, 2023

View reviewed changes

xsebek and others added 7 commits August 5, 2023 16:07

Apply suggestions from code review

449a95a

Co-Authored-By: Brent Yorgey <byorgey@gmail.com> Co-authored-by: Brent Yorgey <byorgey@gmail.com>

Update entities descriptions markdown

9a1eb4c

Update data/entities.yaml

37e617b

Co-Authored-By: Brent Yorgey <byorgey@gmail.com> Co-authored-by: Brent Yorgey <byorgey@gmail.com>

Make drill a verb again

83006a2

Move Failure rendering to its declaration

d89fc58

Remove Failure.Render from cabal

8700e02

Move Markdown to Swarm

6248a41

restyled-io bot mentioned this pull request Aug 5, 2023

Restyle Parse markdown descriptions #1405

Merged

Restyled by fourmolu (#1405)

4f8b0da

Co-authored-by: Restyled.io <commits@restyled.io>

This was referenced Aug 5, 2023

Parse and validate swarm types in descriptions #1406

Open

Check entities in descriptions #1407

Open

xsebek added the merge me Trigger the merge process of the Pull request. label Aug 5, 2023

mergify bot merged commit 1eb2f9c into main Aug 5, 2023
10 checks passed

mergify bot deleted the markdown-code branch August 5, 2023 15:39

This was referenced Aug 5, 2023

Markdown formatting is not displayed properly in entity descriptions #1408

Closed

Words reflowed across a line break in objective text are missing a space between them #1409

Closed

kostmo mentioned this pull request Aug 6, 2023

add more markup to tutorial objectives prose #1412

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parse markdown descriptions #1106

Parse markdown descriptions #1106

xsebek commented Feb 13, 2023 •

edited

xsebek left a comment •

edited

xsebek Feb 13, 2023

byorgey Feb 14, 2023

xsebek Aug 3, 2023

xsebek Feb 13, 2023

byorgey Feb 14, 2023

xsebek Feb 15, 2023

xsebek Aug 3, 2023

byorgey Feb 14, 2023

xsebek commented Aug 3, 2023

xsebek commented Aug 3, 2023 •

edited

xsebek Aug 3, 2023

byorgey commented Aug 4, 2023

byorgey Aug 5, 2023

xsebek Aug 5, 2023

byorgey commented Aug 5, 2023

byorgey commented Aug 5, 2023

xsebek commented Aug 5, 2023

byorgey commented Aug 5, 2023

Parse markdown descriptions #1106

Parse markdown descriptions #1106

Conversation

xsebek commented Feb 13, 2023 • edited

xsebek left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xsebek commented Aug 3, 2023

xsebek commented Aug 3, 2023 • edited

Choose a reason for hiding this comment

byorgey commented Aug 4, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

byorgey commented Aug 5, 2023

byorgey commented Aug 5, 2023

xsebek commented Aug 5, 2023

byorgey commented Aug 5, 2023

xsebek commented Feb 13, 2023 •

edited

xsebek left a comment •

edited

xsebek commented Aug 3, 2023 •

edited