Implement Markdown View #820

robyngit · 2018-11-02T18:25:19Z

The Markdown View will parse markdown contained within the XML associated with each project. This View must be flexible enough to be used as a component in other (non-project) parts of MetacatUI.

This is a subtask of DataONEorg/collections-portals-schemas#35

Added a markdown view with stubbed out methods, plus a markdown template. This view will retrieve and parse markdown. refs #820

refs: NCEAS/metacatui#726, NCEAS/metacatui#589, NCEAS/metacatui#820

robyngit · 2018-11-20T00:37:04Z

This issue is closely related to issues #589 and issue #726. I will record updates in this issue only from now on.

The list of markdown elements that we'd like to support are listed in this Javascript-Markdown-Library-Comparison spreadsheet.

To test a markdown to html converter, I created an example XML doc that includes all of these elements.

I used this xml doc to test the functionality of showdownjs. The resulting demonstration can be viewed in this JSfiddle.

Since showdownjs has almost all of the functionality we require, I will move forward with implementing this in the markdown view.

csjx · 2018-11-20T15:54:26Z

Awesome @robyngit - your fiddle looks like Showdown is really promising. 🤠

Added the most recent version (1.9.0) of the markdown to html converter, `showdown`, to the components directory and mapped its alias in `app.js`. Rendered the MarkdownView as a sub view in ProjectView in order to eventually preview the results of the showdown converter. refs #820, DataONEorg/collections-portals-schemas#35

mbjones · 2018-11-21T16:53:06Z

Fantastic, @robyngit. I really like how thorough your test and demo are. So I see that two major features are not in this demo: math and citations. I'm not clear as to why the math support is hard to test here. Can you explain? And how confident are you that showdown will be able to support those, or that we will be able to write a plugin that supports them? Note that the remark.js supports math through a plugin. Thanks for the clarifications.

csjx · 2018-11-21T16:59:53Z

@robyngit wrote:

Math should be possible using the showdown-katex extension. However, it's too complicated to demonstrate using this JSfiddle environment (no CDN available).

While JSFiddle doesn't provide a CDN-linked copy of the math or citation showdown JS libraries, you can install the library somewhere web-accessible and link directly to it in JSFiddle. Is that the only issue?

csjx · 2018-11-21T17:33:29Z

I also noticed that the DocBook tags don't seem to be supported (at least in the fiddle). I would expect the <para> tag to be translated into a <p> tag, but it looks like it is just passed straight through, which is expected (and then sanitized out by the HTML renderer).

So, we may need to do some pre-processing from DocBook to inline HTML so the markdown processor can then do the rest. I'm not seeing much for DocBook to HTML Javascript libraries out there (server side there is pandoc of course). Since EML only supports a small subset of DocBook tags, writing an XSLT stylesheet that does the conversion would be tractable, and could be applied in the browser as a pre-processing step.

robyngit · 2018-11-21T17:49:28Z

@csjx @mbjones: As Chris suggested, the main problem was a lack of CDN's. I was also getting a Cross-Origin Read Blocking error when trying to link directly to js files hosted elsewhere. (A chrome thing?) From reviewing the documentation for the math and citation extensions, and checking out issues on the respective github repos, I feel pretty confident that showdown will be able to support those features.

You're right, the <para> tags aren't converted, but they do remain in the html. I should have been more clear on that. I think it would also be possible to write a showdown extension that does the conversion to html. For example, see how the 'bindings' and 'codehighlight' extensions are created in the fiddle.

csjx · 2018-11-21T18:17:23Z

Ah, yes, a showdown extension for Docbook XML would be great.

For the non-CDN hosting, you would need to host the files on a web server that supports cross origin requests (CORS). dev.nceas.ucsb.edu, test.arcticdata.io, etc are set up that way. But like you said, showdown support looks probable, so I don't feel like the fiddle needs to be updated. Doing this work directly in MetacatUI should be fine.

robyngit · 2018-11-21T18:23:26Z

Great! Then I'll continue moving forward with getting showdown + extensions working in metacatUI. :)

Added markdown to the example XML that we'll use to test showdown. The new markdown will be used to test showdown's citation, footnote, and math extensions. refs NCEAS/metacatui#820

robyngit · 2018-11-21T20:18:53Z

For reference: The example project XML that includes all the complex markdown is available on dev. knb. Its id is urn:uuid:5a26fb11-e0a1-4e3f-976e-1090f24089c0.

…View This is the first step in setting up the markdown view. The next step is using showdown to convert the markdown to HTML. I added a listener to projectView so that the subviews render after the model syncs, since information from the model is required for all the subviews. This causes a delay before the subviews are rendered, so perhaps there is a better way to achieve this. I added all the subviews to the subviews array and used _.invoke to have them all render. refs #815, #820

Got the basic functionality of showdown working in the markdown view. Extensions still need to be added. refs #820

robyngit · 2018-11-21T23:19:39Z

Showdown is now working in MarkdownView! 🎉 Just need to add in the extensions. This can be previewed with all the markdown elements we will support here:http://<YOUR-LOCAL-METACAT-UI-ADDRESS>/#project/urn:uuid:5a26fb11-e0a1-4e3f-976e-1090f24089c0.

Extensions to add:

Added the highlight.js components and created a highlight-showdown extension in markdownView relates to #820

Added the katex extension to the showdown converter in MarkdownView so block and inline math is now converted to html. The katex extension config could use a little tweaking. Also relocated the showdown extension components into a single directory. related to #820

Added all of the docbook tags that are allowed in EML relates to: NCEAS/metacatui#820

Added footnotes, bootstrap, and docbook extensions to the markdown view. Added a test that checks the markdown and only adds each extension when required. Extensions are still loaded in the top define function though, need to figure out how to properly load them as required. Also need to include more bootstrap classes in the extension (so far it only adds a class to tables). relates to #820 "showdownKatex", "showdownHighlight", "showdownFootnotes", "showdownBootstrap", "showdownDocbook", "showdownCitation",

For the markdown view, added more scientific languages to highlightJS (used by the showdown converter). Languages included in the package can be viewed in the README in the showdown-highlight component folder. relates to #820

Used require() to load showdown extensions as needed. The showdown converter is triggered once all extensions are tested for a loaded if required. relates to #820

Added a showdown xss filter extension that mitigates the risk of some cross-site scripting attacks. relates to #820

Created an extension that connects showdown to citationjs. The extension is now working in the markdownView. Still needs some tweaking, including formatting of the bibliography. relates to #820

robyngit · 2018-12-12T19:15:40Z

A note: The citation extension retrieves some data from doi.org. Because of a metacat bug in how metadata is submitted when we register a doi, author information for any of our cited datasets may be incorrect or look pretty strange (e.g. author shows up as DC=Dataone CN=SASAP for doi 10.5063/f1r78cf3). Once this issue is resolved, dataset citations should look correct.

Added formatting to the rendered citations, and fixed the bibliography so that there are no duplicate entries and all entries are sorted. relates to #820

This way it will only show up on the home tab instead of all project tabs. #820

relates to: #820

…lex markdown example xml. relates to #3; NCEAS/metacatui#820

The new extension replaces all header tags with a header tag one size smaller (e.g. <h1> --> <h2>). This is to prevent top level header tags in markdown views. relates to #820

Started a readme that gives some guidance on how the showdown extension and markdown view work. relates to #820

robyngit · 2018-12-21T20:55:00Z

citations in markdown

The last issue to resolve in the markdown view is getting inline citations and bibliographies to work as we would like.

First attempt

Initially, I made an extension that searched for and extracted any DOI, WikidataID, or other input, contained within square brackets like so:

^[doi:xxxxxxxx]

The extension then took this ID and used citation.js to lookup the associated citation information. It then created an appropriate in-line citation in place of the ^[doi:...], and appended a bibliography to the end of the markdown view.

Issues with this solution

There was a noticeable delay in rendering the view, even with only three citations to retrieve. Showdown does not currently support async filters, although this is a pending enhancement
More importantly, we prefer the bibliographic information to be stored in the project xml for long-term archiving purposes.

New strategy

Bibtex style citation information will be stored in the project xml (and thus in the project model). This information will be passed to the citation showdown extension. The extension should search for bibtex keys in square brackets, like so...

[@darwin_1859]

...and match them to the citation information contained within the model. It should use this information to create an in-line citation. The literature cited/bibliography will be created in a separate view.

Progress

update project schema to add EML 2.2.0 CitationListType element (DataONEorg/collections-portals-schemas@15c66e8)
add example bibtex information to one of the example xml documents (DataONEorg/collections-portals-schemas@15c66e8)
parse the bibtex information in the project model (07c2d71)
modify the showdown-citation extension so that it detects bibtex keys and uses information in the model to create inline citations.

I'm now parsing the citations (e.g., [@jones_2001]) out of the markdown and displaying them without modification if they're not found by citations.js. I still need to figure out why they're not being found by citations.js and get that part going. #820 #870

laurenwalker · 2019-02-14T17:20:57Z

I'm gonna close this issue since we have the remaining task detailed in #870.

robyngit added projectsUI-MVP Cat: projects-UI Projects-Papers labels Nov 2, 2018

laurenwalker mentioned this issue Nov 2, 2018

Implement project page view DataONEorg/collections-portals-schemas#35

Closed

12 tasks

robyngit added a commit that referenced this issue Nov 2, 2018

Stubbed out the markdown view

c6376eb

Added a markdown view with stubbed out methods, plus a markdown template. This view will retrieve and parse markdown. refs #820

robyngit added a commit to DataONEorg/collections-portals-schemas that referenced this issue Nov 19, 2018

Adding new example xml doc to test showdown's features

8a42fc2

refs: NCEAS/metacatui#726, NCEAS/metacatui#589, NCEAS/metacatui#820

robyngit self-assigned this Nov 21, 2018

robyngit added the in progress label Nov 21, 2018

robyngit added a commit that referenced this issue Nov 21, 2018

Converted markdown to html in markdownView

9367595

Got the basic functionality of showdown working in the markdown view. Extensions still need to be added. refs #820

This was referenced Nov 21, 2018

parse Markdown blocks for inclusion in views #726

Closed

Build markdown processor #589

Closed

robyngit added a commit that referenced this issue Nov 27, 2018

Added syntax highlight extension to the MarkdownView

b710463

Added the highlight.js components and created a highlight-showdown extension in markdownView relates to #820

robyngit added a commit to DataONEorg/collections-portals-schemas that referenced this issue Nov 29, 2018

Add to the markdown example xml to test docbook tag conversion

7dafa61

Added all of the docbook tags that are allowed in EML relates to: NCEAS/metacatui#820

robyngit added a commit that referenced this issue Dec 6, 2018

Load showdown extensions as required in markdownView

6fcbcad

Used require() to load showdown extensions as needed. The showdown converter is triggered once all extensions are tested for a loaded if required. relates to #820

robyngit added a commit that referenced this issue Dec 11, 2018

Add showdown xss filter extension

53439d0

Added a showdown xss filter extension that mitigates the risk of some cross-site scripting attacks. relates to #820

robyngit added a commit that referenced this issue Dec 13, 2018

For showdown citation extension: add formatting, fix bibliography

9eb1fc7

Added formatting to the rendered citations, and fixed the bibliography so that there are no duplicate entries and all entries are sorted. relates to #820

jkibele added a commit that referenced this issue Dec 13, 2018

Moved markdown rendering from ProjectSectionView to ProjectHomeView.

5db0ff1

This way it will only show up on the home tab instead of all project tabs. #820

robyngit added a commit that referenced this issue Dec 14, 2018

Added onClose behaviour to markdownView

43a0e77

relates to: #820

robyngit mentioned this issue Dec 18, 2018

Review showdown's XSS filter in the markdown view #860

Closed

robyngit added a commit to DataONEorg/collections-portals-schemas that referenced this issue Dec 19, 2018

Added literature cited element to schema + example bibtex to the comp…

15c66e8

…lex markdown example xml. relates to #3; NCEAS/metacatui#820

robyngit added a commit that referenced this issue Dec 21, 2018

Add a readme in the showdown directory

b322f30

Started a readme that gives some guidance on how the showdown extension and markdown view work. relates to #820

jkibele self-assigned this Feb 12, 2019

jkibele mentioned this issue Feb 14, 2019

Render citations and a reference list in the MarkdownView and project home tab #870

Closed

laurenwalker closed this as completed Feb 14, 2019

laurenwalker removed the in progress label Feb 14, 2019

laurenwalker added this to the 2.6.0 milestone Mar 4, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Markdown View #820

Implement Markdown View #820

robyngit commented Nov 2, 2018

robyngit commented Nov 20, 2018

csjx commented Nov 20, 2018

mbjones commented Nov 21, 2018

csjx commented Nov 21, 2018

csjx commented Nov 21, 2018

robyngit commented Nov 21, 2018 •

edited

Loading

csjx commented Nov 21, 2018

robyngit commented Nov 21, 2018

robyngit commented Nov 21, 2018

robyngit commented Nov 21, 2018 •

edited

Loading

robyngit commented Dec 12, 2018 •

edited

Loading

robyngit commented Dec 21, 2018 •

edited

Loading

laurenwalker commented Feb 14, 2019

Implement Markdown View #820

Implement Markdown View #820

Comments

robyngit commented Nov 2, 2018

robyngit commented Nov 20, 2018

csjx commented Nov 20, 2018

mbjones commented Nov 21, 2018

csjx commented Nov 21, 2018

csjx commented Nov 21, 2018

robyngit commented Nov 21, 2018 • edited Loading

csjx commented Nov 21, 2018

robyngit commented Nov 21, 2018

robyngit commented Nov 21, 2018

robyngit commented Nov 21, 2018 • edited Loading

robyngit commented Dec 12, 2018 • edited Loading

robyngit commented Dec 21, 2018 • edited Loading

citations in markdown

laurenwalker commented Feb 14, 2019

robyngit commented Nov 21, 2018 •

edited

Loading

robyngit commented Nov 21, 2018 •

edited

Loading

robyngit commented Dec 12, 2018 •

edited

Loading

robyngit commented Dec 21, 2018 •

edited

Loading