An experimental Markdown edition #91
Comments
|
After creating the
For (2) and (3) I tried out the following tools - Mou, MacDown, Visual Studio Code and GitHub. Something else that occurred to me is that collaborative editing and commenting were desired when working on the Versioning Scheme document. I am not sure if this is possible yet, but we could consider switching from DocBook to DITA and asking EasyDita to grant a free license for editing the eXist-db documentation perhaps? That might be a nice dogfood'ing solution as EasyDita uses eXist-db. |
|
We would be happy to provide this to the eXist-db community (easyDITA) if there is interest. Let us know if there is anything we can do to help. Cheers! |
|
@adamretter GitHub-Flavored Markdown (GFM) would be the pragmatic choice of flavor given the proposed benefits of the GitHub-based ecosystem for lowering barriers to contributing. I've used Marked (see its documentation on GFM support) and recently oXygen 18.1 for decent local previews of GFM. Pandoc excels at converting Markdown to other formats - PDF, Word, etc. @caseydawsonjordan Wow, that's great, thanks for the offer to the community! I've given some thought to migrating eXist's documentation to DITA too (my experiments are here: |
|
I need to learn about DITA :-) |
|
from a user perspective I really like mbostocks's approach to documenting d3.js. Word and pdf support seem a red-herring, getting the code-base and documentation better aligned, and potentially version-specific are the big ones to me. |
|
@joewiz Just a small sidenote, DocBook also uses topics allowing different paths through the documentation without duplication of contents. :) |
|
Joe,
Like anything with XML there is a lot of ways to do a conversion from
DocBook to DITA, the biggest thing is just creating a strategy for
splitting content up into topics that makes sense for the community. I
believe that there are some automated tools to do this, but we honestly
don't do much conversion between DocBook and DITA.
Markdown certainly has the lowest barriers for contribution, however the
issue is that markdown is very hard to convert to other formats if you need
to in the future or if you need to produce different versions of the
documentation (IE Quickstart guide, Security Guide, Full Guide, etc) while
keeping everything in sync. So if you go down the markdown path, if you
ever need to change to something else in the future, it may be a very
painful process. With DITA or other XML formats, you can automate all those
conversions or anything else you want. As well all know, XML is a powerful
thing, be careful ditching it...
Also, I'll mention that DITA is a lot easier to search, especially in eXist
because you have semantic XML to work with.
There are a lot more things to consider, but these are just a few that I
will mention here. Markdown could be a great choice but it comes with some
long term problems you should consider in depth before you make a decision
on a format.
If you guys are having any upcoming telecons about this I would be happy to
join to chat.
Cheers,
Casey
…On Wed, Mar 8, 2017 at 11:18 AM ljo ***@***.***> wrote:
@joewiz <https://github.com/joewiz> Just a small sidenote, DocBook also
uses topics allowing different paths through the documentation without
duplication of contents. :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#91 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAgl3a33t3ucGVgM7d8iwCfdPopfJVgwks5rjtS-gaJpZM4MPTdu>
.
|
|
@duncdrum Beautiful work! I especially like how you dropped anchors to each function, e.g., https://github.com/duncdrum/cbdb-data/blob/nuseir/doc/function-doc.md#alias and how there's so much interlinking and cross-referencing within the page. This looks really useful! Thank you for sharing it! |
|
In my "markdown" branch I've just committed some new features:
The biggest problem I see with the app is that 7 articles have parse failures. I've listed these in the commit message. I was able to find some symptoms but not the cause: it seems to have to do with fenced code block parsing. For example, when trimming all but the java example from https://github.com/joewiz/documentation/blob/markdown/data/triggers.md#java, the page would trigger the error; removing these java blocks would get rid of the error. In addition, many fenced code blocks have trouble with syntax highlighting. Old (left) vs. new (right): I see two alternatives to this seemingly "showstopper" markdown parsing issue:
Thoughts? |
|
@joewiz very nice. I m SIB but I ll have a look when I m up and running. The fenced codeblocks seem fixable with some white space trickery. As for the unique headings we could switch to alternating numeral + alphabetic. So ''4.2.1.4'' -> ''4b1d''. The dots in the heading number are prone to cause problems, but there are other workarounds. |
|
@joewiz I m not sure the markdown parser is to blame. Looking through your commit history, the initial output from pandoc seems poorer then I would have expected. No wonder the parser tripped. Still switching to ex-markdown might be a smart move. What's the goal here? Find out why pandoc produced poor output (and submit PR's to them)? Change the docbook so the conversion becomes easier? or improve the markdown that is there now? |
|
@duncdrum I don't disagree about the quality of the pandoc output - the markdown needs work and I'm open to either approach of tweaking pandoc or reformatting what we have - but the parser does trip up on a perfectly valid fenced code block, excerpted from https://github.com/joewiz/documentation/blob/markdown/data/triggers.md#java lines 265-420 (notice that in that link github is able to parse the java example). Here is a test showing the error:
I should post this as a bug to https://github.com/wolfgangmm/exist-markdown, but there is the larger question: do we want to take on the task of "supporting" GitHub Flavored Markdown via an XQuery library when there is a Java library that does the task that could be made into an XQuery module? Also, an XQuery module for https://github.com/vsch/flexmark-java like https://github.com/expkg-zone58/ex-markdown would support more than just GFM but the CommonMark umbrella format (which GitHub has committed to; see https://githubengineering.com/a-formal-spec-for-github-markdown/). In addition, check out the CommonMark AST generated by FlexMark-Java - an XML representation of the source as CommonMark sees it. Wouldn't you rather work with this (the result of pasting test.md into http://spec.commonmark.org/dingus/ and selecting the AST tab) and work on transforming this rather than debugging errors like the above each time we trip up on some weird Markdown? <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE document SYSTEM "CommonMark.dtd">
<document xmlns="http://commonmark.org/xml/1.0">
<heading level="1">
<text>Java</text>
</heading>
<code_block info="java">
import java.io.File;
import java.io.FileOutputStream;
import org.exist.collections.triggers.FilteringTrigger;
import org.exist.collections.triggers.TriggerException;
import org.exist.dom.DocumentImpl;
import org.exist.storage.DBBroker;
import org.exist.storage.txn.Txn;
import org.exist.xmldb.XmldbURI;
import org.exist.xquery.value.DateTimeValue;
/**
A simple Java Trigger that
logs all trigger events for which it is executed
in the file triggersLog.xml in the systems temporary folder
*/
public class LoggingTrigger extends FilteringTrigger implements DocumentTrigger
{
private final static String TEMPLATE = "<?xml version=\"1.0\"?><events></events>";
private DocumentImpl doc;
public void configure(DBBroker broker, org.exist.collections.Collection parent, Map<String, List<?>> parameters) throws CollectionConfigurationException {
super.configure(broker, parent, parameters);
XmldbURI docPath = XmldbURI.create("messages.xml");
System.out.println("TestTrigger prepares");
this.doc = parent.getDocument(broker, docPath);
if (this.doc == null) {
TransactionManager transactMgr = broker.getBrokerPool().getTransactionManager();
Txn transaction = transactMgr.beginTransaction();
try {
getLogger().debug("creating new file for collection contents");
// IMPORTANT: temporarily disable triggers on the collection.
// We would end up in infinite recursion if we don't do that
parent.setTriggersEnabled(false);
IndexInfo info = parent.validateXMLResource(transaction, broker, docPath, TEMPLATE);
parent.store(transaction, broker, info, TEMPLATE, false);
this.doc = info.getDocument();
transactMgr.commit(transaction);
} catch (Exception e) {
transactMgr.abort(transaction);
throw new CollectionConfigurationException(e.getMessage(), e);
} finally {
parent.setTriggersEnabled(true);
}
}
}
@Deprecated
public void prepare(int event, DBBroker broker, Txn transaction, XmldbURI documentPath, DocumentImpl existingDocument) throws TriggerException {
}
@Deprecated
public void finish(int event, DBBroker broker, Txn transaction, XmldbURI documentPath, DocumentImpl document) {
}
private void addRecord(DBBroker broker, String xupdate) throws TriggerException {
MutableDocumentSet docs = new DefaultDocumentSet();
docs.add(doc);
try {
// IMPORTANT: temporarily disable triggers on the collection.
// We would end up in infinite recursion if we don't do that
getCollection().setTriggersEnabled(false);
// create the XUpdate processor
XUpdateProcessor processor = new XUpdateProcessor(broker, docs, AccessContext.TRIGGER);
// process the XUpdate
Modification modifications[] = processor.parse(new InputSource(new StringReader(xupdate)));
for (int i = 0; i < modifications.length; i++)
modifications[i].process(null);
broker.flush();
} catch (Exception e) {
e.printStackTrace();
throw new TriggerException(e.getMessage(), e);
} finally {
// IMPORTANT: reenable trigger processing for the collection.
getCollection().setTriggersEnabled(true);
}
}
@Override
public void beforeCreateDocument(DBBroker broker, Txn transaction, XmldbURI uri) throws TriggerException {
String xupdate = "<?xml version=\"1.0\"?>" +
"<xu:modifications version=\"1.0\" xmlns:xu=\"" + XUpdateProcessor.XUPDATE_NS + "\">" +
" <xu:append select='/events'>" +
" <xu:element name='event'>" +
" <xu:attribute name='id'>STORE-DOCUMENT</xu:attribute>" +
" <xu:attribute name='collection'>" + doc.getCollection().getURI() + "</xu:attribute>" +
" </xu:element>" +
" </xu:append>" +
"</xu:modifications>";
addRecord(broker, xupdate);
}
@Override
public void afterCreateDocument(DBBroker broker, Txn transaction, DocumentImpl document) {
//ignore this event
}
@Override
public void beforeUpdateDocument(DBBroker broker, Txn transaction, DocumentImpl document) throws TriggerException {
//ignore this event
}
@Override
public void afterUpdateDocument(DBBroker broker, Txn transaction, DocumentImpl document) {
//ignore this event
}
@Override
public void beforeCopyDocument(DBBroker broker, Txn transaction, DocumentImpl document, XmldbURI newUri) throws TriggerException {
//ignore this event
}
@Override
public void afterCopyDocument(DBBroker broker, Txn transaction, DocumentImpl document, XmldbURI newUri) {
//ignore this event
}
@Override
public void beforeMoveDocument(DBBroker broker, Txn transaction, DocumentImpl document, XmldbURI newUri) throws TriggerException {
//ignore this event
}
@Override
public void afterMoveDocument(DBBroker broker, Txn transaction, DocumentImpl document, XmldbURI newUri) {
//ignore this event
}
@Override
public void beforeDeleteDocument(DBBroker broker, Txn transaction, DocumentImpl document) throws TriggerException {
String xupdate = "<?xml version=\"1.0\"?>" +
"<xu:modifications version=\"1.0\" xmlns:xu=\"" + XUpdateProcessor.XUPDATE_NS + "\">" +
" <xu:append select='/events'>" +
" <xu:element name='event'>" +
" <xu:attribute name='id'>REMOVE-DOCUMENT</xu:attribute>" +
" <xu:attribute name='collection'>" + doc.getCollection().getURI() + "</xu:attribute>" +
" </xu:element>" +
" </xu:append>" +
"</xu:modifications>";
addRecord(broker, xupdate);
}
@Override
public void afterDeleteDocument(DBBroker broker, Txn transaction, XmldbURI uri) {
}
}
</code_block>
</document> |
|
Also, from the screenshot of the rendering of xquery.md above, compare the results of rendering this: # XQuery 3.0 Support
eXist-db implements the following features of the ["XQuery 3.0"](http://www.w3.org/TR/xquery-30/) Working Draft
- Higher Order Functions: eXist-db completely supports higher-order functions, including features like inline functions, closures, and partial function application. For more information, see the article on the eXist-db blog, [Higher-Order Functions in XQuery 3.0](http://atomic.exist-db.org/blogs/eXist/HoF)
- Group by clause in FLWOR expressions: "group by" provides an efficient way to group the sequences generated in a FLWOR expression. For example,
``` xquery
xquery version "3.0";
for $speechBySpeaker in //SPEECH[ft:query(., "king")]
group by $speaker := $speechBySpeaker/SPEAKER
order by $speaker
return
<speaker name="{$speaker}">
{ $speechBySpeaker }
</speaker>
```
queries the Shakespeare plays and groups the result by speaker.exist-markdown <body>
<section>
<h1 id="xquery-3-0-support">XQuery 3.0 Support</h1>
<p>eXist-db implements the following features of the <a href="http://www.w3.org/TR/xquery-30/">"XQuery 3.0"</a> Working Draft</p>
<ul>
<li>Higher Order Functions: eXist-db completely supports higher-order functions, including features like inline functions, closures, and partial function application. For more information, see the article on the eXist-db blog, <a href="http://atomic.exist-db.org/blogs/eXist/HoF">Higher-Order Functions in XQuery 3.0</a>
</li>
<li>Group by clause in FLWOR expressions: "group by" provides an efficient way to group the sequences generated in a FLWOR expression. For example,</li>
</ul>
<p>``` xquery xquery version "3.0"; for $speechBySpeaker in //SPEECH[ft:query(., "king")] group by $speaker := $speechBySpeaker/SPEAKER order by $speaker return <speaker name="<span itemprop="$speaker">$speaker</span>"> <span itemprop=" $speechBySpeaker "> $speechBySpeaker </span> </speaker> ```</p>
<p>queries the Shakespeare plays and groups the result by speaker.</p>
</section>
</body>commonmark ast <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE document SYSTEM "CommonMark.dtd">
<document xmlns="http://commonmark.org/xml/1.0">
<heading level="1">
<text>XQuery 3.0 Support</text>
</heading>
<paragraph>
<text>eXist-db implements the following features of the </text>
<link destination="http://www.w3.org/TR/xquery-30/" title="">
<text>"</text>
<text>XQuery 3.0</text>
<text>"</text>
</link>
<text> Working Draft</text>
</paragraph>
<list type="bullet" tight="false">
<item>
<paragraph>
<text>Higher Order Functions: eXist-db completely supports higher-order functions, including features like inline functions, closures, and partial function application. For more information, see the article on the eXist-db blog, </text>
<link destination="http://atomic.exist-db.org/blogs/eXist/HoF" title="">
<text>Higher-Order Functions in XQuery 3.0</text>
</link>
</paragraph>
</item>
<item>
<paragraph>
<text>Group by clause in FLWOR expressions: </text>
<text>"</text>
<text>group by</text>
<text>"</text>
<text> provides an efficient way to group the sequences generated in a FLWOR expression. For example,</text>
</paragraph>
<code_block info="xquery">xquery version "3.0";
for $speechBySpeaker in //SPEECH[ft:query(., "king")]
group by $speaker := $speechBySpeaker/SPEAKER
order by $speaker
return
<speaker name="{$speaker}">
{ $speechBySpeaker }
</speaker>
</code_block>
<paragraph>
<text>queries the Shakespeare plays and groups the result by speaker.</text>
</paragraph>
</item>
</list>
</document> |
|
yes I see, the indenting (and sha 2dc7957) is really pushing things here on GH and also in the repo, lots of code blocks starting with like 8 whitespaces on the first line. Just a note according to this there shouldn't be a white space behind the code block ticks. here is my 2ct:
|
Sorry, let me make sure I understand. Are you saying there shouldn't be any spaces before the triple-backtick starting a fenced code block, as in the case above where the XQuery is indented inside the list? I think the spec link you included speaks to this though:
|
|
No I meant the infostring after the triple-ticks comes straight after the third tick. No ws. The comment about 8 ws was about some files in the repo having codeblocks like this. <root>
<a>blah<\a>
<\root> |
|
Ah, okay, I see. (I read "behind" as meaning "before", but I can see how you meant "behind" as "following"! Ah, spatial metaphors in language.) I don't think whitespace following the code block ticks is a problem, as the CommonMark spec about fenced code blocks says this:
So any leading or trailing spaces in the text following the opening code fence is trimmed. So it isn't wrong to include whitespace - it'll just be trimmed off by the commonmark processor. That said, this would certainly be something I'd clean up if we were fixing the pandoc output. |
|
For yet another option, we might consider Readthedocs (https://readthedocs.org/), used by many projects such as Mongodb (see live version at https://docs.mongodb.com/manual/ and source at https://github.com/mongodb/docs/blob/master/source/index.txt). Benefits of readthedocs:
(I think better versioning support would be great - so eXist users know which version of eXist the docs are targeting.) Pandoc can convert our Docbook to reStructuredText (the preferred format of Readthedocs) just as easily as it can output Markdown. Perhaps we should dip our toes in with a test, using Adam's much cleaned-up source in #135? Just another idea in advance of our discussion about documentation on Monday (https://docs.google.com/document/d/1aKBHnrYUQnMy2l2b8WUNg1SoN8nN3iLrgRKfggAiIQI/edit?usp=sharing). |
|
@joewiz If we are moving away from XML for our documentation, then I think this is an excellent idea. The documentation attempts at the past always focused on the technology and not the content and git nowhere. If we want to use reStructuredText to enable more authoring of content, than I support that. For delivery, I would like to see us write no code that we have to support, so I would welcome something like readthedocs. |
|
In preparation for my agenda item under the Documentation discussion for Monday's Community Call, I've created a new experimental edition of the eXist documentation, this time as a GitHub Wiki. Check it out here: https://github.com/joewiz/exist/wiki. Pros of GitHub Wiki:
Cons:
Notes on my experimental version:
Comments, questions, concerns welcome - either here and/or on Monday's call. |
|
@joewiz wow, i like that approach very much. And like to throw in my point of view: i'm very much for using something like the Github Wiki, Markdown or reStructed for their simplicity. The most prominent goal IMO must be to ease the process of contributing to the docs. If i don't need to fiddle around with new tools and languages that will very much increase my motivation to contribute content. That's of course a pragmatic decision - if we'd use the Github Wiki we would even have the option of adding other documents (e.g. pdf or so) for content with higher layout needs if that'll ever occur. For me the most important poiints are:
Because of this last point we here internally once discussed an eXist-db app that pulls the repo, converts to some sensible XML and indexes it for a better search interface. The results can even link back to the repo/wiki page. This essentially means to use Github for editing and versioning/workflow and a search frontend and rendering app. Don't know how much room there is but if the Wiki produces HTML then additional CSS styling could help a lot to make more attractive and readable. |
|
Since there are no plans in the foreseeable future to move to markdown, i m going to close this. Thanks for the discussion everyone, we might reopen this if we ever fiddle around with it again. |
One of the ideas kicking around for making eXist's documentation easier to contribute to and maintain is to turn it into Markdown. For recent discussion of this, see #34.
I've started a proof of concept at #90. You can clone my
markdownbranch at https://github.com/joewiz/documentation.git, and try browsing through the Markdown in your web browser at https://github.com/joewiz/documentation/blob/markdown/data/documentation.md. It's a very early step. It's just an experiment to help us think through whether this route seems useful to pursue further.But besides actual code experiments, this issue is about the general topic of pros and cons of moving to Markdown. I'll start us off:
Pros:
Cons:
Ultimately, moving from DocBook to Markdown is just technological musical chairs. It doesn't magically improve the content by itself. That's where the real work is needed. The question is: Is this a better starting point from which to begin improving the content? And will it encourage enough contributions to be worth the effort?
The text was updated successfully, but these errors were encountered: