Revisit TextType to include Markdown

[cboettig]

This is probably out of scope for 2.2, and my apologies if it's already discussed in parts elsewhere that I've overlooked.

It seems like the use of partial doc-book markup in TextType is often falling in a funny middle ground where it's not complex enough to just treat as full doc-book, but otherwise is too complex for common tools. For instance, it appears that common rendering tools for EML, including I think the obvious reference implementation of the MetaCat web display aren't actually rendering all text-type elements in full but are stripping them out. Meanwhile, the structure appears to be often challenging for users to generate or manipulate easily (think word-counts, common words and other text mining operations).

See ropensci/EML#217 for more discussion.

From my limited experience so far here, it seems like it would be preferable to either adopt some rich-text format for which there is more comprehensive tool support (e.g. full docbook, or any other open, archival quality format) or just opt for something much simpler (sections & paras with plain-text content).

[mbjones]

I agree with you in principle, but this would be a backwards-incompatible change, and so would be out of scope for 2.2. If I had it to do today, it might make more sense to support markdown over docbook. Of course, in 2002 or so, docbook had a similar lock on the markup market as markdown does today, so in 10 to 15 years more we might very well have something else that is ascendant. RST came in between and was looking good for a while. Not sure how we predict the future of markup languages. Our bottom line is we need some simple formatting to support some emphasis text, some lists, subscript and superscript as these are common in scientific texts, and a few other things. But you are right that it is hard to just support these. Let's continue this discussion to figure out how to move forward.

[cboettig]

Yup, guess that's as I surmised. I'm curious what insight you have onto how metadata authors are generating TextType content in general though -- I don't image many people just type out abstracts and methods with the relevant tags.

As you've probably seen, in the R package we've attempted to suggest authors can write such long-form docs in markdown, html, or probably most popularly, MS Word, and the package will attempt to convert it into DocBook via pandoc. Obviously this is asking for trouble though, since full Docbook isn't supported. At very least the R package needs some mechanism to filter out invalid tags created this way...

I suppose moving in this direction with the EML spec, e.g. for full Docbook support, would be more backwards compatible (e.g. since it would add rather than remove tags), but I guess would open a larger can of worms (allbeit one that might be largely addressed by existing Docbook tooling).

I completely agree with you that markdown isn't the ideal here; it's great for content creation but only if being rendered into something else, it's far to fractured and loosely defined to be desirable here. (Docbook, being the kindred open XML format, is probably still the most natural choice).

In any event, I think any revision should of course be driven by understanding how users are creating these EML elements, (and also how they want to create these elements) and how they are consuming them (or would like to be able to consume them -- e.g. render a nice Scientific Data article from the relevant EML methods/protocols sections etc). Still probably all beyond what you want to tackle with 2.2

[mbjones]

Not necessarily out of scope. I'd note that #269 is intended to allow formatting and inline images that let us build the equivalent of Scientific Data articles in EML as a structured metadata language. I'm guessing that will require changes to TextType that are similar to what you are requesting. Maybe this becomes EML 3? Yikes!

[mbjones]

After some further contemplation, I have moved this into the EML 2.2 project with the intention of seeing if we can support the full docbook schema in eml:TextType, which would allow inline insertion of figures and tables and other needed formats for data papers. As these would be additions, this would not be a breaking change (all previously valid TextType elements would still be valid here).

Advantages:

• users can use any tool that generates valid docbook to both create and consume the contents of TextType (whereas now its hard to use the current format)
• rich features of Docbook would be available, including the use of inlinemediaobject to provide for in-text images and tables, which are a requirement to support data papers as described in issue add fields to support data papers #269

Disadvantages

• the TextType field is more complex, and so display software will need to be richer to properly display the contents

Overall, though, I think its a win. Commentary, @cboettig ?

[cboettig]

Nice, I think this is the way to go. Certainly it get's more complex to render, but it opens up all the docbook tooling to deal with that, which I think is a big win.

Just because it can be more complex doesn't necessarily mean that complexity will be widely used, which I think is already illustrated by the current textType complexity (e.g. display tools already don't support all textType fields and it's largely not a big problem).

Full docbook support in textType seems like it would appeal to journals with the data articles stuff from ESA etc, right?

[mobb]

sounds like an improvement. Another common use of text type is methods descriptions, which often include images and tables. Currently, the only option is to include these in an external doc (protocol PDF). We'd need good guidelines for when these objects should be in text and when they belong in the dataset entities.

[mbjones]

Thanks, @mobb. I was thinking that inline references would be to data entities in the package, as those are available. So, the steps for including, for example, a site layout figure in the methods would be to first include the figure as an otherEntity, and then reference it inline in the methods section using inlinemediaobject.

[mbjones]

@amoeba — now that you’ve implemented the XSL for TextType for EML 2.1.1, do you have further thoughts on the feasibility of extending TextType to support all (or most?) of docbook, as described in this issue #275? How much more work would be involved in supporting the display of the full docbook spec, and in particular the inlinemediaobject tag to render images and other media in the body of the text?

[amoeba]

Hey @mbjones. I think supporting full DocBook makes a lot of sense here. Certainly would be a big help for authoring EML from Word/md/etc via Pandoc. As far as updating our display systems to fully support all of DocBook in EML TextType modules, I don't expect it would be a ton of work. Supporting EML 2.1.1's TextType as I've recently done was less than a days work. There already exist official DocBook XML -> HTML5 XSLs (though I'm not sure if they'd be of use to us). If fully supporting DocBook rendering was too hard, supporting the rendering of a subset might work for our use case.

As a potential tangent, my recollection is that, for display systems, we've been considering moving away from an XSLT-based rendering process toward a fully client-side process. If we were to support full DocBook in EML TextType, this would be a pain because in-browser rendering facilities for DocBook appear non-existent while the ecosystem on the web for Markdown already exists.