Add AsciiDoc Reader / AsciiDoc input support

Greetings,

I would like to hereby suggest the addition of AsciiDoc input resp. an AsciiDoc Reader.

Besides Markdown, this format is growing in popularity, also in use inside a publishing toolchain (asciidoc -> docbook -> pdf/epub/html). Currently the only other viable implementation is asciidoctor, which uses Ruby or JRuby, but it is AsciiDoc-only in its input format and not a universal markup converter, like pandoc.

I am aware of only one relevant discussion thread regarding this, which showed positive echo for this feature. Someone actually had some basic code there, but I am not sure if kuznero resp. Roman Kuznetsov still has his code from back then to start from, but anyway, it would certainly make sense to have AsciiDoc input in the feature set.

I have started working on this, it might not be ready for a few months.

Have you tried going AsciiDoc -> DocBook -> Pandoc? Can you describe the shortcomings of doing this if you have?

Thanks for your positive comment!

I personally have not tried the conversion chain as you mentioned. The current choices are bulky with regards to its dependencies, while not being and not aspiring to be universal markup translators, like pandoc. Going through intermediate formats instead of one pandoc invocation seems hacky to me.

Recently, I read about two publishing houses switching away from LaTeX and moving to AsciiDoc as their source format, so I gather that it fulfills the needs of technical writing well, also regarding referencing and I find it useful to have another capable plain-text format for documents, if software support is good and can easily convert between different formats and offers multiple output choices. Which is where this feature would come in ;-)

I personally would also like to write articles and possibly a book rather in AsciiDoc than LaTeX, which is - at least my personal - motivation for this feature.

👍 I would love to see support for reading asciidoc files in pandoc. It was actually my primary reason to use pandoc. At the moment i have a bit of a hacky solution converting my asciidoc into html and then feeding that into pandoc. I would love to eliminate the extra project dependency and simplify my build chain.

I can testify that this would be a very useful addition to pandoc, thanks in advance mr. mpickering!

@mpickering

PR #2100 contributes a basic AsciiDoc reader (with many features not yet implemented).
@mpickering, how far did you get on your AsciiDoc reader? Is it farther along than #2100, or not as far along? It would be good to put something in the repository (in a branch) for people to work on to advance the project.

I commented on the #2100

I agree to that asciidoc is getting popular. I've just tried to convert the book Pro Git 2 into epub using pandoc and soon noticed that the book was coded in asciidoc, and pandoc was unable to read it.

I’ve tried an asciidoc > html (via https://github.com/asciidoctor/asciidoctor) > epub (via pandoc) conversion chain and it works extremely well except for the following issue.

Asciidoctor wraps all HTML elements in divs with additional classes. This stops pandoc from splitting the epub automatically at headings, because it will never see a 'naked' h1 etc.

Also mentioned this issue here: asciidoctor/asciidoctor#184

The ePub file I created using this method did not pass epubcheck 3.0.1 because it contained one duplicate ID (something I could have avoided). More seriously, the way footnotes are handled is not compliant, and raised several errors like this:

ERROR: …/EPUB.epub/ch003.xhtml(2918,1223): '_footnote_4': fragment identifier is not defined in 'ch003.xhtml'
ERROR: …/EPUB.epub/ch008.xhtml(2065,27): '_footnoteref_1': fragment identifier is not defined in 'ch008.xhtml'

(paths truncated with ellipsis at start)

At the risk of stating the obvious, it's important that pandoc-generated epubs from any source format avoid epubcheck validation errors, as authors and publishers may need to submit these epubs to storefronts that will require epubcheck compliance (i.e. Smashwords, iBooks). Many of the Github-hosted CLI epub generators I've tried (e.g. https://github.com/avdgaag/rpub) omit consideration of epubcheck compliance, so this may not be an obvious point after all. The most common point of failure seems to be the manifest, which pandoc does correctly, which is great. But it could be yet more robust, as the above errors indicate.

+++ Ben Hourigan [Jul 05 15 08:56 ]:

I’ve tried an asciidoc > html (via
[1]https://github.com/asciidoctor/asciidoctor) > epub (via pandoc)
conversion chain and it works extremely well except for the following
issue.

Asciidoctor wraps all HTML elements in divs with additional classes.
This stops pandoc from splitting the epub automatically at headings,
because it will never see a 'naked' h1 etc.

Also discussed this issue here: [2]asciidoctor/asciidoctor#184

You could handle this easily with a filter that strips out the
outer Divs before the EPUB writer sees it.

The ePub file I created using this method did not pass epubcheck 3.0.1
because it contained one duplicate ID (something I could have avoided)
and, more seriously, the way footnotes are handled is not compliant,
and raised several errors like this:
ERROR: …/EPUB.epub/ch003.xhtml(2918,1223): '_footnote_4': fragment identifier is
not defined in 'ch003.xhtml'
ERROR: …/EPUB.epub/ch008.xhtml(2065,27): '_footnoteref_1': fragment identifier i
s not defined in 'ch008.xhtml'

When I convert pandoc's README to epub3, I see no errors
with epubcheck 3.0.1 (and README has several footnotes).

My guess is that the HTML footnotes produced by asciidoctor
are not read by pandoc as native pandoc footnotes, and that
is the underlying issue.

If you attach a short sample file (of HTML produced by
asciidoctor), we could confirm that.

Unfortunately, there's no standard way of doing footnotes
in HTML, so the HTML reader never produces a Note element.

Hope this is a sufficient sample:

<div class="paragraph">
<p>… the kind of politics that the liberal economist F. A. Hayek called &#8220;socialist.&#8221; <span class="footnote">[<a id="_footnoteref_1" class="footnote" href="#_footnote_1" title="View footnote.">1</a>]</span></p>
</div>

Could you attach or link to the generated (noncompliant) epub itself?

I'm confused.

https://pandoc.org/demos.html 28.

AsciiDoc:

pandoc -s MANUAL.txt -t asciidoc -o example28.txt

Why it supports only .txt extension and if you pass .adoc it triggers an error: Unknown input format asciidoc?

Also if as output I set example.pdf it will trigger cannot produce pdf output from asciidoc which is expected if it's not supported but if I put example.docx it will generate a docx file containing the Asciidoc document (same as txt output) in a docx and not something converted so if docx is not supported it should trigger an error too.

Pandoc assumes that files with .txt extension contain Markdown, while those with a .adoc extension contain Asciidoc. However, while pandoc can write Asciidoc, it cannot read it. Hence this issue.

I'm not sure what you are asking in the second paragraph. Please raise this on the pandoc-discuss mailing list. We want to reserve this tracker for bug reports and feature requests, questions should go to the mailing list.

AsciiDoc has gained in importance in the standardization community: see MetaNorma.
Electrical CIM people want to use it:
- [ENHANCEMENT] Embed Pandoc for Word generation cimug-org/CIMTool#185,
- https://github.com/Sveino/Spec4CIM-KG/ is a migration if IEC CIM standards to adoc (but that repo is closed for licensing reasons)

We don't have to be convinced that it would be good to have an asciidoc reader. What is needed is just for someone to contribute one. asciidoc-hs noted above might be a good starting point if someone wants to take this on.

In the mean time, you can probably get good results using asciidoctor to convert to HTML or DocBook and then converting the result using pandoc to your target format.

@jgm

Is It Viable?

Since I use both pandoc and AsciiDoc (Asciidoctor Ruby) on a daily basis I'd really love to see a native AsciiDoc reader in pandoc, but ... having given it quite a lot of though, I'm just wondering whether it's viable.

The AsciiDoc syntax is much richer and broader than pandoc, so it stands to reason that except for simple documents it will tend to be a lossy conversion. Which makes me wonder if it wouldn't make sense to approach the issue the other way round, and create a dedicated pandoc AsciiDoc backend instead. There are currently various AsciiDoc incarnations (Python, Ruby, JS, Java) and some of them might offer an easier approach to this task — indeed, the Haskell library mentioned by @jgm (gmarpons/asciidoc-hs) would pave the path to possibly including a full-fledged AsciiDoc parser into pandoc.

I did look into this for Asciidoctor Ruby, but at the time the AST wasn't documented enough to make the task easy. I'm not sure if the situation has changed, but once you have access the AsciiDoc document in AST format, converting it to a pandoc AST should be fairly straightforward (but still be a lossy conversion, e.g. in case of book Parts, callouts, and other features not present in pandoc native elements).

So, ideally, the creation of an AsciiDoc backend capable of emitting a pandoc AST would solve the problem, even though it might not be as practical as having pandoc natively support AsciiDoc as an input format.

What I'm trying to say is that, given that in many cases the conversion could be lossy, it might make more sense to handle it on the AsciiDoc side, which might make it easier to control how unsupported elements are to be rendered (or ignored) during conversion. Having to control such settings from within pandoc might be rather complicated, and surely not something that could be easily handled via command line switches (maybe a YAML settings file? or a dedicated defaults file?).

The Haskell AsciiDoc library could indeed be a game changer in this respect, since it would allow inclusion of a real AsciiDoc parser into pandoc. But even so, the question of how to handle unsupported elements would have to be faced.

I've used various intermediate formats as a workaround for AsciiDoc to pandoc conversion, but the issues with unsupported features (and what to do with/to them) remains. DocBook is probably the closest syntax to AsciiDoc, and for simple documents it works pretty well out of the box.

Yes, exactly. It would certainly be lossy. I just assumed that this was what explained our lack of it. Are there any other lossy inputs? How they deal with the problem?

…

On Fri, Dec 6, 2024 at 6:10 PM Tristano Ajmone ***@***.***> wrote: Is It Viable? Since I use both pandoc and AsciiDoc (Asciidoctor Ruby) on a daily basis I'd really love to see a native AsciiDoc reader in pandoc, but ... having given it quite a lot of though, I'm just wondering whether it's viable. The AsciiDoc syntax is much richer and broader than pandoc, so it stands to reason that except for simple documents it will tend to be a lossy conversion. Which makes me wonder if it wouldn't make sense to approach the issue the other way round, and create a dedicated pandoc AsciiDoc backend instead. There are currently various AsciiDoc incarnations (Python, Ruby, JS, Java) and some of them might offer an easier approach to this task — indeed, the Haskell library mentioned bu @jgm <https://github.com/jgm> (gmarpons/asciidoc-hs) would pave the path to possibly including a full-fledged AsciiDoc parser into pandoc. I did look into this for Asciidoctor Ruby, but at the time the AST wasn't documented enough to make the task easy. I'm not sure if the situation has changed, but once you have access the AsciiDoc document in AST format, converting it to a pandoc AST should be fairly straightforward (but still be a lossy conversion, e.g. in case of book Parts, callouts, and other features not present in pandoc native elements). So, ideally, the creation of an AsciiDoc backend capable of emitting a pandoc AST would solve the problem, even though it might not be as practical as having pandoc natively support AsciiDoc as an input format. What I'm trying to say is that, given that in many cases the conversion could be lossy, it might make more sense to handle it on the AsciiDoc side, which might make it easier to control how unsupported elements are to be rendered (or ignored) during conversion. Having to control such settings from within pandoc might be rather complicated, and surely not something that could be easily handled via command line switches (maybe a YAML settings file? or a dedicated defaults file?). The Haskell AsciiDoc library could indeed be a game changer in this respect, since it would allow inclusion of a real AsciiDoc parser into pandoc. But even so, the question of how to handle unsupported elements would have to be faced. I've used various intermediate formats as a workaround for AsciiDoc to pandoc conversion, but the issues with unsupported features (and what to do with/to them) remains. DocBook is probably the closest syntax to AsciiDoc, and for simple documents it works pretty well out of the box. — Reply to this email directly, view it on GitHub <#1456 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABUYZYP7YX5CKULV7UBM3D2EIVHBAVCNFSM6AAAAABTE3ZLO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMRUGU4DONBRGM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- "I have delusions of grandeur. I believe myself to be Steve Jones"

Sure, many formats are lossy, as explained at the beginning of the manual. Pandoc just does its best. And we can do a lot with Div and Span elements in the AST; with these, we can probably capture most or perhaps all of the semantic information in an asciidoc document.

@tajmone

@tajmone thanks for the information! Is there a comparison between their respective features?

@tajmone

@tajmone thanks for the information! Is there a comparison between their respective features?

I can't think of a single document with a direct comparison of features, especially the pandoc Markdown flavor vs AsciiDoc. The AsciiDoc Language Documentation is a good source of information when it comes to AsciiDoc features, and contains various Appendixes and summaries which come in handy.

A main obstacle that comes to mind is the level of book partitioning supported by AsciiDoc, which includes book parts and special sections, for which pandoc doesn't have any equivalent native elements, so I don't see how such documents could be converted without loosing the book structure. In theory, it should be possible to use Lua filters to emulate book parts, e.g. by using Level 1 headings as Part titles and shifting down one level all chapter titles, etc., e.g. in chunked documents, but this would also require to manually rebuild the ToC to keep into account how headings are shifted within book parts, compared to sections which are not inside parts (Introduction, Appendices, etc.). But then, since there are many possible ways to organize such book partitioning schemes, it might be hard to find a unified solution.

But even for simple documents, elements like admonitions, the various block types supported by AsciiDoc, callouts in code listings, etc., will be problematic to render in pandoc without agreeing on some non-native ways to represent them, according to format (e.g. in HTML admonitions could be represented by divs with agreed-upon classes) and elements like callouts might require dedicated Lua filters in ordered to be recreated in pandoc markdown and be usable in different output formats. Similar choices would inevitable be arbitrary and somewhat opinionated, unless the list of native elements is expanded to officially accommodate them as first class citizens of the pandoc AST.

AsciiDoc also relies quite heavily on attributes (native or custom) which allow conditional behaviors via preprocessor directives, e.g. to include or exclude certain blocks of contents, or to dynamically include external files (recursively). It's somewhat similar to how pandoc templates work, except that it takes place directly inside the source document, whereas with pandoc similar functionality only applies to templates, not the body text.

A native pandoc reader would have to resolve all the preprocessing before conversion, which seems quite a lot of work and its the reason why I think it might make more sense to be handling it from within AsciiDoc itself, since a same source project might produce different results depending on the output format (chunked HTML, EPub, DocBook, etc.), so preserving the various preprocessor directives allows better control over different editions.

The real problem here is that in order to be able to properly support AsciiDoc, the list of pandoc native elements would have to be expanded to include book parts, special sections, and possibly support admonitions. These are not the only elements available in AsciiDoc and absent in pandoc, but they are definitely good examples of commonly used elements which are going to be problematic to handle.

As I understand it, AsciiDoc's document model is a subset of DocBook's. We support DocBook as an input format, so there is no reason we couldn't support AsciiDoc -- and the same sorts of tradeoffs and workarounds would be needed.

If someone is interested in doing this, the best way forward would be to develop a separate Haskell library with an AST that is tailored to AsciiDoc. Pandoc could then add an AsciiDoc reader that calls this library's parser and then "downshifts" from its AST to the Pandoc AST. The library could also be used independently for people who want the Asciidoc-tailored AST.

In the mean time, I think you should get decent results by using asciidoctor to produce docbook and then converting that with pandoc.

The following example generates Markdown with headings shifted down one level (e.g., <h3> to <h2>):

asciidoctor -b docbook5 -o - README.adoc | pandoc -f docbook -t markdown

To generate Markdown with the same heading level as AsciiDoc, you need to specify leveloffset=+1 as in the example below (see asciidoctor/asciidoctor#1907 (comment)):

asciidoctor -b docbook5 -o - -a leveloffset=+1 README.adoc | pandoc -f docbook -t markdown

I think this is a bit inconvenient.

I have written a Haskell parser for AsciiDoc, which I plan to hook into a pandoc reader in the near future.
https://github.com/jgm/asciidoc-hs

Add asciidoc as an input format.

asciidoc: use pandocs built-in reader

Navigation Menu

Repository navigation

Add AsciiDoc Reader / AsciiDoc input support #1456

56 remaining items

Is It Viable?

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Add AsciiDoc Reader / AsciiDoc input support #1456

Description

Activity

mpickering commented on Jul 31, 2014

mpickering commented on Aug 2, 2014

ERnsTL commented on Aug 7, 2014

alexborisov commented on Oct 3, 2014

ciampix commented on Nov 19, 2014

jgm commented on Apr 20, 2015

mpickering commented on Apr 20, 2015

romario89 commented on May 19, 2015

benhourigan commented on Jul 5, 2015

jgm commented on Jul 5, 2015

benhourigan commented on Jul 5, 2015

jgm commented on Jul 5, 2015

56 remaining items

noraj commented on Jun 6, 2022

tarleb commented on Jun 6, 2022

VladimirAlexiev commented on Dec 6, 2024

jgm commented on Dec 6, 2024

tajmone commented on Dec 6, 2024

Is It Viable?

sjfloat commented on Dec 7, 2024

jgm commented on Dec 7, 2024

VladimirAlexiev commented on Dec 14, 2024

tajmone commented on Dec 15, 2024

jgm commented on Dec 15, 2024

sorairolake commented on Dec 27, 2024

jgm commented on Nov 22, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions