Markdown Input Documentation Incomplete

[maxmes]

I cannot create a PDF from a Markdown input.
Maybe I am doing something wrong. Maybe there is a bug in DITA-OT. Or the documentation is incomplete.

Background: I am a DITA-OT beginner. I want to evaluate if DITA-OT can be something useful for our project. I want to verify that DITA-OT can export and import Markdown.

Steps to reproduce

Test 1

First create some folders:

mkdir test1

cd ./test1
mkdir in
mkdir out
mkdir temp
mkdir log

Manual Steps:

1. Download develop.zip from: https://github.com/dita-ot/docs/archive/develop.zip
2. Extract the samples subfolder from develop.zip to /test1/in

You now have a folder /test1/in/samples, containing hierarchy.ditamap. Let's run DITA-OT.

$DITA_HOME/bin/dita --input=in/samples/hierarchy.ditamap --format=pdf --output=out/pdf/ --temp=temp/pdf -Dclean.temp=no
$DITA_HOME/bin/dita --input=in/samples/hierarchy.ditamap --format=markdown_gitbook --output=out/md --temp=temp/md -Dclean.temp=no

Result: The 2 DITA statements work. They produce valid output: PDF and Markdown.

Test 2

Create some folders:

cd ..
mkdir test2
cd test2
mkdir in
mkdir out
mkdir temp
mkdir log
mkdir in/md

Create a folder /test2/in/md that is a copy of /test1/out/md/. In other words: Let's use the output of test1 as an input for test2.

cp -r ../test1/out/md/ in

Create a file test.ditamap in /test2/in. The content of /test2/in/test.ditamap is:

<map>
  <topicref href="md/SUMMARY.md" format="markdown"/>
</map>

Note: This file was inspired by https://www.dita-ot.org/dev/topics/markdown-input.html

$DITA_HOME/bin/dita --input=in/test.ditamap --format=pdf --output=out/pdf/ --temp=temp/run1/ -Dclean.temp=no

Result: No PDF is created and the following error is displayed: "Error: Failed to run pipeline: [DOTJ022F][FATAL] Failed to parse the input file '/test2/in/test.ditamap' because all of its content has been filtered out. This will happen if the input file has filter conditions on the root element, and a ditaval excludes all content based on those conditions."

Because of the error I want to see some debug information.

$DITA_HOME/bin/dita --input=in/test.ditamap --format=pdf --output=out/pdf/ --temp=temp/run1/ -Dclean.temp=no --logfile=log/log.txt --debug

This produces only a file named "--debug" in /test2/

$DITA_HOME/bin/dita --input=in/test.ditamap --format=pdf --output=out/pdf/ --temp=temp/run1/ -Dclean.temp=no -l log/log.txt -d

Now a log file /test2/log/log.txt is created but it does not give much help

Test 2, 2nd Version

Create a second file test2.ditamap in /test2/in. As is /test1/in/samples/hierarchy.ditamap add now a DOCTYPE. The content of /test2/in/test2.ditamap looks like:

<?xml version="1.0" encoding="utf-8"?>

<!DOCTYPE map PUBLIC "-//OASIS//DTD DITA Map//EN" "map.dtd">
<map>
  <topicref href="md/SUMMARY.md" format="markdown"/>
</map>

Now run:

$DITA_HOME/bin/dita --input=in/test2.ditamap --format=pdf --output=out/pdf/ --temp=temp/run2/ -Dclean.temp=no -l log/log2.txt -d

Result: No PDF output but a different error.

In log/log2.txt there seems to be a problem with /test2/in/md/SUMMARY.md or temp\run2\0ea31965dc6fe7a884b6eedc38b4659184ec7094.md

map-reader:
[map-reader] Using Xerces grammar pool for DTD and schema caching.
[map-reader] Processing file:.../test2/in/test2.ditamap to file:.../test2/temp/run2/b3c5ad391073aab6f15d53aaf079cd09d3aa07f4.ditamap
[map-reader] Serializing job specification
[map-reader] MapReaderModule processing took 960 ms
[job-helper] Using class org.apache.tools.ant.taskdefs.optional.TraXLiaison
[job-helper] Processing ...\test2\temp\run2\.job.xml to ...\test2\temp\run2\fullditamap.list
[job-helper] Loading stylesheet C:\DEV\dita-ot-3.0.3\xsl\job-helper.xsl
   [delete] Deleting: ...\test2\temp\run2\fullditamap.list
...
topic-reader:
[topic-reader] Using Xerces grammar pool for DTD and schema caching.
[topic-reader] Reading file:.../test2/temp/run2/b3c5ad391073aab6f15d53aaf079cd09d3aa07f4.ditamap
[topic-reader] Loading subject schemes
[topic-reader] Reading file:.../test2/temp/run2/b3c5ad391073aab6f15d53aaf079cd09d3aa07f4.ditamap
[topic-reader] Processing file:.../test2/in/md/SUMMARY.md to file:.../test2/temp/run2/0ea31965dc6fe7a884b6eedc38b4659184ec7094.md
[topic-reader] [DOTJ013E][ERROR] Failed to parse the referenced file 'file:.../test2/in/md/SUMMARY.md'.: com.elovirta.dita.markdown.MarkdownReader
[topic-reader] Serializing job specification
[topic-reader] TopicReaderModule processing took 129 ms
...
[move-meta] I/O error reported by XML parser processing file:.../test2/temp/run2/0ea31965dc6fe7a884b6eedc38b4659184ec7094.md: ...\test2\temp\run2\0ea31965dc6fe7a884b6eedc38b4659184ec7094.md (The system cannot find the file specified)
[move-meta] file:.../test2/in.../test2.ditamap:4:53: [DOTX023W][WARN]: Unable to retrieve navtitle from target: '0ea31965dc6fe7a884b6eedc38b4659184ec7094.md'.
[move-meta] file:.../test2/in.../test2.ditamap:4:53: [DOTX027W][WARN]: Unable to retrieve linktext from target: '0ea31965dc6fe7a884b6eedc38b4659184ec7094.md'.

There is a b3c5ad391073aab6f15d53aaf079cd09d3aa07f4.ditamap but no 0ea31965dc6fe7a884b6eedc38b4659184ec7094.md in my folder /test2/temp/run2/.

Could it be that some copy job has failed?

I have invested a few hours and I am at a loss. I cannot accomplish anything with Markdown input. That's a pity, because DITA-OT looks extremely promising. You really have made a lot of efforts for a complex but mature solution.

Just an idea: How about making a beginner's documentation in the form of some examples, similar to what I do above. Folders: in, out, temp, log, plugin etc. and a file README.md with a short description and the command line statements. If you then put the command line statements into a shell script, you could even re-use it for your test automation.

[infotexture]

@maxmes Welcome to GitHub, and congratulations on opening your first issue. 🎉

Thanks for taking the time to describe the process you followed. Sorry to hear your first steps with the toolkit didn't go as you expected, but there's a lot going on here.

At first glance, it looks like the path to the SUMMARY.md file in your /test2/in/test.ditamap is wrong.

Assuming the summary file is at the root level of /test2/in/ alongside test.ditamap, the path would need to be adjusted as follows:

<map>
- <topicref href="md/SUMMARY.md" format="markdown"/>
+ <topicref href="SUMMARY.md" format="markdown"/>
</map>

(Since the referenced file is missing, the map is effectively empty, which leads to the error you saw.)

Unfortunately, after that you'll run into another error, which is caused by an open issue on our side (dita-ot/dita-ot#2929):

Error: java.lang.RuntimeException: No renderer configured for com.vladsch.flexmark.ext.typographic.TypographicSmarts

Are you using DITA-OT 3.0.2 or 3.0.3?

— If so, you should be able to get past that one by setting the format attribute to mdita:

<map>
- <topicref href="SUMMARY.md" format="markdown"/>
+ <topicref href="SUMMARY.md" format="mdita"/>
</map>

That should produce output, but not what you might expect, since the GitBook SUMMARY.md file is itself something like a map file in that it defines a publication's table of contents, so you'd typically want to point directly to topic files from the map.

I appreciate your suggestion of a beginner's tutorial for the Markdown publishing use case. The toolkit's Markdown support is a relatively new addition, and so far we've assumed an audience of experienced DITA practitioners who are interested in adding Markdown topics to their existing projects.

However, you're not the first to ask such questions, so we should certainly consider clarifying these use cases in an upcoming version of the docs.

[maxmes]

Hi Roger,
Thanks for your answer.

I am using the latest version of DITA-OT: 3.03. As you had recommended I have changed the format in the Ditamap file to "mdita" and I have referenced a Markdown file with "normal text" instead of SUMMARY.md.
Sadly this doesn't work either.

My Ditamap file (/test2/in/test3.ditamap):

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE map PUBLIC "-//OASIS//DTD DITA Map//EN" "map.dtd">
<map>
  <topicref href="md/concepts/shelving.md" format="mdita"/>
</map>

My Markdown file (/test2/in/md/concepts/shelving.md). Note: This is https://github.com/dita-ot/docs/blob/develop/samples/concepts/shelving.xml converted by DITA-OT to Markdown.

# Shelving

Wooden or metal shelving is useful for storing things in the garage.

Shelving also keeps things neat and organized.

**Related information**  


[Organizing the workbench and tools](../tasks/organizing.md)

Then I have executed:

cd ./test2
$DITA_HOME/bin/dita --input=in/test3.ditamap --format=pdf --output=out/pdf/ --temp=temp/run3/ -Dclean.temp=no -l log/log3.txt -d

Result: No PDF was created.

Excerpts from the log file (/test2/log/log3.txt):

...
map-reader:
[map-reader] Using Xerces grammar pool for DTD and schema caching.
[map-reader] Processing file:...test2/in/test3.ditamap to file:...test2/temp/run3/8a2e19db6249f18ded36785824134a6916d8f9a7.ditamap
...
topic-reader:
[topic-reader] Using Xerces grammar pool for DTD and schema caching.
[topic-reader] Reading file:...test2/temp/run3/8a2e19db6249f18ded36785824134a6916d8f9a7.ditamap
[topic-reader] Loading subject schemes
[topic-reader] Reading file:...test2/temp/run3/8a2e19db6249f18ded36785824134a6916d8f9a7.ditamap
[topic-reader] Processing file:...test2/in/md/concepts/shelving.md to file:...test2/temp/run3/6af489a7fbb9cec6202a51df785848f763f239f9.md
[topic-reader] [DOTJ013E][ERROR] Failed to parse the referenced file 'file:...test2/in/md/concepts/shelving.md'.: com.elovirta.dita.markdown.MDitaReader
[topic-reader] Serializing job specification
[topic-reader] TopicReaderModule processing took 132 ms
...

Have you tried the Markdown input format yourself? Did it work for you?

I understand your difficulties to write documentation for beginners. I have already mentioned the page "Markdown content" which contains a Ditmap without a DOCTYPE. For a beginner that small thing (missing DOCTYPE) is really a challenge, for a DITA expert it is obvious.

There are two reasons why people use things like Stackoverflow.

1. You can ask silly questions - as I do right now ;)
2. You often find working code samples.

I have found no examples for Markdown or Lightweight DITA in your GitHub repositories. An input folder with some simple documents including a Ditamap and a DITA-OT command to produce output might be more helpful for a beginner than the formal specification.

[infotexture]

As mentioned above, I think the cause for the DOTJ013E error here is the path to the SUMMARY.md file.

If I understand your directory structure correctly, you need to remove the intermediate md/ directory from the path to that file (or to the concepts/shelving.md file):

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE map PUBLIC "-//OASIS//DTD DITA Map//EN" "map.dtd">

<map>
- <topicref href="md/SUMMARY.md" format="markdown"/>
+ <topicref href="SUMMARY.md" format="markdown"/>
</map>

With that change, I am able to generate PDF output with the command line you used.

The other error I mentioned above (dita-ot/dita-ot#2929) has since been resolved, so in the next maintenance release of DITA-OT, it will work with the format attribute set to either markdown or mdita.

[infotexture]

I added the XML processing instruction and DOCTYPE reference to the sample map, so that example should now be more useful as is.

We'll consider adding a simple Markdown/LwDITA example to the samples/ folder in an upcoming release. Thanks for suggesting that.

If you need more help getting started with DITA and DITA-OT, check out the Stack Overflow questions tagged DITA and DITA-OT, or join the Yahoo! dita-users group.

[maxmes]

Thank you, Roger! I have found now the root cause.

The problems comes from using (Git-)Bash on a Windows system. Apparently the Unix-like path is misinterpreted by the dita command (i.e. with / instead of \).

When I change the call to Windows notation and call it from the Windows command line, it works!

%DITA_HOME%\bin\dita --input=in\test2.ditamap --format=pdf --output=out\pdf\ --temp=temp\run2\ -Dclean.temp=no -l log\log2.txt -d

[infotexture]

Glad to hear you were able to get it working. I will now close this issue.