Problems including Asciidoctor Diagrams for PDF and EPUB

[ianflett]

While trying to produce documentation in PDF or EPUB format, I've had problems using Asciidoctor Diagram.

With the asciidoctor task, everything just works. However, with asciidoctorPdf, the diagram is expected in the imagedir folder from the project root. With asciidoctorEpub the expectation is the same, except it fails to find the file d (i.e. the first character of the generated filename):

asciidoctor: ERROR: _content.adoc: image not found or not readable: %project-root%/images/d

Additionally, it seems setting the folder between asciidoctorPdf and asciidoctorEpub is inconsistent, apparent in my build.gradle (below). I've added an example project to https://github.com/ianflett/test-asciidoctor.

plugins {
    // Gradle IDE plugins
    id 'eclipse'
    id 'idea'

    // Documentation
    id 'org.asciidoctor.jvm.convert' version '2.2.0'
    id 'org.asciidoctor.jvm.epub'    version '2.2.0'
    id 'org.asciidoctor.jvm.gems'    version '2.2.0'
    id 'org.asciidoctor.jvm.pdf'     version '2.2.0'
}

apply plugin: 'java'

repositories {
    jcenter()
    maven { url 'http://rubygems-proxy.torquebox.org/releases' }
}

dependencies {
    asciidoctorGems 'rubygems:asciimath:1.0.8'
}

wrapper { gradleVersion = '5.4.1' }

asciidoctorj {
    modules {
        diagram.version '1.5.18'
        epub.version '1.5.0-alpha.9'
        pdf.version '1.5.0-alpha.17'
    }

    requires += 'asciimath'

    attributes = [
        sourcedir: project.sourceSets.main.java.srcDirs[0],
        'source-highlighter': 'coderay'
    ]
}

asciidoctor {
    dependsOn asciidoctorGemsPrepare
}

asciidoctorEpub {
    dependsOn asciidoctorGemsPrepare
    ebookFormats EPUB3

    attributes += [
        // Required to place image in project root.
        'imagesoutdir': '../../../images'
    ]
}

asciidoctorPdf {
    dependsOn asciidoctorGemsPrepare

    attributes += [
        // Required to place image in project root.
        'imagesoutdir': 'images'
    ]
}

kindlegen {
    agreeToTermsOfUse = true
}

[mojavelinux]

With asciidoctorEpub the expectation is the same, except it fails to find the file d (i.e. the first character of the generated filename):

This was a known bug that has since been fixed. See asciidoctor/asciidoctor#3283 and asciidoctor/asciidoctor-epub3#188. In short, use Asciidoctor >= 2.0.9.

However, with asciidoctorPdf, the diagram is expected in the imagedir folder from the project root.

This has been fixed in Asciidoctor PDF 1.5.0.alpha.17. See asciidoctor/asciidoctor-pdf#1033

[jagedn]

Not sure if it can help but I solved it using useIntermediateWorkDir and baseDirFollowsSourceDir

You can check it at https://gitlab.com/puravida-asciidoctor/asciidoctor-amazon-template

[ysb33r]

@ianflett Thanks for reporting this inconsistency. I'll have a look into it. In the meantime I suggest to use the two settings that @jagedn has suggested.

@mojavelinux It is not possible to use Asciidoctorj 2.0 (which will have Asciidoctor 2) with the 2.x version of the Gradle plugin when additional GEMs are involved. This is due to the API change in Asciidoctorj 2.0. For this 3.0.0-alpha.1 will be required. I cannot guarantee that the same inconsistency does not exist in 3.x as I have not tested that yet.

[jagedn]

in my project there is a 3.0.0-alpha.1 branch where I use this version of the plugin and I think we still need these settings but not sure if I'm using last versions of pdf.

[ysb33r]

Default versions in 3.0.0-alpha.1 are listed here ->

asciidoctor-gradle-plugin/module-versions.properties

Line 4 in 28f9f93

asciidoctorj=2.0.0

and default versions in 2.2.0 are here ->

asciidoctor-gradle-plugin/asciidoctor-gradle-jvm/src/main/groovy/org/asciidoctor/gradle/jvm/AsciidoctorJExtension.groovy

Line 62 in 9203ed1

final static String DEFAULT_ASCIIDOCTORJ_VERSION = '1.6.2'

[mojavelinux]

It's still possible to implement what I suggested with 1.5.8. It's just necessary to set the data-uri attribute in that situation.

[ianflett]

Thanks for all the responses, and sorry for the delay getting back.

I could only get the PDF working with @jagedn's code; the EPUB didn't seem to open, but I might have done something wrong.

Using the data-uri method worked for PDF with less configuration. I thought it didn't work for EPUB until checking today - it appears it does, it just still generates a location warning.

However, it appears using the new 3.0.0-alpha.2, the PDF's now working without the extra configuration.

[ysb33r]

@ianflett So there is still an outstanding issue on EPUB?

[ianflett]

Yes, it appears so. Since your last comment I've tried ruling out user error, but I'm still experiencing problems:

13:58:47: Executing task 'asciidoctorEpub'...

Configuration on demand is an incubating feature.
> Task :asciidoctorGemsPrepare UP-TO-DATE

> Task :asciidoctorEpub
asciidoctor: ERROR: _content.adoc: image not found or not readable: %PROJDIR%/test-asciidoctor/images/d
Post-processing kindlegen files on Windows not yet implemented.

I've also updated my test project to reflect what I'm testing: https://github.com/ianflett/test-asciidoctor

[ysb33r]

Some thing I have noticed on this

• KF8 is not supported on Windows at present. From what I see from your test project you are only trying to create EPUB format.
• Post-processing kindlegen files on Windows not yet implemented.. This messages should not be printed on Windows unless KF8 is specified as an ebook format.

[slonopotamus]

This was a known bug that has since been fixed. See asciidoctor/asciidoctor#3283 and asciidoctor/asciidoctor-epub3#188. In short, use Asciidoctor >= 2.0.9.

If I understand things properly, there is no way to use Asciidoctor >= 2.0.9 from Gradle, because asciidoctor-gradle-plugin doesn't use Asciidoctor directly, but instead depends on AsciidoctorJ. But, the latest release of AsciidoctorJ is 2.0.0 and it includes Asciidoctor 2.0.8 instead of required Asciidoctor >= 2.0.9.

Worse, even when AsciidoctorJ with Asciidoctor >= 2.0.9 is released, I am not aware of any way how asciidoctor-gradle-plugin user is able to specify what AsciidoctorJ version to use.

So, seems like we need to wait for a new release of AsciidoctorJ, then bump AsciidoctorJ dependency in asciidoctor-gradle-plugin, then release a new version of asciidoctor-gradle-plugin and only then image not found or not readable issue with ditaa will be fixed.

Overrall, "in short, use Asciidoctor >= 2.0.9" solution doesn't work because there is no way asciidoctor-gradle-plugin user can do that. Please, correct me if I am wrong and there exists a way to specify what version of Asciidoctor (without J) to use through asciidoctor-gradle-plugin.

[mojavelinux]

@slonopotamus You are correct. We need a new release of AsciidoctorJ (2.0.1) to collect all the bug fixes from Asciidoctor.

[slonopotamus]

@mojavelinux nice. And is there any open issue similar to asciidoctor/asciidoctor-pdf#1033 for asciidoctor-epub3?

With asciidoctor-gragle-plugin 3.0.x, ditaa in PDF works out of the box, but we still need to specify data-uri + imagesoutdir for epub3 like @ianflett does here. Otherwise, images are generated in src/docs/asciidoc, which is definitely a wrong dir to place generated files.

[mojavelinux]

I believe it was addressed here: asciidoctor/asciidoctor-epub3#188 (comment)

[ysb33r]

Thinking about it we should probably set a default for imagesoutdir for epub & pdf. User can always override if necessary.

We want the final output dir to be as pristine as possible.

[mojavelinux]

The imagesoutdir should be computed automatically from the output directory and the imagesdir. If it is not (perhaps because the output directory is not set in the document options), only then would you need to set the imageoutdir.

For details, see https://github.com/asciidoctor/asciidoctor-diagram/blob/master/lib/asciidoctor-diagram/extensions.rb#L254-L267

[slonopotamus]

Fresh attempt. I'm using:

asciidoctorj 2.1.0 (includes asciidoctor 2.0.10)
asciidoctorj-epub 1.5.0-alpha.9
asciidoctorj-diagram 1.5.18
asciidoctor-gradle-plugin 3.0.0-alpha.3

HTML/PDF just works, without any hacks. EPUB no longer warns about /d directory. But ditaa still generates files into src/docs/asciidoc for EPUB. As far as I understand, this happens because EPUB doesn't set outdir unlike HTML/PDF.

I tried to explicitly set outdir tobuild/docs/asciidocEpub via attributes, without touching data-uri or imagesoutputdir. That way, ditaa stops generating stuff in src/docs/asciidoc and instead puts it into build/docs/asciidocEpub/images which is a Good Thing. But EPUB doesn't find images then and complains:

asciidoctor: ERROR: file.adoc: image not found or not readable: /path/to/src/docs/asciidoc/images/diag.png

Sooo... Seems like even if I use latest releases of all components, ditaa still doesn't properly work for EPUB.

@mojavelinux how PDF knows that it needs to search images both in src/docs/asciidoc/images and build/docs/asciidocPdf/images and why EPUB doesn't know that? UPD: okay, there's no mechanism to collect images from multiple dirs and PDF doesn't do that. Instead, PDF sets data-uri (empty string also works) and that forces diagram plugin to use absolute paths: asciidoctor/asciidoctor-pdf#1034.

[ysb33r]

Add

asciidoctor {
    useIntermediateWorkDir()
}

This will place everything that is needed in an intermediate directory before processing. In that way ditaa should no longer write images to src/docs/asciidoc

[slonopotamus]

@ysb33r

1. This is still a hack, if EPUB needs that to operate, it should set it itself. HTML/PDF don't need that for some reason.
2. Using useIntermediateWorkDir() breaks chapter includes for me because they're not copied to intermediate workdir, only the spine file is there.

[ysb33r]

@slonopotamus Rename your chapter files not to be underscore-led.

I cannot remember the original motivation for this, but for some reason in asciidoctor past, files startign with underscores are not valid. Maybe @mojavelinux can remember.

See:

asciidoctor-gradle-plugin/asciidoctor-gradle-base/src/main/groovy/org/asciidoctor/gradle/base/AbstractAsciidoctorBaseTask.groovy

Line 663 in cd528cc

protected PatternSet getDefaultSourceDocumentPattern() {

[slonopotamus]

Undercore is a default pattern used by asciidoctor-gradle-plugin to determine which files to convert, that's why I used it for chapters. Chapters are only meant to be included, but not processed on their own as standalone documents.

[ianflett]

Just to support @slonopotamus, that was my experience with chapter includes too. I originally tried without underscores and received feedback that led me to add them.

[ysb33r]

💡

That's a bug in the Gradle plugin. Those files will need to be copied. #410 created for that.

@slonopotamus To workaround that in the interim until we release 3.0.0-alpha.4 I present you with a terrible hack

asciidoctorEpub {
    useIntermediateWorkDir()
   
    resources {
      from sourceDir, {
          include '_*.adoc'
      }
}

Yes, I had to use resources to get around the problem for now.

[slonopotamus]

One more attempt:

1. I manually set outdir to build/docs/asciidocEpub. This way images are generated there instead of src dir. Why this works out-of-the-box for PDF is still an open question.
2. I manually set data-uri to empty string. This is what PDF does in resolves #1033 automatically set data-uri so converter can find images generated by Asciidoctor Diagram asciidoctor-pdf#1034 in order to properly cooperate with asciidoc-diagram.

But... EPUB still doesn't work! It turns out, there's one more bug in it, asciidoc-epub doesn't handle absolute image paths: asciidoctor/asciidoctor-epub3#199

[ysb33r]

Thanks people for the effort you are putting in to using EPUB. Some good bugs have been discovered. EPUB has definitely not received the same amount of love as a project as PDF.

[ysb33r]

This was a known bug that has since been fixed. See asciidoctor/asciidoctor#3283 and asciidoctor/asciidoctor-epub3#188. In short, use Asciidoctor >= 2.0.9.

3.0.0-alpha.3 contains AsciidoctorJ 2.10 which uses Asciidoctor 2.0.10.

This has been fixed in Asciidoctor PDF 1.5.0.alpha.17. See asciidoctor/asciidoctor-pdf#1033

3.0.0-alpha.3 uses asciidoctorj-df 1.5.0-alpha.17 by default.

This is still a hack, if EPUB needs that to operate, it should set it itself. HTML/PDF don't need that for some reason.
Using useIntermediateWorkDir() breaks chapter includes for me because they're not copied to intermediate workdir, only the spine file is there.

Will be fixed in 3.0.0-alpha.4

That's a bug in the Gradle plugin. Those files will need to be copied. #410 created for that.

Also addressed in 3.0.0-alpha.4.

I am closing this one. If you find an issue post 3.0.0-alpha.4 please open a new issue.