Changing severity and intentionally failing builds

[stefan-jung]

I would like to fail a build when a severe problem occurs. Severe is in my case, if content is missing, e.g. an image or a topic. The DITA-OT project file based publishing creates PDFs without complaining too much. This leads to the risk, that incomplete assets are passed to our factories for mass production.

For example, when the input map is missing, the build fails. This is expected.

[exec] Error: Failed to run pipeline: [DOTA069F][FATAL] 
Input file 'file:/x.ditamap' cannot be located or read. 
Ensure that file was specified properly and that you have permission to access it.

This error indicates a missing image, this is critical for us. The severity is INFO and the build does not fail.

[exec]    [keyref] file:/x.dita:72:33: [DOTJ048I][INFO] 
Unable to find key definition for key reference "y" in scope "z". 
The href attribute may be used as fallback if it exists

This leads to a missing chapter. This is critical for us. The severity is ERROR, but the build does not fail.

[exec] [topic-merge] [DOTX008E][ERROR] File 'x-1.dita' does not exist or cannot be loaded.

What's the recommended solution? Should I try to implement a custom log parser searching for strings in a --verbose log? This feels wrong to me. How can I do this in a DITA-OT project files based publishing environment?

[raducoravu]

I think @jelovirt made some recent changes for DITA OT 3.7.1 to handle missing references to images: #3858
but probably your specific requests are not handle by strict mode processing. Not sure how best to approach this, maybe you could add a new issue with a sample project.
About missing key definitions, I think the DITA 1.3 specification states something like:

https://www.oxygenxml.com/dita/1.3/specs/archSpec/base/processing-key-references-general.html#processing_key_references__error-conditions

If a referencing element contains a key reference with an undefined key, it is processed as if there were no key reference, and the value of the @href attribute is used as the reference. If the @href attribute is not specified, the element is not treated as a navigation link. If it is an error for the element to be empty, an implementation MAY give an error message; it also MAY recover from this error condition by leaving the key reference element empty.

I'm not sure if this gives the publishing engine enough flexibility to fail in such cases.

[kirkilj]

@xephon2, one of my colleagues at Nokia created #3858 that was fixed in 3.7.1 and we now default to strict mode. We're glad that it was addressed quickly in a fix release. We want our Jenkins doc pipeline to appropriately set the status of a Jenkins build to FAIL or UNSTABLE (Jenkins options) on certain OT messages. Missing content is very problematic, especially when generating 1000s of pages/topics in a product's doc set. We may decide to turn off strict mode and let a custom log analyzer set the Jenkins job's status according to our own rules, because if you fail a build, no other documents in a docset will be generated and you'll be in a loop of fix one problem, rerun the build, fix the next error, ad nauseum. If we're lucky, we might even be able to turn some file references into links in a tabular HTML report to the writer's local repo for the specific product. The writer might need to enter the path to the local repo on their workstation however, since we don't prescribe where local repos on each writers workstation should be stored. An HTML form field and some Javascript would take care of the link generation.

At a prior company, the DocTools team created a post-processor that would create additional log files that just included messages for each severity level. If a writer just wanted to see errors, they could focus just on that log file. They would also reword the messages' text if the default OT message was not as clear or actionable as it could be. We have an open ticket for our pipeline to make the logs more actionable, particularly for writers who may be new to DITA.

We support writers building docs both locally on their workstation as well as our DevOps server cluster, so we want whatever we do in this regard to be portable between Windows and Linux. We're also packaging our rendering pipeline into Docker images that can be reused at different layers. So we want to include such log filtering to be portable as well so it's part of these images. Hopefully, it can just be regex based. We might also want to scan for Jenkins errors or others in addition to OT messages. For example, if our Jenkins server can't reach our Git server, the error message doesn't make it obvious what might be the problem. We did build a custom Log Viewer using the Oxygen SDK in Java that supports regex searches, but we're not going to continue down that route since it wouldn't be usable in our DevOps pipeline.

Another fairly recent consideration is that we co-author with R&D who uses Markdown for their internal documentation, which can also use our DITA pipeline. The R&D developers know nothing of DITA, so if Markdown content results in a DITA-OT error, it could make even less sense to them. They may need to see a different set of error text that describes things for a Markdown writing audience.

[chrispy-snps]

I see that @xephon2 filed the following issue to somehow increase the severity of missing keys:

#3950: Fix broken key issue by raising severity of DOTJ047I

I would find this ability very useful. I agree with @xephon2 and @kirkilj about the severity of missing content. To us, unresolved references of any kind - images, keys, or anything else - are a sign that something is incomplete or wrong and the result should not go to production.

[kirkilj]

We discussed this at length in one of the Monthly Contributor calls.

1. One of the options I brought up was for OT to provide a default mapping of messages to severity, but also allow an organization to provide an override to elevate a given error message to be a failure in strict mode when by default, it's only a warning.
2. The other option I mentioned would be for an organization to post-process an OT log and do their own reinterpretation of the severity of certain messages.

In general, however, I feel that ANY missing content or links to nowhere should fail in strict mode. Searching through OT logs for writers is like looking for needles in haystacks. Failing builds gets their attention however.

Has there been any further discussion as to whether the first option is a feature candidate for OT?

[kirkilj]

If we create a plugin to use the extension, can we override the severity of an existing std OT message there in some fashion or is it a possibility that OT could somehow use this to allow an organization to override the severity. The last character of the message number, indicates the severity, but there's also a separate type attribute that indicates the same.

Ideally, we'd like to have a mechanism where we could specify the name of log file to which any standard OT message could be appended to in addition to the standard log. For example, writing a log called something like "missing-assets.log" in which all messages with certain ids are written to.

I'm just spit-balling here.

[pieterclaeys]

Just FYI: I also encountered this issue and discussed in the OxygenXML DITA forum:
https://www.oxygenxml.com/forum/viewtopic.php?t=25704
A community member was so friendly also to refer to this github issue ; so I'm referring back via this reply.

Our issue being: If a ditaval file reference is invalid, due to some mistake, we expect the build process to abort (especially if processing-mode strict is set, instead of simply continuing and producing content with wrong audience content).

[chrispy-snps]

Hi everyone,

Thanks for getting some discussion going on this topic! It is also important to us.

In our company, we want the following conditions to stop publishing:

• There is an unresolved @keyref reference (image, topic, etc.).
  • Some related issues are DOTX028E (missing link target) should not be satisfied by linktext #4069, DOTJ047I (unresolved key reference) should not be issued for references into declared peer maps #4068, DOTX028E (missing link target text) is issued for <link> but not <xref> #4067, and Fix broken key issue by raising severity of DOTJ047I  #3950.
• There is an unresolved @href or file reference (image, topic, DITAVAL, DITA-OT project file, etc.).
• There is profiled content that has no applicable DITAVAL guidance.
  • Some related issues are DOTJ031I messages during global filtering do not consider subsequent branch filtering #4085 and DOTJ031I (no specified DITAVAL rule) does not handle attribute groups correctly #4077.

Some users implement custom message severity using log file parsing. This requires each user to build machinery specific to their environment. The machinery must either parse the transformation STDOUT/STDERR in real-time (complexity cost) or wait for the transformation to complete (time cost). Work might be required to synchronize the behavior of map-based transformations and DITA-OT project file publishing. Work might be required to synchronize the behavior of the authoring and publishing environments (for example, having transformations in Oxygen behave consistently with CI/CD runs as @kirkilj mentioned).

I like the idea of a plugin-based method of specifying message severity overrides, similar to how we can specify text variable overrides today. I tried an experiment where I used the dita.xsl.messages extension point to change the following error from ERROR to FATAL with different response text:

<messages xmlns:dita="http://dita-ot.sourceforge.net">
  <message id="DOTX028E" type="FATAL">
    <reason>Link or cross reference ***MUST*** contain a valid href or keyref attribute; no link target is specified.</reason>
    <response>This is an unrecoverable error.</response>
  </message>
</messages>

DITA-OT plugin integration included my message definition along with the default definition:

$ grep DOTX028E config/messages.xml
  <message id="DOTX028E" type="ERROR">
  <message id="DOTX028E" type="FATAL">

Transformation failed with:

[topicpull] Message level ERROR FATAL not supported

because the message lookup in output-message.xsl did not anticipate duplicate matches. Perhaps the code could use the last matching message using [last()] to allow message definition overrides as a near-term improvement.

[raducoravu]

@chrispy-snps maybe when we want to override a message ID we could suffix it with an "-override" in the plugin's messages.xml

<messages xmlns:dita="http://dita-ot.sourceforge.net">
  <message id="DOTX028E-override" type="FATAL">
    <reason>Link or cross reference ***MUST*** contain a valid href or keyref attribute; no link target is specified.</reason>
    <response>This is an unrecoverable error.</response>
  </message>
</messages>

so in the XSLT we could first look for "idvalue-override" and if not found, look for "idvalue" and thus allow plugins to override a certain ID.
Or add some override="true" attribute on the message to signify it should be loaded first.

[chrispy-snps]

@raducoravu - that's an interesting idea but it might not be needed after all. In config/messages_template.xml, the default messages are explicitly defined first, and the extension point for user messages appears at the end:

<messages xmlns:dita="http://dita-ot.sourceforge.net">

  <!-- Start of Ant Messages -->
  <message id="DOTA001F" type="FATAL">
    <reason>"%1" is not a recognized transformation type.</reason>
    <response>Supported transformation types are <dita:extension id="dita.conductor.transtype.check" behavior="org.dita.dost.platform.ListTranstypeAction" separator=", "/>.</response>
  </message>

  <!-- ...many messages omitted... -->
  <message id="DOTX077I" type="INFO">
    <reason>Resolving content references results in duplicate ID '%1'.</reason>
    <response>Rewriting resolved version to '%2'.</response>
  </message>

  <!-- End of XSL Messages -->


  <!-- Add any messages defined by plugins. -->
  <dita:extension xmlns:dita="http://dita-ot.sourceforge.net"
    behavior="org.dita.dost.platform.InsertAction" id="dita.xsl.messages"/>

</messages>

This means a last() lookup from the messages file should work as expected. (And multiple plugins trying to override the same message is no different than how multiple plugins can try to override the same XSLT template today.)

I think the last() message lookup is something we should implement.

[chrispy-snps]

MessageUtils.java will need a similar update for messages issued from Java. Interestingly, the Java code infers the severity from the last letter of the message code instead of looking at the @type attribute of the message definition, so that would probably need to be updated too.

Here is a testcase: 3922-message-override.zip

It contains a DITA map and topic file, plus a DITA plugin that attempts to apply the following message overrides:

  <message id="DOTX028E" type="WARN">
    <reason>OVERRIDE: Link or cross reference must contain a valid href or keyref attribute; no link target is specified.</reason>
    <response>(This message's severity was changed to WARN.)</response>
  </message>

  <message id="DOTJ047I" type="FATAL">
    <reason>OVERRIDE: Unable to find key definition for key reference "%1" in root scope.</reason>
    <response>(This message's severity was changed to FATAL.)</response>
  </message>

[chrispy-snps]

The DITA-OT development team is discussing whether unresolved @keyrefs should be an error by default. Currently they are reported with information messages:

[keyref] file:/mnt/c/nobackup/test_saxon/topic.dita:7:37: [DOTJ047I][INFO] Unable to find key definition for key reference "NOTFOUND" in root scope. The href attribute may be used as fallback if it exists.

And indeed, per the DITA 1.3 specification (2.3.4.6 Processing key references), an unresolved @keyref is not an error:

If a referencing element contains a key reference with an undefined key, it is processed as if there were no key reference, and the value of the @href attribute is used as the reference. If the @href attribute is not specified, the element is not treated as a navigation link. If it is an error for the element to be empty, an implementation MAY give an error message; it also MAY recover from this error condition by leaving the key reference element empty.

This behavior enables @keyrefs to be used as optional overrides. For example, here is an @href used as a fallback if the @keyref does not resolve:

<xref href="limitations-default.dita" keyref="limitations-override"/>

But for some users—especially those that "@keyref all the things"—unresolved @keyrefs are a serious condition to be treated as an error.

The DITA-OT developers would like to hear from the DITA community about how you want unresolved @keyrefs to be handled. Feel free to share your thoughts here. Thanks!

[stefan-jung]

I'm super happy to see that this topic is now finally getting discussed. ☺️♥️

[shaneataylor]

Practically, an unresolved keyref is likely to result in a content error in your output. It is easy to define keys as null if it's your intent that they not resolve to a topic. For those reasons, my team treats unresolved keyrefs as warnings.

[shaneataylor]

FWIW @chrispy-snps we parse the xml log output to adjust severity for our own usage. Other things we change besides unresolved keyrefs include minor pdf space overruns (do I care about 10 millipoints?) and ditavals not specifying rules for a conditional processing attribute. If you're interested, I could share the full list of messages we update.

[kirkilj]

Shane, I'd be very interested in seeing your message modifications, as well as any implementation code.

At a former employer, they also supported the ability to override the actual message text via regex to make the wording more accessible and actionable to writers new to DITA.

[chrispy-snps]

@shaneataylor, @kirkilj - a plugin-based override of <message> definitions would provide the following benefits:

• Both the message text and severity can be overridden.
• The message definitions can be packaged as a DITA-OT plugin (or incorporated into an existing plugin).
• Because the severity/text overrides apply within the DITA-OT transformation itself (versus post-processing), they apply to transformations run within Oxygen or other DITA-OT-integrated applications.

The <message> redefinition mechanism would not apply to XSL-FO messages though.

[shaneataylor]

@chrispy-snps @kirkilj

Here's the part that we use to adjust the severity to match our needs. Some items are specific to our plugins, but most are not. I'm also attaching the whole XSL (with .txt extension since .xsl is verboten in Github)—we use it to build a more-usable HTML version of our build logs from the XML log that is output from OT.

<xsl:variable name="level" as="xs:string">
            <xsl:choose>
                <xsl:when test="$code='DOTX056W'">WARNING</xsl:when><!-- onlytopic.in.map exclusion -->
                <xsl:when test="$code='DOTX073I'">WARNING</xsl:when><!-- remove-broken-links exclusion -->
                <xsl:when test="$code='DOTJ047I'">WARNING</xsl:when>
                <xsl:when test="$code='DOTJ048I'">WARNING</xsl:when>
                <xsl:when test="$code='DOTJ013E' and contains(.,'com.elovirta.dita.html.HtmlReader')">INFO</xsl:when>
                <xsl:when test="$code='DOTJ007E' and contains(.,'Duplicate condition in filter file')">INFO</xsl:when>
                <xsl:when test="$code='DOTX040I' and contains(.,'Draft comment area found')">INFO</xsl:when>
                <xsl:when test="$code='DOTJ045I'">INFO</xsl:when>
                <xsl:when test="contains(.,'[DOTJ031I][INFO] No specified rule')">NONE</xsl:when>
                <xsl:when test="$code='DOTJ031I'">INFO</xsl:when>
                
                <xsl:when test="contains(.,'args.draft=yes')">WARNING</xsl:when>
                <xsl:when test="contains(.,'may lead to errors')">NONE</xsl:when><!-- style import warning -->
                <xsl:when test="matches(.,'.+?Excluding.+? from search.*?')">NONE</xsl:when>
                <xsl:when test="starts-with(.,'Revision date')">NONE</xsl:when>
                <xsl:when test="contains(.,'Using copyright date from bookmap.')">NONE</xsl:when>
                <xsl:when test="starts-with(.,'INFO: Default page-')">NONE</xsl:when>
                <xsl:when test="contains(.,'Including MathML image')">NONE</xsl:when>
                <xsl:when test="contains(.,'previously used; ID values must be unique within a document')">NONE</xsl:when>
                <xsl:when test="contains(.,'[DS03][INFO]')">NONE</xsl:when>
                <xsl:when test="contains(.,'[DS04][INFO]')">INFO</xsl:when>
                <xsl:when test="contains(.,'[DS05][INFO]')">INFO</xsl:when>
                <xsl:when test="starts-with(.,'[GIDX103I][INFO]')">NONE</xsl:when>

                <!-- FOP Messages -->
                <xsl:when test="starts-with(.,'===========  CoverArtFile =')">NONE</xsl:when>
                <xsl:when test="contains(.,'org.apache.fop.apps.FopFactoryConfigurator configure')">NONE</xsl:when>
                <xsl:when test="contains(.,'org.apache.fop.apps.FopConfParser configure')">NONE</xsl:when>
                <xsl:when test="contains(.,'Index generation is not supported in FOP')">NONE</xsl:when>
                <xsl:when test="contains(.,'Discarding font cache file')">NONE</xsl:when>
                <xsl:when test="contains(.,'org.apache.fop.fonts.FontCache loadFrom')">NONE</xsl:when>
                <xsl:when test="contains(.,'org.apache.fop.events.LoggingEventListener processEvent')">NONE</xsl:when>
                <xsl:when test="starts-with(.,'INFO: Rendered page')">NONE</xsl:when>
                <xsl:when test="contains(.,'Adjusting end-indent based on overconstrained geometry rules (XSL 1.1, ch. 5.3.4)')">NONE</xsl:when>
                <xsl:when test="matches(.,'.+?Font .+ not found. Substituting with .+\.')">INFO</xsl:when>
                <xsl:when test="matches(.,'(.+?org.apache.fop.fo.properties.CommonHyphenation getHyphChar|Substituted specified hyphenation character \(0x25ba\) with 0x2d.+)')">INFO</xsl:when>
                <xsl:when test="matches(.,'WARNING: The contents of fo:block .+ exceed the available area in the inline-progression direction by \d{1,3} millipoints')">INFO</xsl:when>
                <xsl:when test="contains(.,'[WARN] Alternate text is missing on external-graphic')">NONE</xsl:when><!-- only flags intentional empty alt text, not missing -->
                <xsl:when test="matches(.,'coverage set class table not yet supported')">INFO</xsl:when>
                
                <!-- Fallbacks -->

                <xsl:when test="contains(., '[ERROR]')">ERROR</xsl:when>
                <xsl:when test="matches(., 'java.+Exception.+')">ERROR</xsl:when>
                <xsl:when test="@priority='warn' and contains(.,'error')">ERROR</xsl:when>
                <xsl:when test="@priority='warn'">WARNING</xsl:when>
                <xsl:when test="contains(., '[WARN]') or contains(., 'Warning!')">WARNING</xsl:when>
                <xsl:when test="@priority='error'">ERROR</xsl:when>
                <xsl:when test="contains(.,'Error!')">ERROR</xsl:when>
                <xsl:otherwise>NONE</xsl:otherwise>
            </xsl:choose>
        </xsl:variable>

log-xml-to-html.xsl.txt

[chrispy-snps]

@shaneataylor - how do you generate an XML log from the DITA-OT? Is this log processing something that you've integrated into the transformation via plugin, such that it runs in Oxygen, etc.?

[ekimbernow]

I definitely vote for having the option to report unresolved keyrefs as at least warnings if not errors.

I’m also very much in the “do not ever use @href as a fallback for a keyref” camp—you want fast and early failure for references.

I wouldn’t normally want an unresolved keyref to fail a build (except when I do, like for a final production build) but I would like to have it reported clearly as a warning or error in the log.

However…Oxygen’s map validation also clearly (and quickly) reports unresolved keys so, if you’re fastidious about your map validation, you shouldn’t have unresolved keys when you go to publish.

Or at least, it is realistically possible to have clean maps with all keys resolved, even in a context like ours at ServiceNow, with 10s of 1000s of keys and key references across scores of publications.

Not that we often achieve that ideal, but it is possible.

[chrispy-snps]

@ekimbernow - so you would want the severity of unresolved keys to be configurable, failing a final build but not a test build? Maybe a dita.xsl.messages plugin override of DOTJ048I could change its severity to "error", then the processing-mode parameter could be switched betweeen strict and lax to accomplish this? I wish I knew enough Java to implement a similar last() fix in MessageUtils.java to test this.

I agree that @href should never be used as a fallback for @keyref. If you want context-specific overrides, do it all in the key domain!

For us, Oxygen's validation falls in the "necessary but not sufficient" category. With the content reuse we have, it still sometimes happens that one writer commits something to Git that breaks another writer's content. If the publishing pipeline reliably detects issues, then these situations are detected before the content goes public.

(Starting in Oxygen v25.1, Validate and Check for Completeness considers branch filtering, which significantly improved the validation of our reused conditional content!)

[drmacro]

Yes, I would like key resolution failure severity to be configurable.

For non-production builds (i.e., the prevew builds we run three times a day for the entire ServiceNow Platform content), we would never want unresolved keys to fail a build, even if we were using "strict" processing (which we currently do not and probably couldn't because strict also fails on unresolved topicrefs, which we currently don't consider to be a build breaker).

For production builds we should be treating unresolved keys as build breakers, although we currently do not.

So having the option to use strict error handling but have unresolved keys not be errors would be useful.

In the production case, we might have a need to publish a time-sensitive update where we can't tolerate an unresolved topicref but can tolerate an unresolved key reference, so would want the option to make unresolved keys warnings rather than errors.

[kirkilj]

I agree with @ekimbernow that the severity should be configurable by release type, in Maven terms that would be SNAPSHOT, CANDIDATE, RELEASE. Other life cycle management systems have their own nomenclature. Snapshots are the builds that are triggered by daily pushes, candidates are stamped with a release candidate label and are typically for final internal reviews, and releases are for general public release.

I'd lean toward lax, recoverable messages that can funneled into focused and actionable log files for frequent daily pushes. An alternate XML OT log file format would open other possible ways to leverage the log info by tool chains so groupable, filterable, and sortable TODO lists could be generated and monitored.

Configurable rules could support the variety of workflows out in the field.

[sig-ozana]

I support the idea of severity of unresolved keys to be configurable.
However, there should be no error if scope="peer".

[chrispy-snps]

@sig-ozana - for peer maps, a solution to #4068 (DOTJ047I (unresolved key reference) should not be issued for references into declared peer maps) is needed.

I have an idea how the solution could be implemented. When I get time, I will post the idea there and see what the DITA-OT developers think.

[sig-ozana]

@chrispy-snps we don't use peer maps, but we do have cross-book references such as

See "Topic title" in Related Book.

These keys get resolved when we publish to WebHelp.
When we publish to PDF, we get the text without the hyperlink.

[chrispy-snps]

In the next DITA-OT monthly contributor's call, I will make a request for #4068 to be fixed in the Java code, which will then subsequently allow #3950 to be fixed in the XSLT code.