Can no longer use conref in keydef metadata [DOT 2.3]

[raducoravu]

If in my DITA Map I define a key which has a keyword with a conref to another one defined in a topic:

     <keydef keys="company_name_full">
      <topicmeta>
        <keywords>
          <keyword
            conref="abc.dita#abc/company_full"/>  
        </keywords>
      </topicmeta>
    </keydef>

and in some topic I use the key:

<ph keyref="company_name_full"/>

it no longer gets expanded. Using DITA OT 1.8 the key got expanded.

[raducoravu]

Any hint in the specs about what stage should be applied first? Conref or keyref?
In what issue was the order of these stages changed?

[infotexture]

@raducoravu: This was changed for 2.0 with #1631. From the Release Notes:

The order of preprocessing stages has been optimized to move keyref processing before conref processing. This provides significant performance gains for source that uses multiple key references within reusable content.

[robander]

Any hint in the specs about what stage should be applied first? Conref or keyref?

Because no order was specified way back in 1.0 the spec has had great difficulty giving a mandated processing order for any of the features. Even newer features have trouble, because giving an order like "Between existing features A and B" establishes an order for A and B. An additional complication is that while I think the spec implies you should evaluate keyref before conref, that doesn't rule out "evaluating" as a way of simply moving the conref from the map context into the topic.

[raducoravu]

Somehow it seemed natural for me (speaking also as an end user) that the sample DITA content I gave should have worked, unfortunately even if I would use conkeyref instead of conref it will still not work.

Looking at what the specs says about conref-ing to content which has a keyref in it:

https://www.oxygenxml.com/dita/1.3/specs/archSpec/base/handling-xref-and-conref-within-topics.html

Key reference
When the address is a key reference, processors MUST resolve it relative to the location of the content reference (referencing context).

in my opinion this can no longer be accomplished because the keyrefs have been resolved in a previous stage. So having the keyref stage done before the conref stage does not comply with this paragraph in the specification.

In my opinion maybe a solution would be do have a single stage in which conrefs and keyrefs are resolved.

[jelovirt]

I think the cause may be that the key resolution module fails to verify a resolved topic has conrefs.

[drmacro]

My analysis is that the branch filtering and key scope features effectively require a specific order of processing and that implementations should do preprocessing in a "filtering aware" way. I captured this analysis in a dita-ot wiki page here: https://github.com/dita-ot/dita-ot/wiki/DITA-1.3-Preprocessing-Architecture-proposal

[jelovirt]

This is something that would require processing keyref and conref in a loop when you keep on resolving them until no keyref or conref remains. It would be a large architectural change.

[drmacro]

Resolution of direct conrefs should be done before any key-based processing is performed, otherwise there is no way to ensure you get the right answer.

Once key spaces are constructed it should then be possible to resolve key references in any context, including in the content of key defining-topicrefs.

[robander]

Ran into a variant of this issue today. The topic that uses the key actually has other uses of @conref, so we do run the conref process. But, paths are messed up -- when the key's link text (including a content reference) is resolved in another directory, the path to that content reference is not modified. Our exact scenario was:

• Common keys are defined in a sub-map, inside of a subdirectory subdir/
• Link text for some includes a @conref to pull in product names from the main directory: <keyword conref="../common.dita#common/prodname"/>
• The keys are pulled into the root map, and the content reference path is adjusted, so in the temp structure it becomes: <keyword conref="common.dita#common/prodname"/>
• The key is used from a topic within subdir/. But when the key is resolved inside subdir/, the content reference in the text is unchanged -- conref="common.dita#common/prodname"
• The build reports errors that subdir/common.dita does not exist

I tried changing the reuse to @conkeyref, but that didn't get resolved and didn't give me any errors - the result was just missing part of the link text, and the topic in the temp dir at the end of the build still had <keyword conkeyref="reuseTopic/prodname"/> inside of the link text.

[robander]

Results now with 3.2.1 differ in a way that makes this more difficult to resolve: when the link text is resolved in the key, all sub-elements are dropped. This does make sense in some ways, but not in others. When I use @keyref on <keyword>, resolving link text in a way that preserved sub-elements from link text would almost certainly result in out-of-context nested elements. But, if I resolve the same link text on <ph>, it would not. In both cases today, the markup is dropped. I'm guessing this was related to a change in the last couple releases but not sure which one, because my November test had the elements preserved -- but I don't know for sure what level of DITA-OT I tested, it could have been anything from 2.5.4 on as I'm using all of those in different places.

Additional update: it looks like this could / should work better with preprocess2, but the results are the same. Keys are resolved in the map => conref is resolved in the map => topics are pulled in => keys are resolved in the topics. So at the point we resolve these keys, the proper link text is available ... but it looks like the link text is already set as part of the job, so it isn't updated after conref resolution.

I'm not sure how best to resolve this. Leaving the sub-elements in (with corrected @conref) will result in some invalid markup. Another alternative I don't really care for would be to leave in some sort of namespaced or stub value on the sub-element, which leaves the conref + lets it get resolved but then strips out that container.

Test files I'm working with now using 3.2.1:
2420.zip

[raducoravu]

👍 We got one more report here:

https://www.oxygenxml.com/forum/viewtopic.php?p=55127#p55094

[raducoravu]

Attaching samples for both situations (conref inside key definition and conkeyref inside key definition). Both do not work and as @robander said no error is reported and the text is just not present in the output.

samples-conkeyref-in-keydef.zip
sample-conref-in-keydef.zip

[GlennEmerson]

We have a client experiencing this issue.
Keydef in a ditamap referenced from a bookmap.
Topic with a conref.
Conref source refers to the key using keyword:
<ph id="conrefSource"><keyword keyref="variableTextString"/> ispo facto lorem didum dadum wantum choo.</ph>

In the 2.4.3 OT, this resolves in PDF:
Text of Variable String ipso facto lorem didum dadum wantum choo.

In 3.2 OT, the keyref is dropped:
ipso facto lorem didum dadum wantum choo.

However, in Oxygen 21.1 with the 3.x OT packaged, using the XSL-FO or CSS flavors of PDF, the keyref does resolve.

Until this thread was pointed out to me, I thought it was something in the XSLT for the OT Plugin, especially since it worked in Oxygen 21.1 using the default Syncrosoft plugins. So, before I spend a lot of time and energy investigating the XSL for the plugin, Radu, is this something your team optimized in your Oxygen 21 package? If so, should I use that version of the OT in stead of the GitHub 3.2?
Thanks,
Glenn Emerson

[raducoravu]

@GlennEmerson this should not work even in Oxygen, I'm attaching a sample DITA project exemplifying my original problem.
flowers.zip

If your problem is different maybe you can also attach a small DITA project exemplifying it.

[antonyterrence]

I have a use case as well. We publish a document called Day One Impact Overview. This document essentially lists all features (product-wise) that impacts customers on the day 1 of the release. Each chapter in the map represents a product. The introduction topic contains a standard text and only the product names vary for each chapter. We want to create one single topic and utilize the DITA key scope to inject the relevant product name for each chapter during run time. Currently, we have stored all product names as conrefs in a ware house file using the ph element.

Since keyref requires a topic ID and since conrefs in keys are not expanded, I was experimenting to use conkeyrefs by storing the product names as Key Definitions. I have a list of key definitions. Each definition contains a product name. Then I defined another key called product using key scopes. The product key does conkeyref to a key in the product list. The product key in the intro topic expands correctly for each chapter in Oxygen. Meaning the reused intro topic now shows relevant product name depending on the chapter it is used in. However, the key does not expand at all in the output. I am attaching a sample.

bm_product_names.zip

If keys using conrefs as in Radu'x example are expanded, we do not have to change our current warehouse file. Or if it can at least expand the conkeyrefs, that would be a great solution.

[sh511]

I have a use case similar to GlennEmerson. I am trying to insert text based on a keyscoped keydef. Oxygen XML Editor 22.1 can resolve the reference to a keyref that is part of a keyscope (other than root) but DITA-OT looks for it in the root scope and I get the following error:

[DOTJ047I] Unable to find key definition for key reference "myvalue" in root scope. The href attribute may be used as fallback if it exists

I have attached a simple project that generates the error. When the keyref is not part of a conref, it is resolved correctly (proper keyscope). I am new to DITA but I believe that I used conref and keyref properly.

my-conref-trouble.zip

[raducoravu]

Pasting here some of the advice I gave via email on the Oxygen Support email address to @sh511:

As the "included_step.dita" is not referenced anywhere in the DITA Map, the publishing does not know in what keys context it is.

Maybe you can refer to it in the DITA Map in that specific key scope:

 <topicref href="topics/included_step.dita" processing-role="resource-only" keyscope="myscope" />

As a best practice you should refer to all your topics containing reusable content in the DITA Map using the processing-role="resource-only" attribute which states that it does not contribute content to the table of contents and it's just a resource indirectly used. This gives a better indication to the publishing engine as to what resource are indirectly used.

You can also declare the keyscope attribute higher on a parent element like this:

<topicgroup keyscope="myscope" >
   <topicref href="topics/main_text.dita" format="dita" type="topic"></topicref>
     <topicref href="topics/main_task.dita" format="dita" type="task"></topicref>
     <topicref href="topics/included_step.dita" processing-role="resource-only" />
 </topicgroup>

to avoid declaring it on each topicref.

But there are also some bugs in the publishing engine related to conrefs and key scoped key references so thinks might not work precisely as you want all the time, you will need to check all your changes against the publishing engine.

In your case I would have probably used conkeyref instead of conref, also declare the key scope only once and add all topicrefs inside the key scope, please see the sample project with my changes applied.
my-conref-trouble 2.zip

[jelovirt]

Reproduced the original case. If you reference abc.dita in the map, the output is as expected. If abc.dita is only in the @conref attribute, conref is not processed.

<keydef keys="company_name_full">
  <topicmeta>
    <keywords>
      <keyword conref="abc.dita#abc/company_full"/>  
    </keywords>
  </topicmeta>
</keydef>
<topicref href="abc.dita" processing-role="resource-only"/>

[raducoravu]

@jelovirt oh so this is also related to the fact that with "preprocess2" all topics are now required to be referenced in the DITA Map.

[jelovirt]

#4258 is a partial fix to this issue. You no longer need to reference conref targets separately with a <topicref>. It doesn't fix this issue completely. A workaround is Instead of

<keydef keys="company_name_full">
  <topicmeta>
    <keywords>
      <keyword conref="abc.dita#abc/company_full"/>  
    </keywords>
  </topicmeta>
</keydef>

you need to use

<keydef keys="company_name_full">
  <topicmeta>
    <keywords>
      <keyword><text conref="abc.dita#abc/company_full"/></keyword>  
    </keywords>
  </topicmeta>
</keydef>

[raducoravu]

@jelovirt maybe you can still close this issue as a partial fix. There have been quite a number of comments on this issue but the intent of the original issue has been about conrefs placed in keydef.
I do not understand why the case with the conref placed on the keyword does not work though.

[raducoravu]

Thanks @jelovirt