[feature] intent-isa attribute #426

dginev · 2022-09-19T15:03:06Z

I want to propose a third intent attribute that solves two needs:

Enable the applications traditionally possible in HTML via aria-describedby
Provide additional fine-grained contextual information, as a more precise alternative to specifying a subject area.

For now I will use a simple example: Imagine a group theory paper, where we have a single letter variable B and an expression |B| where there is an unknown operation via the vertical bars. It is clear to a reader, from the textual context, whether B is a group, or a set. B is meant to be simply read as cap-B, so no intent annotation is needed. Can we annotate B in a way where:

the vertical bars operation becomes clear?
the AT user can ask for auxiliary information on what B represents?

My proposal is:

<mrow> <mo>\|</mo> <mi intent-isa="group">B</mi> <mo>\|</mo> </mrow>	<mrow> <mo>\|</mo> <mi intent-isa="set">B</mi> <mo>\|</mo> </mrow>

AT can use this information to infer:

"circumfix vertical bars over var:set" → intent="cardinality(B)"
"circumfix vertical bars over var:group" → intent="order(B)"

In addition, an AT user can use an auxiliary keyboard shortcut to request the underlying "set" and "group" annotation, in e-learning applications. This should be very useful in long expressions, where object types may be hard to keep track of.

Why is this needed if we can add intent to the operation mrow?

Technically it isn't - annotating the mrow as usual will do the trick:

<mrow intent="order($B)"> <mo>\|</mo> <mi arg="B">B</mi> <mo>\|</mo> </mrow>	<mrow intent="cardinality($B)"> <mo>\|</mo> <mi arg="B">B</mi> <mo>\|</mo> </mrow>

It may come down to authoring trade-offs. If a text (or scope of a text - a chapter, a section) uses a variable name consistently, it could be the more intuitive annotation for an author to specify the types of their variables once, as the text begins. Compare that to having to examine every single equation where vertical bars are used, and annotating each individual subexpression.

This is an early idea, and discussion is most welcome - if the direction appears useful the final form of such an attribute may be quite different than the one originally proposed here. I would like to thank Neil for the discussions around "subject area", that helped me crystalize this alternative direction.

The text was updated successfully, but these errors were encountered:

dginev · 2022-09-19T15:14:58Z

It also just occurs to me that since the values of this attribute will also be NCNames, one could make a case for staying within the single intent attribute framework and instead devise an @isa hint.

I am not sure if that increases or reduces clarity, but it would certainly make large intent expressions more powerful. I'm also unsure if such extra power is a net benefit or not.

Here is the example in question realized via hint:

<mrow> <mo>\|</mo> <mi intent="B@isa(group)">B</mi> <mo>\|</mo> </mrow>	<mrow> <mo>\|</mo> <mi intent="B@isa(set)">B</mi> <mo>\|</mo> </mrow>

davidcarlisle · 2022-09-20T15:27:42Z

I think something along those lines could be useful. Do you think this could also make a more usable annotation for tables?
eg #402 we were struggling with

 <mtable arg="table"
     intent="system-of-equations($row1,$row2,equals($row3lhs,_($row3rhs,$row4rhs)),
             $row5,$row6,$row7,$row8,$row9,$row10,$row11,$row12)">

where you are needing to copy the row structure into the intent, to preserve the functional form but really I (think) I just want

 <mtable arg="table"
     intent-isa="system-of-equations">

to give the AT a hint not to read it as an array but otherwise infer the structure from the markup.

On the syntax, the @isa form is tempting but

B@isa(group) the parens bind to isa but in factorial@postfix(x) the parens bind to factorial which is I am sure resolvable in the grammar but might end up being confusing but @isa[group] or even @isa group would be alternatives (perhaps...)

davidfarmer · 2022-09-20T15:31:26Z

How would you do ***@***.***(group) if you wanted tbe B to be a reference?

dginev · 2022-09-20T15:55:59Z

So the hint variation is getting more attention so far? Interesting!

A more careful suggestion on my part would be:
An "isa" relationship is a simple typing judgement. And there is a very standard syntax in most well-typed programming languages for assigning types to variables, via a colon. I suspect almost every programmer has encountered that, even if they don't program a well-typed language as their primary choice.

Hence, B:group and B:set.
To imagine a larger expression with more hints (sorry, a bit of a contrived example for now):

<mrow intent="union:set-operation@infix(B:set,C:set)">
  <mi>B</mi><mo>∪</mo><mi>C</mi>
</mrow>

However, in a parallel thread Moritz filed a complaint that the intent attribute already seems rather complex to him, so I will also tag along the extra attribute variant, for completeness. Using an attribute somewhat forces the hand of a remediator to annotate as low as possible in the subtrees, so that they have more available nodes on which to attach "-isa" attributes.

<mrow>
  <mi intent-isa="set">B</mi>
  <mo intent-isa="set-operation" intent="union">∪</mo>
  <mi intent-isa="set">C</mi>
</mrow>

There seem to be trade-offs that make either choice interesting.

dginev · 2022-09-20T16:00:21Z

@davidcarlisle

 <mtable arg="table"
     intent-isa="system-of-equations">

is interesting, but kind of unpredictable?
Say we have a system of equations using 5 table <mtr> rows. But we only have 3 complete equations. That could mean:

Eq 1 split on rows 1 and 2; Eq 2 on row 3; Eq 3 split on rows 4 and 5
Eq 1 on row 1; Eq 2 split on rows 2 and 3; Eq 3 split on rows 4 and 5
Eq 1 on row 1; Eq 2 on row 2; Eq 3 split on rows 3, 4 and 5
...

So the hint is nice (and maybe even nice to speak as an auxiliary piece of information?), but not self-sufficient to recover the fragments into complete material. I think?

davidcarlisle · 2022-09-20T16:07:52Z

is interesting, but kind of unpredictable?

Yes although on the last call (which I think you were not on) we went round in circles again and basically decided to punt on making a "nicer" alternative here at this stage, as no one liked any of the proposed changes (including people making proposals) so I thought I'd at least cross ref to this issue to see if it adds anything to break out of the circle.

So the hint is nice (and maybe even nice to speak as an auxiliary piece of information?), but not self-sufficient to recover the fragments into complete material. I think?

Yes exactly.

NSoiffer · 2022-09-29T03:12:45Z

I think at one time, @dginev proposed (and prototyped) to use aria-describedby for adding type info ('x is a vector', etc). There is also a proposed aria-description. Whereas aria-label has baggage in regards to braille, I don't believe aria-description has any baggage. Hence, it remains open as a possible attribute to use. However, intent-isa makes it obvious that this related to intent and so may be a better name (if we are looking at using a second attribute).

Update (after call today): maybe they serve different purposes with "isa" being concise and meant for software; "aria-description" is likely more verbose and meant for humans.

NSoiffer · 2022-09-29T18:45:42Z

In the call today, I mentioned that in MathJax, \ce{Na+(aq)Cl-(aq)} turns into mess (below). Below is an attempt to add intent and intent-isa. Some things to note:

there is no mrow around the state (aq) on which to attach an "isa". I somewhat artificially added it to the aq. This is like adding an intent-isa to the mfrac part of a binomial (which also seems wrong).
the charge on the element makes it an ion, but there is no positional inference that is easily made. Instead I made up ChemicalCharge since the +/- are kind of dangling and likely need some sort of hint
there is no mrow around the "Na+" or "Na+(aq)" parts, so everything gets crammed onto the toplevel mrow
I put both intent-isa and intent on the toplevel mrow, but probably the former can be inferred from the later. I'm assuming ChemicalFormula would be a core name.

Here's the result -- I'm not claiming this is good...

<math>
<mrow intent-isa="ChemicalFormula" intent=ChemicalFormula( $elem1, $charge1, $state1, $elem2, $charge2, $state2 )",>
  <mrow>
    <mi intent-isa="ChemicalElement" arg="elem1">Na</mi>
  </mrow>
  <msup>
    <mrow>
      <mrow>
        <mpadded width='0'>
          <mphantom>
            <mi>A</mi>
          </mphantom>
        </mpadded>
      </mrow>
    </mrow>
    <mrow>
      <mo intent-isa="ChemicalCharge" arg="charge1">+</mo>
    </mrow>
  </msup>
  <mo stretchy='false'>(</mo>
  <mrow>
    <mi intent-isa="ChemicalState" arg="state1">aq</mi>
  </mrow>
  <mo stretchy='false'>)</mo>
  <mrow>
    <mi intent-isa="ChemicalElement" arg="elem2">Cl</mi>
  </mrow>
  <msup>
    <mrow>
      <mrow>
        <mpadded width='0'>
          <mphantom>
            <mi>A</mi>
          </mphantom>
        </mpadded>
      </mrow>
    </mrow>
    <mrow>
      <mo intent-isa="ChemicalCharge"  arg="charge2">&#x2212;</mo>
    </mrow>
  </msup>
  <mspace width='0.111em'></mspace>
  <mo stretchy='false'>(</mo>
  <mrow>
    <mi intent-isa="ChemicalState" arg="state2">aq</mi>
  </mrow>
  <mo stretchy='false'>)</mo>
</mrow>
</math>

dginev · 2022-10-05T17:42:42Z

I had an action item from last Thursday to try a couple of examples using MathML produced by
MathJax, then hand-annotating with "intent" and "isa".

raw column MathML via the live mathjax demo
for the annotated columns I wrote the markup myself, by hand
in cases where I pack intent-isa into the main intent attribute, I use the colon : syntax discussed in previous comments.

TeX: a + \left| S \right|

Easier case, mathjax generates an <mrow> for the fenced operator, thanks to the \left+\right hints in TeX.

raw

annotated (minimal)

annotated (full)

<math>
  <mi>a</mi>
  <mo>+</mo>
  <mrow data-mjx-texclass="INNER">
    <mo data-mjx-texclass="OPEN">|</mo>
    <mi>S</mi>
    <mo data-mjx-texclass="CLOSE">|</mo>
  </mrow>
</math>

<math>
  <mi>a</mi>
  <mo>+</mo>
  <mrow data-mjx-texclass="INNER">
    <mo data-mjx-texclass="OPEN">|</mo>
    <mi intent-isa="set">S</mi>
    <mo data-mjx-texclass="CLOSE">|</mo>
  </mrow>
</math>

<math>
  <mi intent-isa="integer-variable">a</mi>
  <mo intent="plus" intent-isa="arithmetic-operator">+</mo>
  <mrow intent="cardinality($s)" data-mjx-texclass="INNER">
    <mo data-mjx-texclass="OPEN">|</mo>
    <mi arg="s" intent-isa="set">S</mi>
    <mo data-mjx-texclass="CLOSE">|</mo>
  </mrow>
</math>

TeX: a + |S|

Harder case, no fine-grained <mrow> structure.
If you horizontally scroll to the third snippet, I also added an example of using a single top-level intent attribute that contains all information. It is a mouthful, but manages to pack a lot of information.

raw

annotated (fine-grained)

annotated (single attr)

<math>
  <mi>a</mi>
  <mo>+</mo>
  <mrow data-mjx-texclass="ORD">
    <mo stretchy="false">|</mo>
  </mrow>
  <mi>S</mi>
  <mo stretchy="false">|</mo>
</math>

<math intent="$op($a,cardinality($s))">
  <mi arg="a" intent-isa="integer-variable">a</mi>
  <mo arg="op" intent="plus" intent-isa="arithmetic-operator">+</mo>
  <mrow data-mjx-texclass="ORD">
    <mo stretchy="false">|</mo>
  </mrow>
  <mi arg="s" intent-isa="set">S</mi>
  <mo stretchy="false">|</mo>
</math>

<math intent="plus:arithmetic-operator@infix(a:integer-variable, cardinality(S:set))">
  <mi>a</mi>
  <mo>+</mo>
  <mrow data-mjx-texclass="ORD">
    <mo stretchy="false">|</mo>
  </mrow>
  <mi>S</mi>
  <mo stretchy="false">|</mo>
</math>

Determinant of an explicit matrix (in mtable)

One possible TeX:

\begin{vmatrix}
     a_{11} & a_{12} \\
     a_{21} & a_{22} \\
\end{vmatrix}

Here we have to pay attention to the matrix elements, their scripted indexes, and the outer vertical bars that vmatrix renders with.

raw

annotated

<math display="block">
  <mrow data-mjx-texclass="INNER">
    <mo data-mjx-texclass="OPEN">|</mo>
    <mtable columnspacing="1em" rowspacing="4pt">
      <mtr>
        <mtd>
          <msub>
            <mi>a</mi>
            <mrow data-mjx-texclass="ORD">
              <mn>11</mn>
            </mrow>
          </msub>
        </mtd>
        <mtd>
          <msub>
            <mi>a</mi>
            <mrow data-mjx-texclass="ORD">
              <mn>12</mn>
            </mrow>
          </msub>
        </mtd>
      </mtr>
      <mtr>
        <mtd>
          <msub>
            <mi>a</mi>
            <mrow data-mjx-texclass="ORD">
              <mn>21</mn>
            </mrow>
          </msub>
        </mtd>
        <mtd>
          <msub>
            <mi>a</mi>
            <mrow data-mjx-texclass="ORD">
              <mn>22</mn>
            </mrow>
          </msub>
        </mtd>
      </mtr>
    </mtable>
    <mo data-mjx-texclass="CLOSE">|</mo>
  </mrow>
</math>

<math display="block">
  <mrow intent="determinant($m)" data-mjx-texclass="INNER">
    <mo data-mjx-texclass="OPEN">|</mo>
    <mtable arg="m" intent-isa="matrix" columnspacing="1em" rowspacing="4pt">
      <mtr>
        <mtd>
          <msub intent="index(a,1,1)" intent-isa="matrix-entry">
            <mi>a</mi>
            <mrow data-mjx-texclass="ORD">
              <mn>11</mn>
            </mrow>
          </msub>
        </mtd>
        <mtd>
          <msub intent="index(a,1,2)" intent-isa="matrix-entry">
            <mi>a</mi>
            <mrow data-mjx-texclass="ORD">
              <mn>12</mn>
            </mrow>
          </msub>
        </mtd>
      </mtr>
      <mtr>
        <mtd>
          <msub intent="index(a,2,1)" intent-isa="matrix-entry">
            <mi>a</mi>
            <mrow data-mjx-texclass="ORD">
              <mn>21</mn>
            </mrow>
          </msub>
        </mtd>
        <mtd>
          <msub intent="index(a,2,2)" intent-isa="matrix-entry">
            <mi>a</mi>
            <mrow data-mjx-texclass="ORD">
              <mn>22</mn>
            </mrow>
          </msub>
        </mtd>
      </mtr>
    </mtable>
    <mo data-mjx-texclass="CLOSE">|</mo>
  </mrow>
</math>

For this example, one could imagine that omitting the top-level intent="determinant($m)" would still allow an advanced AT implementation to infer/guess that this is a determinant notation, by leveraging the <mtable intent-isa="matrix" ...> information.

Minor aside: wikipedia claims that "matrix-entry" and "matrix-element" are both used in English practice, so it appears the aliasing considerations would apply to an "isa" attribute as well.

These can (and should) be refined as we keep discussing here, but I think they are a good start at seeing the structural challenges when the MathML markup doesn't offer enough tags on which to set attributes.

P.S. @NSoiffer - if possible, please also use the "dashed-lowercase" convention for the isa values, as with intent, rather than CamelCases or some other. Thanks!

dginev · 2023-02-20T16:52:45Z

Moving a discussion from David's comment in #436.

In my eyes the discussion about evolving intent="matrix" into something clear-cut overlaps the isa discussion.
Over a series of conversations with Neil (and others) I am now fond of using the colon syntax : for isa, without an additional attribute.
A new piece that may be helpful for clarity is some keyword (or symbol) for self-reference to "the current node". Two tentative suggestions here are $this or *.

So, one isa/type take on a matrix annotation would be:

<mtable intent="$this:matrix"> ... </mtable>

The additional clarity brought by isa is that matrix can be used for the "accessible description" of the mtable element, available as secondary information, usually via some keyboard shortcut.

If instead matrix was meant to determine the primary generated speech of that presentation subtree, then using it in the intent slot is a better fit. As David C originally suggested:

<mtable intent="matrix($this)">...</mtable>

I think that this can bring a well-defined difference in behavior, where matrix in the intent value slot is expected to be a part of the "accessible name" and not the description.

davidcarlisle · 2023-02-20T17:03:36Z

@dginev I'm still not keen on a pre-declared $this prefering an explicit arg=this or * but really that's getting down to minor notational preferences. I agree with the two basic forms you describe above and think this should help solve several of the remaining intent issues,

davidcarlisle · 2023-03-23T18:19:50Z

isa now merged with properties

dginev added enhancement New feature or request intent Issues involving the proposed "intent" attr labels Sep 19, 2022

dginev mentioned this issue Nov 19, 2022

Make the sample presentation of Strict Content use intent #284

Open

NSoiffer mentioned this issue Jan 12, 2023

Chemistry and semantics/AT #93

Closed

dginev mentioned this issue Feb 7, 2023

include (or not) a sample-set of default conversion from plain-MathML to MathML-with-intent #433

Open

dginev mentioned this issue Feb 21, 2023

Syntax of isa and hint #436

Closed

davidcarlisle closed this as completed Mar 23, 2023

dginev mentioned this issue Mar 26, 2023

Must core intent work as a literal expression? #455

Closed

dginev mentioned this issue Jul 28, 2023

List of core intents #432

Open

dginev mentioned this issue Apr 17, 2024

Clarify property use as descriptions #493

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feature] intent-isa attribute #426

[feature] intent-isa attribute #426

dginev commented Sep 19, 2022

dginev commented Sep 19, 2022

davidcarlisle commented Sep 20, 2022

davidfarmer commented Sep 20, 2022 via email

dginev commented Sep 20, 2022

dginev commented Sep 20, 2022

davidcarlisle commented Sep 20, 2022

NSoiffer commented Sep 29, 2022 •

edited

Loading

NSoiffer commented Sep 29, 2022 •

edited

Loading

dginev commented Oct 5, 2022

dginev commented Feb 20, 2023

davidcarlisle commented Feb 20, 2023

davidcarlisle commented Mar 23, 2023

[feature] intent-isa attribute #426

[feature] intent-isa attribute #426

Comments

dginev commented Sep 19, 2022

Why is this needed if we can add intent to the operation mrow?

dginev commented Sep 19, 2022

davidcarlisle commented Sep 20, 2022

davidfarmer commented Sep 20, 2022 via email

dginev commented Sep 20, 2022

dginev commented Sep 20, 2022

davidcarlisle commented Sep 20, 2022

NSoiffer commented Sep 29, 2022 • edited Loading

NSoiffer commented Sep 29, 2022 • edited Loading

dginev commented Oct 5, 2022

dginev commented Feb 20, 2023

davidcarlisle commented Feb 20, 2023

davidcarlisle commented Mar 23, 2023

NSoiffer commented Sep 29, 2022 •

edited

Loading

NSoiffer commented Sep 29, 2022 •

edited

Loading