MathML Next, Content Selection, Web Annotation, and Service Providers

[AdamSobieski]

Introduction

Ideas and brainstorming are presented towards delivering a “Proof Web”, “Argument Web”, or “Inference Web” and towards delivering real-time fact-checking capabilities to end-users.

As envisioned, end-users could select document content and utilize context menus on those selections to access, in new tabs, relevant proofs, arguments, justifications, or fact-checking information about the selected content. Interestingly, contents in these provided documents could be subsequently selected and explored in a recursive manner. End-users would simply select natural-language and mathematical content and make use of context menus to access available first-, second-, and third-party services.

These features can be provided to end-users without document authors having to modify their documents or having to use special, new document-markup techniques. Document authors could, however, choose to make use of new metadata, markup, or style techniques to fine-tune or refine these technologies and features.

Use Cases

Artificial Intelligence

Outputs from artificial intelligence systems (e.g., answers, arguments, or explanations) could be examined and explored by selecting portions of interest and using context menus to invoke first-, second-, or third-party services.

Education

Learners could use any logic, mathematics, critical thinking, argumentation, philosophy, and law educational resources, e.g., digital textbooks, to explore reasoning supporting and opposing any document content interesting to them.

Public Interest

With the technologies under discussion, end-users could perform real-time fact-checking with respect to selections of arbitrary Web documents (see also: Wikifact).

Content Selection

End-users could select portions of documents with these selections containing hypertext, mathematics, and multimedia, e.g., pictures.

Hints provided using document metadata, markup, or style could allow content-selection algorithms to better guess users' intents across input modalities.

Context Menus

End-users could open context menus upon selections of content, e.g., by right-clicking. Presently, context menus in Web browsers include clipboard- and search-related items. Context menus could provide new items for end-users which invoke first-, second-, or third-party services with selected content. Web browsers could open and present content provided by invoked services in new tabs.

Ideas for new items for context menus include, but are not limited to, those pertaining to proofs, arguments, and justifications, and those pertaining to fact-checking.

Services

Items in context menus opened on selections of document content could invoke first-, second-, or third-party services which provide hypertext content for end-users in new tabs.

With respect to implementing services, artificial intelligence, natural-language processing, argument mining, and computer algebra system technologies could be of use. Also, some crowdsourced, e.g., wiki-based, third-party services can be envisioned.

Requests to these services would include the selections of content, e.g., utilizing the Web Annotations selector model. It recently occurred in another thread (see: #74) that Web Annotations selectors could be of use in these regards.

In some cases, the URL's of those Web documents containing selections would be useful for third-party services. Third-party services could, then, retrieve those documents containing the selections and, in this way, could process the documents to resolve any ambiguities occurring in individual selections of content. In other cases, those documents containing selections of content might be unavailable to a third-party service provider.

In particular for those cases where a third-party service provider cannot access the documents containing end-users' selections, supplemental context data could accompany the selections of content to enhance natural-language processing, interpretation, and disambiguation.

Responses from services would include or be comprised of hypertext documents, the contents of which could be identically selected and explored.

Web Annotation

As end-users make use of second- and third-party services upon selections of arbitrary Web documents' contents, at scale, these services could additionally provide end-users with Web-annotation-related functionalities.

With Web-annotation-related functionalities, end-users could, for instance, toggle visualizations of which portions of Web documents' contents already have explanations, proofs, arguments, justifications, or fact-checking-related information available.

Questions

With respect to remote procedure call requests for invoking first-, second-, or third-party services with selections of content and accompanying context data, could JSON representations of content selections from the Web Annotations selectors model be of use? If scenarios involving multimedia selections, e.g., pictures, are also desired, could multipart MIME with HTTP POST be of use?

Are selections which include mathematical content compatible with the Web Annotations selectors model?

What about expanding content selection techniques? End-users could, using a keyboard and mouse, press the CTRL key to make a selection comprised of multiple contiguous regions of hypertext. End-users could, using a keyboard and mouse, press the ALT key to make a selection inside of a larger selection of hypertext. So, end-users could make a selection, e.g., a paragraph containing necessary context to interpret a sub-selection, and then press the ALT key and make a sub-selection, e.g., a specific sentence or a phrase occurring within the paragraph.

How might document authors indicate whether to allow subparts of mathematical expressions to be selected or to prevent subparts of mathematical expressions to be selected? How might document authors, more broadly, express the desired behavior of content selection algorithms for document markup elements, e.g., in <p>, <span>, or <math> elements?

How might hints in document metadata, markup, or style allow content-selection algorithms to better guess users' intents across input modalities?

Could context menu items vary based upon the markup or style of selected document content? That is, could document markup or style, when occurring in selections, be of use for algorithmically determining which context menu items to display or enable?

How might end-users select and configure default service providers?

How might document authors provide recommendations with respect to first-, second-, and third-party service providers? How might document metadata be of use for indicating available and/or preferred first-, second-, and third-party service providers and for otherwise providing hints regarding context menu items for selections of hypertext and mathematical content?

How might document authors or software attach semantics to natural-language hypertext content, e.g., indicating the noun to which a pronoun refers in a sentence?

[NSoiffer]

What you describe sounds like some sort of research project that if implemented, would be an application on top of MathML. What is the ask for changes in MathML?

[AdamSobieski]

@NSoiffer, it would be convenient were there a specification, perhaps a Note, about the topics of content selection where selected content is or includes MathML. Envisioned deliverables would include the capabilities to select, to open context menus, as discussed above, and to clipboard hypertext content which includes mathematics across Web browsers without requiring JavaScript.

It appears that it is possible, today, to deliver some of the context-menu-based solutions discussed above for natural-language content. It appears that it is not yet possible to deliver these for mixed-content hypertext selections, selections of content which include both text and mathematics or both text and multimedia, e.g., pictures.

A case study is the Chromium-based Web browsers. The Chromium API for extending the browser with context menus does not yet include a means of extending and customizing context menus on mathematics content, nor does it yet support scenarios where MathML occurs in selected content.

In Chromium, context menus can be extended for: all, page, frame, selection, link, editable, image, video, audio, launcher, browser_action, page_action, and action. There isn’t yet an option for context menus for math content.

In this list of options, selection appears to be the appropriate option for mixed-content selections: selections of content which include both hypertext and math, or both hypertext and pictures. However, extensions can, presently, only access text-string-based representations of selections of content via the selectionText property on the OnClickData data structure.

I recently wrote a letter about these topics in a relevant mailing list, here, brainstorming about and feature-requesting the addition of a selection property which could provide extension developers with either a hypertext string, a DOM node, something from the Selection API, something from the Clipboard API, or a more sophisticated data structure or interface.

If there is interest, I could review and report here, in this thread, about more case studies, more Web browsers, in these regards.

Perhaps a specification Note would be useful for delivering to Web developers and end-users the capabilities to select, to open context menus on, and to clipboard mathematics content and mixed-content hypertext which includes mathematics.