linking to terms

[amitu]

Feature: terms

Status: Discarded

Owner: @Heulitig
Started On: 10th Aug 2022
Feature: Terms

Done

• Refactor Common arguments to Common Kernel arguments and update the documentation in ftd.dev
  Merged PR - renaming common attributes to common kernel attributes ftd.dev#25
• Create a documentation for universal attributes (which we currently have)
• Delete overlapping attributes in common and universal attributes and shift those to universal.
  Merged PR - Adding more universal attributes ftd.dev#27
• Refactored default arguments to universal arguments (in code)
• Add term as universal property.
• Tests to be done on the below sample FTD code to ensure its working

 -- ftd.text foo: $term

-- foo:
term: hello

---

• Need to make test blocks to assert these universal attributes properly. [In Progress]
  Current PR (in progress) - adding term as universal attribute fastn#364
• Update term in ftd.dev under universal attributes.
• Implement StuckOnTerm in ftd. Test this by creating a hard coded term map
• Construct Term Map on package in fpm
• Implement StuckOnTerm which will use the actual Term map constructed from package

---

Note: We are keeping the crux of this feature, but are doing it via globally unique id.

We currently support linking using markdown link syntax, eg [text](/target-link/). But when writing definitions it can get tedious, especially if definitions can move from one file to another.

What if we supported something like this:

-- ft.markdown:
term: adjusted call operand

An `adjusted call operand` is a :t:`call operand` with possible [auto 
refs](term:auto referencing) or [auto referencing] or {term: auto referencing}.

Any FTD component can have term defined. Heading, markdown, code block, table, image etc.

We have shown three possible syntaxes. I am leaning towards the second and third, third in most cases, second if you want to use a different text while linking other than the name of the term. Fourth is in line with markup feature of ftd, though it is special case and does not easily allow changing the link text. Least favourite option.

What we will have to do is convert the body of ft.markdown to something like this:

<p id="unique-id-for-this-para">
An `adjusted call operand` is a [`call operand`](/doc-for-call-operand/#doc-operand-para-id)
with possible [`auto dereferencing`](/doc-for-auto-dereferencing/#auto-dereferencing-para-id).
</p>

We will have to maintain a map of each term defined using :dt:, with the document and the paragraph id where the term is defined. With this map available we can translate all :t: to links as shown in example above.

Manual Paragraph ID

In this solution author is supposed to explicitly give a unique id to every markdown block. If the id is not unique in the document (not across documents), then it is an error (this uniqueness check should be part of ftd itself as id is a ftd provided attribute).

Tomorrow our editor can detect if you have used :dt: in any markdown block and automatically generate a unique id, and update the source code with it, if id is not already present. This can make the life of author a bit easier.

Definitions Can Be In Any File: Two Pass Parsing

The terms we are using can be defined in another file in the FPM package. This means we have to do a quick pass, a cheaper ftd parser which only looks for term definition (only p1 level parser) in all files in current and imported packages, before the actual interpreter starts.

fpm will implement this during import pass

We can not implement this using $processor$ feature, else every markdown block will have to have $processor$ line added, which is tedious.

So fpm will scan the document before passing it to ftd for processing by converting term to id, and converting all instances of [term] with proper [term](<actual-link>).

Cross Package Linking

If a term is defined in another package, the link format would be [package-name:term], package name is full package name, eg [fpm.dev:user-group]. This is a unique link, which can be used in any package.

Documenting Code

One problem we have when documenting code is, if we write lot of code comments, the information in those comments are lost to people who do not have access to that code. So for example for your company you are defining a term, the source code is not the proper place for definition of that term. It must be in a place available to everyone, editable by everyone, so people can give explanations, examples etc, to help improve the definition of that term and clarify the term etc.

In fact even rules like we only accept max 20 char username, must not be in a source code file as a comment, but it should be given a unique term, eg username-max-20-chars or username-max-length. A lot of our code deals with rules and definitions, and if we create terms, and just mention the terms in source code as comment, and if our site support term search, it becomes trivial for a person to look up definition of terms from source code.

It is even possible to create editor plugins, which will accept terms in some syntax, and show a tooltip with webkit embedded that shows the page for the term (maybe even scroll to the exact place in the page where the term is defined).

Term Only Search

We can create a client search engine powered only by (sitemap augmented with (term: t + dt data)), as against a full text search. The term/reference/url/title is information dense. So index size would be really small.

term-map.json

Like sitemap.xml we can create a file that lists the data of our fpm.sitemap augmented with the term definitions and term references. This file can be used by search engines, our in site client search, the editor plugin, maybe we can do a terminal based search also etc etc.

Term Review Tool

With term-map.json, we can create a tool that asks the developers if the term definition is still accurate. We can do this on a periodic basis, say each term's definition should be re-evaluated every 6 months, so we know if we are using terms correctly, or if they are to be updated.

We can do it on document basis also, but doing it on term basis is more fine grained.

[amitu]

Construct The Terms Map

This is a json file that will be constructed before any document can be rendered. This can be done while constructing an instance of fpm::Config object.

{
    "<term>": "<document where it is defined>"
}

If we invert it:

{
    "<document>": ["<list>", "<of>", "<terms>"],
}

Term is defined like this, say this is foo/bar.ftd

-- foo:
term: the term

1. Scan all ftd files in the package.2.
2. Grep for any line starting with term:

{
    "/foo/bar/": ["the-term"]
}

You will have to convert document name to id.

You will also have to use the slugify to convert the term to the-term.

struct Config {
    // existing fields
    terms: HashMap<String, String>
}

JSON should be inverted, but in rust it should be forward map from term to document. We are going to do term lookup, we will have term, and we will need to find the document in which term is defined, so forward map is good here, else we will have to do iterate over the entire map to find the document.

We prefer inverted map because it is smaller for transfer over internet. We will in future send this JSON to browser for client based term search feature. This is not in the scope of this task.

Once the term map is constructed we have to replace the links.

Replace Links

The syntax are going with is [term: <term>] if you want to use the term as the link text as well. If you want to change link text you can use [<whatever you want>](term: <term>).

In fpm::Library.get() function, modify the line that says: return get_for_package(name, packages, self).await; and change it to return replace_links(get_for_package(name, packages, self).await, self.config.terms);

Example input:

-- ft.markdown:

We are using [term: the term] here.

Output:

-- ft.markdown:

We are using [the term](/foo/bar/#the-term) here.

[amitu]

We are discarding this approach and using globally unique id but keeping the term linking syntax here.