Resolve links in Markdown text. Support mistune 2.x

[dairiki]

Here's another giant PR. This one refactors lektor.markdown with two main goals:

• Clean up and fix a number of bugs with link and image resolution in Markdown text.
• Support using the latest version of mistune (==2.0.2), while also maintaining support for the previous major version of mistune (0.9.4).

Issue(s) Resolved

Fixes #313
Fixes #966
Fixes #950
Supercedes #944
Supercedes #685
Supercedes #470

Related Issues / Links

Doc PR: lektor/lektor-website#338

Description of Changes

• Wrote at least one-line docstrings (for any new functions)
• Added unit test(s) covering the changes (if testable)

• Link to corresponding documentation pull request for getlektor.com

---

Link and image URL resolution via the Lektor DB

Previously any URLs in Markdown text which were not external links were interpreted as URL paths relative to either the Lektor site root (if the path starts with a /) or the page containing the markdown text (if the path does not start with a /). As discussed in #966, this causes issues, e.g.:

• when linking to attachments, when alternatives are in use
• when linking to pages or attachments that use a custom slug

This PR changes the default behavior so that an attempt is made to resolve links as Lektor db paths to a source object. If that resolution is successful, then the URL to that source object is used. If that Lektor db lookup fails, then fall back to interpreting the path as a URL path (as is done now.)

Note that this new behavior is exactly the same as is currently implemented by Lektor’s url jinja filter.

Since, in many cases, resolution via the DB produces the same result as just interpreting the path as a URL path, while this is a backward-incompatible change, most of the time it will not make a difference. And when it does, I think, that most of the time, the new behavior is what was probably wanted in the first place.

New resolve_links markdown field config option

This PR adds a new field configuration option for markdown fields. This option can take one of three values:

• resolve_links = never
  Never resolve links to Lektor source objects. This is the old (current) behavior, equivalent to prefixing all paths with '!'.
• resolve_links = when-possible
  This is the new default behavior. When possible, links are resolved to Lektor source objects, then the URL to those source objects is used. When the resolution to source object fails, the links are interpreted as URL paths.
• resolve_links = always
  Resolve links to Lektor source objects, then use the URL of those source objects. If that resolution fails, an error is raised.

This option may be set in the field definition section of a model.ini file. E.g. to restore the legacy behavior to a specific body field of your pages, you could put in models/page.ini:

...
[fields.body]
label = Body
type = markdown
resolve_links = never
...

---

Cleanup and of SourceObject.url_to.

This PR refactors SourceObject.url_to. I think the new code is much more readable than the old version.

Change of behavior with absolute=True

It does make one behavior change. Previously if absolute=True was passed, the SourceObject.url_to would return an "absolute" URL path (one which starts with a /), but that path was relative to the site root configured in the project file (if any). The PR changes that so that a real absolute path is returned (relative to the root of the webserver). E.g., if in the project file a site URL was set:

[project]
url = http://example.org/some/path/

SourceObject.url_to("/", absolute=True) used to return "/". This PR changes things so that it returns "/some/path/".

It can be argued that the previous behavior was a bug. Consider writing — not unreasonably — in a template:

See <a href="{{ this.get('subpage').url_to(absolute=True) }}">this sub-page</a>.

If there is a [project]url set with a non-empty path, that link will be a broken one under the current behavior.

Addition of resolve and strict_resolve parameters

This PR also adds two new parameters to SourceObject.url_to: resolve, and strict_resolve.
The same new parameters are also added to Context.url_to and to the url jinja filter.

Passing resolve=False will prevent any attempt to resolve the path as a db path. This is equivalent to what happens currently if one prefixes the path with '!'.

Passing strict_resolve=True will cause an error to be raised if the path can not be resolved as a db path.

Support for lektor: URL scheme

As described in #966, SourceObject.url_to is modified to recognize the new lektor: URL scheme. Prefixing a path with lektor: is now essentially equivalent to passing the new strict_resolve=True parameter (see above).

With the addition of the strict_resolve parameter, the addition of the lektor: URL scheme is probably overkill. At this point I plan on removing it from this PR before merging have removed it from this PR.

---

Support for mistune 2.x

This also adds support for the latest version of mistune.

Note that it is unlikely that existing Lektor plugins that further customize Lektor's Markdown rendering will “just work” if running under the new version of mistune. (Most such plugins work by subclassing the mistune Renderer class. The API to that class has changed in the latest version of mistune. The arguments to the constructor of the mistune parser have changed as well.)

Lektor plugins that work now should continue to work if mistune is pinned to the old version (mistune<2.)

(A little more thought should probably be given to the issue of making life under multiple versions of mistune easy for plugins.)

[dairiki]

@yagebu @xlotlu (and any other interested parties):

I would like to merge this PR soon. However, since it does introduce a backward-incompatibility — namely that links in markdown text will be resolved via the Lektor DB rather than interpreted only as URLs (see the main PR description, above) — I want to double-check that there are no strong objections before doing so.

Also, would appreciate opinions on whether this is a significant enough change to warrant a major version bump (to 4.x) rather than just a minor version bump (to 3.4.x).

[yagebu]

LGTM :)

I'd be happy to ship this with 3.4, since this is, where incompatible, probably closer to the expected/intended behaviour (as evidenced by the long chain of issues and PRs that this PR would finally close :))