Add a primitive lexer for Typst

[jvoisin]

This should close #2558.

I tried to add tests but:

jvoisin@home 20:49 (tyspt) ~/dev/pygments tox -- --update-goldens ~/text.typ 
Traceback (most recent call last):
  File "/usr/bin/tox", line 33, in <module>
    sys.exit(load_entry_point('tox==3.21.4', 'console_scripts', 'tox')())
  File "/usr/lib/python3/dist-packages/tox/session/__init__.py", line 44, in cmdline
    main(args)
  File "/usr/lib/python3/dist-packages/tox/session/__init__.py", line 65, in main
    config = load_config(args)
  File "/usr/lib/python3/dist-packages/tox/session/__init__.py", line 81, in load_config
    config = parseconfig(args)
  File "/usr/lib/python3/dist-packages/tox/config/__init__.py", line 280, in parseconfig
    ParseIni(config, config_file, content)
  File "/usr/lib/python3/dist-packages/tox/config/__init__.py", line 1276, in __init__
    raise tox.exception.ConfigError(
tox.exception.ConfigError: ConfigError: pylint failed with ConfigError: skip_install: boolean value "True # doesn't need installing Pygments into the venv" needs to be 'True' or 'False' at Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/tox/config/__init__.py", line 1252, in run
    results[name] = cur_self.make_envconfig(name, section, subs, config)
  File "/usr/lib/python3/dist-packages/tox/config/__init__.py", line 1407, in make_envconfig
    res = meth(env_attr.name, env_attr.default, replace=replace)
  File "/usr/lib/python3/dist-packages/tox/config/__init__.py", line 1696, in getbool
    raise tox.exception.ConfigError(
tox.exception.ConfigError: ConfigError: skip_install: boolean value "True # doesn't need installing Pygments into the venv" needs to be 'True' or 'False'

[1]
jvoisin@home 20:49 (tyspt) ~/dev/pygments 

Also, while the lexer is working ~ok, there is still a lot of space for improvements.

[jeanas]

You have tox 3.21.4; you need a more recent version (latest is 4.11.3).

[jvoisin]

Getting there, but I can't manage to make pygments understand that inline code can become regular code via the open parenthesis :/

[jvoisin]

It's now looking ok-ish I think.

[jvoisin]

I've tried to add tests with a more recent version of tox, but:

$ tox -- --update-goldens ~/cv_jvoisin.typ 
.pkg: _optional_hooks> python /home/jvoisin/dev/pygments/.venv/lib/python3.9/site-packages/pyproject_api/_backend.py True hatchling.build
.pkg: get_requires_for_build_editable> python /home/jvoisin/dev/pygments/.venv/lib/python3.9/site-packages/pyproject_api/_backend.py True hatchling.build
.pkg: build_editable> python /home/jvoisin/dev/pygments/.venv/lib/python3.9/site-packages/pyproject_api/_backend.py True hatchling.build
py: install_package> python -I -m pip install --force-reinstall --no-deps /home/jvoisin/dev/pygments/.tox/.tmp/package/5/pygments-2.17.1-py3-none-any.whl
py: commands[0]> pytest --update-goldens /home/jvoisin/test.typ
ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --update-goldens
  inifile: None
  rootdir: /home/jvoisin

py: exit 4 (0.14 seconds) /home/jvoisin/dev/pygments> pytest --update-goldens /home/jvoisin/test.typ pid=27838
.pkg: _exit> python /home/jvoisin/dev/pygments/.venv/lib/python3.9/site-packages/pyproject_api/_backend.py True hatchling.build
  py: FAIL code 4 (0.59=setup[0.44]+cmd[0.14] seconds)
  evaluation failed :( (0.66 seconds)
[4]
$```

[birkenfeld]

You can see that pytest uses your homedir as the rootdir because the file you're pointing to is there. (Why?)

It needs to be in the pygments test hierarchy for pytest to pick up our customizations.

[jvoisin]

You can see that pytest uses your homedir as the rootdir because the file you're pointing to is there. (Why?)

The main reason is because I'm an idiot who didn't read the well written documentation :/

Things should be working now :)

[jeanas]

I've applied a few fixes that I missed on the initial review.

The rule for #include isn't working, though. Try

#include "foo.typ"

in the example. AFAICS, #include is eaten as Name.Variable by the rule in the 'markup' state, then "foo.typ" is outside code mode. Can it be fixed? What are the exact rules on code mode? (BTW, #(...) seems not to be working either.)

If you don't want to fix it now, we can always merge the PR without #include.

[jvoisin]

The rules are outlined here: https://typst.app/docs/reference/syntax/

Feel free to merge it as is, I'll try to improve it afterwards :)

[jeanas]

Thank you @jvoisin!

[miliog]

Thanks! The test file isn't valid Typst code though. You can check it on either https://typst.app (official web editor), https://pandoc.org/try/ or use the cli program.

This is what typst-lsp in vscode gives me:

Feel free to use the following code instead:

#show link: set text(navy)
#set text(font: "Comic Sans MS", hyphenate: true, lang: "en")

#set page(
  margin: (top: 16pt, bottom: 24pt),
  numbering: "1 / 1",
)

= Title

#table(
  columns: (auto, 50%, 1fr),
  [1], [2], [
    #let x = 123 + 123
    #{
      x *= 5
    }
    #x
  ],
)

This is an example link: https://example.org

#set text(fill: purple)

$ A = pi r^2 $