Special !! values are reset to null in mkdocs.yml

[pawamoy]

Details

As reported here (the issue is now stale, not sure if you'd prefer the discussion to continue over there), values like !!python/name:module.name in mkdocs.yml are dropped during the build process. They end up being null in the final mkdocs.yml, which in turn breaks build in various places (mainly plugins).

• Build URL (if applicable): https://readthedocs.org/projects/copier/builds/11817460/

Expected Result

These values should not be reset to null as it breaks the build. More details in the linked issue.

Actual Result

Special values starting with !!python/name: in mkdocs are reset to null, and it breaks the build.

[stsewd]

The solution in #6889 (comment) should do the trick. We parse those inside a docker environment, so it should be safe.

[stsewd]

So, we execute that in the builders, but the parsing code is still executed inside the application layer, we are safe to execute commands that we send to docker, not the ones we execute inside the application layer.

So, we can't use the normal parse function here. Some options:

• Make this a script that would be executed inside the docker container.
• Find another way to parse the yaml file without executing random code and without modifying the original code (maybe another library?).
• Plugins shouldn't rely on that and find another way for users to set such options.

[pawamoy]

Here's a way to "fake" load !!python/name: tags without losing their value when redumping them (second option above):

from pathlib import Path
import yaml


class ProxyPythonName(yaml.YAMLObject):
    def __init__(self, value):
        self.value = value


class CustomLoader(yaml.SafeLoader):
    def construct_python_name(self, suffix, node):
        return ProxyPythonName(suffix)


class CustomDumper(yaml.SafeDumper):
    def represent_name(self, data):
        return self.represent_scalar("tag:yaml.org,2002:python/name:" + data.value, "")


CustomLoader.add_multi_constructor("tag:yaml.org,2002:python/name:", CustomLoader.construct_python_name)
CustomDumper.add_representer(ProxyPythonName, CustomDumper.represent_name)

string = Path("mkdocs.yml").read_text()
loaded = yaml.load(string, Loader=CustomLoader)
dumped = yaml.dump(loaded, Dumper=CustomDumper)
print(dumped)

Input:

markdown_extensions:
  - pymdownx.superfences:
      custom_fences:
        - class: mermaid
          format: !!python/name:pymdownx.superfences.fence_div_format
          name: mermaid

Output:

markdown_extensions:
- pymdownx.superfences:
    custom_fences:
    - class: mermaid
      format: !!python/name:pymdownx.superfences.fence_div_format ''
      name: mermaid

No Python code was imported / executed 🙂

[FeeeeK]

Two years later, the problem reappeared again, but this time with !!python/object/apply, needed for pymdown-extensions:
https://facelessuser.github.io/pymdown-extensions/faq/#function-references-in-yaml

[pawamoy]

Shouldn't be too hard to open a new issue asking if maintainers would be OK to support it, and to copy-paste the code of the previous PR adding support for name and change it a bit to support apply.

[humitos]

@FeeeeK it seems we implemented this only for !!python/name, see #7507 (comment). I think we could expand that list now that there is a needing for this.

As a workaround for now, you could use the feature to Override the build process by using build.commands config key. This way, Read the Docs won't do anything with your mkdocs.yml file and you MkDocs plugin will work.

Here is an example of how to create the .readthedocs.yaml file using this approach:

version: 2

build:
  os: ubuntu-22.04
  tools:
    python: "3"
  commands:
    pip install -r requirements.txt
    mkdocs build --site-dir $READTHEDOCS_OUTPUT/html

[FeeeeK]

Thanks, @humitos, I already did it, but it would be nice if there were any warnings about this, now there is only traceback from toc without any additional info.