prepend_prolog is broken

[jbms]

Describe the bug

In sphinx/util/rst.py, the prepend_prolog function is used to insert the rst_prolog at the "beginning" of each document:

sphinx/sphinx/util/rst.py

Line 76 in 93215f0

def prepend_prolog(content: StringList, prolog: str) -> None:

sphinx/sphinx/util/rst.py

Line 26 in 93215f0

docinfo_re = re.compile(':\\w+:.*?')

The intent is for it to be inserted just after the docinfo field list, if one is present, so that the document itself can still specify docinfo/metadata fields, even if rst_prolog produces docutils nodes itself (it is not clear to me what the use cases are for an rst_prolog that produces docutils nodes, but I suppose there may be some).

Unfortunately, there are numerous problems with this regular expression:

1. It does not handle hyphens in field names.
2. It does not handle blank lines in between or before fields.
3. It does not handle multi-line field bodies.
4. In principle a directive could be used to generate an initial field_list node. (This is surely rare, if ever done, and is probably not solvable.)

In fact, in most cases rst_prolog does not generate any docutils nodes, and therefore it makes no difference if it gets inserted before or after the docinfo fields. A problem arises, however, if it gets inserted in the middle of the docinfo fields, because the regular expression matches only some, but not all, of the docinfo fields.

In that case, the initial fields will be parsed as two separate field lists, and docutils will treat only the first one as the docinfo fields. Consequently, the second field list will remain as a normal field list in the document, and only the first field list will be used as the sphinx metadata.

How to Reproduce

conf.py:

rst_prolog = """
.. role:: python(code)
   :language: python
"""

index.rst:

:hero: xyz
:foo-bar: abc def

Environment Information

Platform:              linux; (Linux-5.18.16-1rodete1-amd64-x86_64-with-glibc2.33)
Python version:        3.10.7 (main, Sep  8 2022, 14:34:29) [GCC 12.2.0])
Python implementation: CPython
Sphinx version:        6.0.0b3+/93215f0a6
Docutils version:      0.19.1b.dev 
Jinja2 version:        3.1.2

Sphinx extensions

No response

Additional context

No response

[jbms]

It appears that the special handling of rst_prolog was added in response to: #1944