invalid characters break insert tag replacement

[fritzmg]

Affected version(s)
4.4.34, probably 4.7.0 too

Description
If the preg_replace for insert tag replacement

contao/core-bundle/src/Resources/contao/library/Contao/InsertTags.php

Line 78 in 51f3d79

$tags = preg_split('~{{([\pL\pN][^{}]*)}}~u', $strBuffer, -1, PREG_SPLIT_DELIM_CAPTURE);

gets fed with an invalid character somewhere in $strBuffer, no insert tag will be matched, and thus potentially no insert tags will be replaced on the page. Instead they will be shown as they are.

This is because of the u flag, that was introduced in 70a1403

@leofeyer is there a specific reason for adding this flag? Otherwise it can simply be removed to fix the issue.

How to reproduce

This is a little more difficult. The test case involved a site which contained characters in the database that were not encoded as UTF-8. This itself is not yet a problem. The issue manifests itself on the search result page of the search module. The search module was cutting of a word in the middle of a non-UTF-8 character, which produced the invalid output. So if the original search result context contained the word erschlieüen (erschließen with the ß encoded in a different character set), the search result module was clipping the content right in between the invalid character resulting in

<p class="context">... erschlieÃ</p>

/cc @ausi

[ausi]

If we want to support unicode characters at the beginning of insert tags (as we currently do) the regex should be /{{([a-zA-Z0-9\x80-\xFF][^{}]*)}}/ IMO. Otherwise we could use /{{([a-zA-Z0-9][^{}]*)}}/

[leofeyer]

Fixed in de82452. Thank you @ausi.