Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Different results of template parser for arrays on different operating systems #449

Open
dlubitz opened this issue May 2, 2019 · 11 comments

Comments

@dlubitz
Copy link

commented May 2, 2019

I had issues with shorthand viewhelpers. The arguments I passed as an array where handled as a string at the end. I dug deeper into the issue and found the root cause in the regex \TYPO3Fluid\Fluid\Core\Parser\Patterns::$SPLIT_PATTERN_SHORTHANDSYNTAX_ARRAY_PARTS for splitting the arrays recusive.

My viewhelper code:

{f:translate(id: 'some.id.to.translate', package: 'Vendor.Package', arguments: {0: some.value})}

which leads to the exeption:

The argument "arguments" was registered with type "array", but is of type "string" in view helper

I could reproduce this error just on a FreeBSD server and could find related bug reports.

Related issues:


The regex pattern fetches there for any reason VariableIdentifier and Subarray. Which leads to the issue described above, because the Subarray gets handled as ObjectAccessorNode.

if (!empty($singleMatch['VariableIdentifier'])) {
$arrayToBuild[$arrayKey] = new ObjectAccessorNode($singleMatch['VariableIdentifier']);
} elseif (array_key_exists('Number', $singleMatch) && (!empty($singleMatch['Number']) || $singleMatch['Number'] === '0')) {
// Note: this method of casting picks "int" when value is a natural number and "float" if any decimals are found. See also NumericNode.
$arrayToBuild[$arrayKey] = $singleMatch['Number'] + 0;
} elseif ((array_key_exists('QuotedString', $singleMatch) && !empty($singleMatch['QuotedString']))) {
$argumentString = $this->unquoteString($singleMatch['QuotedString']);
$arrayToBuild[$arrayKey] = $this->buildArgumentObjectTree($argumentString);
} elseif (array_key_exists('Subarray', $singleMatch) && !empty($singleMatch['Subarray'])) {
$arrayToBuild[$arrayKey] = new ArrayNode($this->recursiveArrayHandler($singleMatch['Subarray']));
}

Test code

$string = 'id: \'some.id.to.translate\', package: \'Vendor.Package\', arguments: {0: some.value}';
preg_match_all(\TYPO3Fluid\Fluid\Core\Parser\Patterns::$SPLIT_PATTERN_SHORTHANDSYNTAX_ARRAY_PARTS, $string, $matches, PREG_SET_ORDER);
print_r($matches);

Expected result:

Array
(
    [0] => Array
        (
            [0] => id: 'some.id.to.translate'
            [ArrayPart] => id: 'some.id.to.translate'
            [1] => id: 'some.id.to.translate'
            [Key] => id
            [2] => id
            [QuotedString] => 'some.id.to.translate'
            [3] => 'some.id.to.translate'
        )

    [1] => Array
        (
            [0] => package: 'Vendor.Package'
            [ArrayPart] => package: 'Vendor.Package'
            [1] => package: 'Vendor.Package'
            [Key] => package
            [2] => package
            [QuotedString] => 'Vendor.Package'
            [3] => 'Vendor.Package'
        )

    [2] => Array
        (
            [0] => arguments: {0: some.value}
            [ArrayPart] => arguments: {0: some.value}
            [1] => arguments: {0: some.value}
            [Key] => arguments
            [2] => arguments
            [QuotedString] =>
            [3] =>
            [VariableIdentifier] =>
            [4] =>
            [Number] =>
            [5] =>
            [Subarray] => 0: some.value
            [6] => 0: some.value
        )

)

Wrong result:

Array
(
    [0] => Array
        (
            [0] => id: 'some.id.to.translate'
            [ArrayPart] => id: 'some.id.to.translate'
            [1] => id: 'some.id.to.translate'
            [Key] => id
            [2] => id
            [QuotedString] => 'some.id.to.translate'
            [3] => 'some.id.to.translate'
        )

    [1] => Array
        (
            [0] => package: 'Vendor.Package'
            [ArrayPart] => package: 'Vendor.Package'
            [1] => package: 'Vendor.Package'
            [Key] => package
            [2] => package
            [QuotedString] => 'Vendor.Package'
            [3] => 'Vendor.Package'
        )

    [2] => Array
        (
            [0] => arguments: {0: some.value}
            [ArrayPart] => arguments: {0: some.value}
            [1] => arguments: {0: some.value}
            [Key] => arguments
            [2] => arguments
            [QuotedString] =>
            [3] =>
            [VariableIdentifier] => some.value
            [4] => some.value
            [Number] =>
            [5] =>
            [Subarray] => 0: some.value
            [6] => 0: some.value
        )

)

These values should be empty:

    [2] => Array
        (
            ...
            [VariableIdentifier] => some.value
            [4] => some.value
            ...

Tested environments:

  • Ubuntu 16.04.6 LTS (Xenial Xerus), PHP7.3: expected result
  • CentOS release 6.10 (Final): expected result
  • Windows 10 Subsystem Linux / Ubuntu, PHP 7.3: expected result
  • FreeBSD 11.2-RELEASE-p9 #7 r344427, PHP7.3: wrong result

I'm not sure how to solve this issue in Fluid, but maybe the regex could become more strict or optimized to behave equal on all systems.

@NamelessCoder

This comment has been minimized.

Copy link
Member

commented May 2, 2019

Oh boy, regular expressions come back to bite us again. Thanks for the very detailed reports and testing, @dlubitz - that's extremely appreciated!

We had a similar issue a while back with older PHP versions and now-outdated versions of linked preg libraries. If I recall (it's been many years) it was also a problem with the named patterns and recursion.

And more recently we discovered that preg has an internal limitation on the size of matches when doing recursive matches - which we cannot change. This one prevents you from declaring large arrays - at some point, Fluid starts seeing the array as a string due to the limit.

I believe the only good way to move forward is to abandon the use of regular expressions, at least for this complex recursive matching, in favor of some more naive (tokenisation-like) procedures that recursively split and tokenise Fluid templates then creates the syntax tree from those.

There is progress being made in this department but unfortunately it's very far from easy to do this right, without causing significant performance impacts. I'm at my 4th attempt using a mix of naive string searches and splits without requiring multiple stacks during parsing, plus making nodes in the syntax tree more interchangeable so they don't necessarily need to "know" what type of node they are until all tokens have been collected.

If you have some experience with similar things and would like to involve yourself you would be more than welcome!

@dlubitz

This comment has been minimized.

Copy link
Author

commented May 2, 2019

If you have some experience with similar things and would like to involve yourself you would be more than welcome!

Sorry, I I'm not that experienced with parsing/tokenizing. :/

@jonnitto

This comment has been minimized.

Copy link

commented Aug 19, 2019

Does the translate view helper work with the tag annotation <f:translate /> on BSD?

@dlubitz

This comment has been minimized.

Copy link
Author

commented Aug 19, 2019

Didn't test it. But I guess so. Was just effecting the inline / shorthand Regex.

@janhelke

This comment has been minimized.

Copy link
Contributor

commented Aug 20, 2019

I have two more failing environments.

The current ddev environment (that might make it easy to reproduce and work on this issue)

Debian GNU/Linux 9 (stretch)
Linux version 4.9.184-linuxkit (root@a8c33e955a82) (gcc version 8.3.0 (Alpine 8.3.0) )
PHP 7.3.8-1+020190807.43+debian91.gbp7731bf (cli) (built: Aug 7 2019 19:46:25) ( NTS )

  • Failing Site Configuration Edit and New
  • Failing Reports module

One of our production environments

Gento
Linux version 4.14.127-gentoo (gcc version 7.3.0 (Gentoo 7.3.0-r3 p1.4))
PHP 7.3.6 (cli) (built: Jul 15 2019 16:26:07) ( ZTS )

  • Failing TranslateViewHelper upon BE login
@lyxys

This comment has been minimized.

Copy link

commented Aug 22, 2019

I could not reproduce the issue, but maybe my test setup was to simple. I used composer to get a TYPO3 installation and used the testcode posted by dlubitz above in the root of the webpage (after inserting

$classLoader = require DIR . '/../vendor/autoload.php';
in front of it) to run the test from the command line on FreeBSD 11.2, php 7.3.8, pcre2 10.32_1. As output I get the "expected result". Could someone who has the problem try to reproduce it with this setup?

@lyxys

This comment has been minimized.

Copy link

commented Aug 22, 2019

Well, after switching back to a slightly older environment, I CAN reproduce the problem with this small setup. Both environments have Freebsd 11.2-RELEASE-p9, but the failing environment has php 7.3.6 instead of 7.3.8 and pcre2 10.32 instead of 10.32_1. The difference between pcre2 10.32 and 10.32_1 appears to be that the former is linked against readline 7.0.5 while the latter is linked against readline 8.0.0. Maybe we can get the info about the used readline version for the other working and failing environments?

@alexanderschnitzler

This comment has been minimized.

Copy link

commented Aug 25, 2019

Just adding a comment that I am struck by that issue as well on Ubuntu 19.04 with PHP 7.3.7 and PCRE 10.32, installed via brew.

@lyxys

This comment has been minimized.

Copy link

commented Aug 26, 2019

Just adding a comment that I am struck by that issue as well on Ubuntu 19.04 with PHP 7.3.7 and PCRE 10.32, installed via brew.

Can you check which version of readline your installation is using? Currently I suspect readline 7 to be the culprit, as the problem disappeared on FreeBSD after the pcre2 package switched to using readline 8.

@alexanderschnitzler

This comment has been minimized.

Copy link

commented Aug 27, 2019

Well, I have different versions installed but I guess that while I installed PHP via brew, it uses the version of readline that is installed via brew as well:

$ brew info readline
readline: stable 8.0.0 (bottled)
Library for command-line editing
https://tiswww.case.edu/php/chet/readline/rltop.html
/home/linuxbrew/.linuxbrew/Cellar/readline/8.0.0_1 (48 files, 1.9MB) *
  Poured from bottle on 2019-08-09 at 17:29:54
From: https://github.com/Homebrew/linuxbrew-core/blob/master/Formula/readline.rb
==> Dependencies
Required: ncurses ✔
==> Analytics
install: 334,096 (30 days), 1,142,350 (90 days), 4,655,012 (365 days)
install_on_request: 27,414 (30 days), 104,308 (90 days), 459,949 (365 days)
build_error: 0 (30 days)
@robertvonhackwitz

This comment has been minimized.

Copy link

commented Aug 27, 2019

Just FYI: same problem on this environment:
OS: CentOs 7.6
Http Server: httpd-2.4.6-89
Mysql: mysql-community-server-5.7.27-1
Php: php-7.3.8-1 (mod_php NOT php-fpm)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.