Appending an array item to another one doesn't work as expected

[simonschaufi]

I'm creating a rector rule and do the following:

1. create a native php array
2. parse it with the php parser
3. append the new array item to another php array which is there as parsed array

Expected:

<?php\n
\n
return [\n
    'tools_toolsmaintenance' => [\n
        'parent' => 'tools',\n
        'access' => 'systemMaintainer',\n
        'path' => '/module/tools/toolsmaintenance',\n
        'iconIdentifier' => 'module-install-maintenance',\n
        'labels' => 'LLL:EXT:install/Resources/Private/Language/ModuleInstallMaintenance.xlf',\n
        'routes' => [\n
            '_default' => [\n
                'target' => 'TYPO3\\CMS\\Install\\Controller\\BackendModuleController::maintenanceAction',\n
            ],\n
        ],\n
    ],\n
    'tools_toolssettings' => [\n
        'parent' => 'tools',\n
        'access' => 'systemMaintainer',\n
        'path' => '/module/tools/settings',\n
        'iconIdentifier' => 'module-install-settings',\n
        'labels' => 'LLL:EXT:install/Resources/Private/Language/ModuleInstallSettings.xlf',\n
        'routes' => [\n
            '_default' => [\n
                'target' => 'TYPO3\\CMS\\Install\\Controller\\BackendModuleController::settingsAction',\n
            ],\n
        ],\n
    ],\n
];\n

Actual:

<?php\n
\n
return [\n
    'tools_toolsmaintenance' => [\n
        'parent' => 'tools',\n
        'access' => 'systemMaintainer',\n
        'path' => '/module/tools/toolsmaintenance',\n
        'iconIdentifier' => 'module-install-maintenance',\n
        'labels' => 'LLL:EXT:install/Resources/Private/Language/ModuleInstallMaintenance.xlf',\n
        'routes' => [\n
            '_default' => [\n
                'target' => 'TYPO3\\CMS\\Install\\Controller\\BackendModuleController::maintenanceAction',\n
            ],\n
        ],\n
    ], 'tools_toolsmaintenance' => [\n
        'parent' => 'tools',\n
        'access' => 'systemMaintainer',\n
        'path' => '/module/tools/toolsmaintenance',\n
        'iconIdentifier' => 'module-install-maintenance',\n
        'labels' => 'LLL:EXT:install/Resources/Private/Language/ModuleInstallMaintenance.xlf',\n
        'routes' => [\n
            '_default' => [\n
                'target' => 'TYPO3\\CMS\\Install\\Controller\\BackendModuleController::maintenanceAction',\n
            ],\n
        ],\n
    ],\n
];\n

The strange thing is that the merged array looks good in xdebug:

but the output is a duplicate of the original array item and I don't understand why this is happening.

My PR of the rule is here sabbelasichon/typo3-rector#4243 and the important section is here: https://github.com/sabbelasichon/typo3-rector/pull/4243/files#diff-6659bbad2dc4a8d574769de61529b6e6d9387889d39813bbe2f246c873ba3bfeR231

and here: https://github.com/sabbelasichon/typo3-rector/pull/4243/files#diff-6659bbad2dc4a8d574769de61529b6e6d9387889d39813bbe2f246c873ba3bfeR425-R447

Can you please help me as I'm totally stuck? I don't understand what is happening in the PrettyPrinterAbstract class as it is quite complex

[nikic]

Sorry, I can't help with issues involving Rector infrastructure, I'd need a way to reproduce this issue independently. Rector devs will more likely be able to help here.

[simonschaufi]

This issue has no direct dependency on rector as it is basically merging 2 array items and then printing them with the PrettyPrinter. Rector just extends the PrettyPrinter but the main issue is within nikic/php-parser. It is probably related to the attributes that the copied array item has. I just presented you the PR where you can easily reproduce the issue without creating a standalone code (which is not so easy to create).

I've reached out to rector devs already but they were also not able to help me so far.

[nikic]

Do I understand correctly that you are parsing two different files and then copying over nodes from one file into the other?

If so, that's unfortunately not supported :( The nodes contain offsets into the file (or rather, its token array), but they don't know which file they belong to. The formatting-preserving pretty printer is provided the tokens for a single file only, and uses them to inspect the original formatting.

So if you mix AST nodes from two different files, what happens is basically that offsets into one file are instead interpreted as offsets into a different file.

A somewhat hacky workaround would be to concatenate both files before parsing...

[simonschaufi]

Do I understand correctly that you are parsing two different files and then copying over nodes from one file into the other?

Yes, exactly.

So if you mix AST nodes from two different files, what happens is basically that offsets into one file are instead interpreted as offsets into a different file.

That is what I thought.

A somewhat hacky workaround would be to concatenate both files before parsing...

ok, let me give it a try.

[simonschaufi]

@nikic the hack actually worked! I'm merging the 2 php files together and then remove the node again afterwards.

I hope this will be supported in the future officially or render a nice php array not as a single line out of the box.