2.4.3-p2 breaks OAuth integrations, does not upgrade secrets and tries to decrypt them

[evolbug]

Preconditions (*)

1. Have OAuth integrations created before forced secret encryption
2. Upgrade to 2.4.3-p2

Steps to reproduce (*)

1. Create and connect an integration on a version before 2.4.3-p2
2. Upgrade to 2.4.3-p2

Expected result (*)

1. OAuth integration can be successfully reauthorized

Actual result (*)

1. Authorization fails with The attempt to post data to consumer failed due to an unexpected error. Please try again later.
2. The secrets are not upgraded along with the update, so it attempts to decrypt unencrypted values
   
3. Internal exception: [2022-04-14 15:01:08] main.CRITICAL: Notice: iconv_strlen(): Detected an illegal character in input string in /var/www//data/releases/20220414144351/vendor/magento/zendframework1/library/Zend/Validate/StringLength.php on line 250 {"exception":"[object] (Exception(code: 0): Notice: iconv_strlen(): Detected an illegal character in input string in /var/www//data/releases/20220414144351/vendor/magento/zendframework1/library/Zend/Validate/StringLength.php on line 250 at /var/www//data/releases/20220414144351/vendor/magento/framework/App/ErrorHandler.php:61)"} []
4. Patch list shows that OAuth upgrade patches were applied
   

---

Please provide Severity assessment for the Issue as Reporter. This information will help during Confirmation and Issue triage processes.

• Severity: S0 - Affects critical data or functionality and leaves users without workaround.
• Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
• Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
• Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
• Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

[m2-assistant]

Hi @evolbug. Thank you for your report.
To speed up processing of this issue, make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, Add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, review the Magento Contributor Assistant documentation.

Add a comment to assign the issue: @magento I am working on this

To learn more about issue processing workflow, refer to the Code Contributions.

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

⚠️ According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, join the Community Contributions Triage session to discuss the appropriate ticket.

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

[m2-assistant]

Hi @engcom-November. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• 4. Verify that the issue is reproducible on 2.4-develop branch

  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• 5. Add label Issue: Confirmed once verification is complete.

• 6. Make sure that automatic system confirms that report has been added to the backlog.

[engcom-November]

Verified the issue by upgrading Magento version from 2.3.7 to 2.4.3-p2 but could not able to reproduce the issue.
Admin - System - Integrations - Created new integration before upgrade
Upgraded to 2.4.3-p2
Reauthorize integration - No issue/ Exception

@evolbug kindly recheck the issue again and elaborate steps to reproduce if you are still facing issues.
Thank you.

[kenseiatwork]

I encountered this problem as well. It looks to be an issue for older magento sites when the module.xml's setup_version was still in use.

The last setup_version for the integration module looks to be 2.2.0, you can check the row for Magento_Integration in the setup_module table.

In the patch files for UpgradeConsumerSecret.php and UpgradeOauthToken.php, the method for getVersion() is set to 2.0.0. I believe this should be 2.2.1 or greater -- or removed altogether.

public static function getVersion()
{
    return '2.0.0';
}

Per devdocs
https://devdocs.magento.com/guides/v2.4/extension-dev-guide/declarative-schema/data-patches.html#will-old-scripts-work-in-newer-versions

You would also need to remove the rows in the table patch_list:
Magento\Integration\Setup\Patch\Data\UpgradeOauthToken
Magento\Integration\Setup\Patch\Data\UpgradeConsumerSecret

Then rerun setup:upgrade

[evolbug]

@engcom-November You need to reauthorize the integration, also check the integration details for incorrectly displaying secrets. The migration we did was from 2.4.3 to 2.4.3-p2, perhaps check that

[YiffyToys]

We can confirm this updating 2.4.3-p1 to 2.4.3-p2
@evolbug reauthorize does NOT help. The consume secret stays mangled and unusable.
You have to recreate the integration with every single access permission the old one had.
A very time consuming process exagerated by the fact that you can't right clitk "open in new tab" most links in the admin area.

[engcom-November]

Verified the issue again by upgrading from 2.4.3-p1 to 2.4.3-p2
Recreated the Integration with same Resources access as the old one is also working fine
Reauthorize Integration also worked fine in 2.4.3-p2

@evolbug We are not able to reproduce the issue by upgrading from 2.4.3 to 2.4.3-p2 as well. Reauthorize the integration working fine in local system.
Kindly elaborate steps to reproduce/configuration details or if any other extensions are enabled.
Thank you.

[YiffyToys]

Reauthorize can be performed here too but entering the integration damaged in the upgrade
still shows a mangled and unusable "Consumer Secret" and using the new token yields 401 unauthorized.
Completely new integrations created after the update to 2.4.2-p2 work just fine.

[m2-assistant]

Hi @engcom-Hotel. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• 4. Verify that the issue is reproducible on 2.4-develop branch

  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• 5. Add label Issue: Confirmed once verification is complete.

• 6. Make sure that automatic system confirms that report has been added to the backlog.

[m2-assistant]

Hi @engcom-Dash. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
     Details

     If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.
• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.
• 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.
• 4. Verify that the issue is reproducible on 2.4-develop branch
     Details

     - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
     - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
     - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

[engcom-Dash]

Hi @evolbug
We have tried to reproduce the issue in Magento 2.4.3-p2. the issue is not reproducible.

1.Verified the issue again by upgrading from 2.4.3-p1 to 2.4.3-p2
2.Recreated the Integration with same Resources access as the old one is also working fine
3.Reauthorize Integration also worked fine in 2.4.3-p2

Let us know in case we have missed anything. Thanks

[hostep]

Can reproduce due to the hints of @kenseiatwork, thanks for that!

@engcom-Dash: please try the following:

1. Using composer v1 and PHP 7.2, setup a clean Magento 2.2.11 installation
2. In the backoffice, configure at least one integration and activate it
3. Now, switch to composer v2, PHP 8.1, remove your entire vendor directory and use composer to upgrade to Magento 2.4.4
4. Execute bin/magento setup:upgrade
5. In the backoffice, click on Edit in your integration(s) and notice that the consumer secret now contains a bunch of weird symbols

[engcom-Hotel]

Hello @hostep,

Thanks for the support!

Hello @evolbug,

We have tried to reproduce the issue by following this #35313 (comment). But for us, the issue is still not reproducible.

We are able to reauthorize the integrations but we have observed that after the upgrade Consumer Secret is looks like below:

Also please refer to the screencast of reauthorizing after the upgrade:

Integrations-Extensions-System-Magento-Admin.mov

Thanks

[evolbug]

@engcom-Hotel the garbled consumer secret is a reproduction of the issue, this is due to the secret not being encrypted in the migration, so after it, it ends up trying to decrypt the plaintext value from the db

[zepgram]

I'm not sure that you realise it can break whole service using API integration: this should be a P1 ticket

By launching a 2.4.3-p2 or 2.4.4 Magento on production you could break every API service.

Guys you should at least create a quality patch for this issue.

[nimbus2300]

Upgraded to 2.4.4 over the weekend. This bug has broken all my Royal Mail Click and Connect integrations.

I'm not sure that you realise it can break whole service using API integration: this should be a P1 ticket

By launching a 2.4.3-p2 or 2.4.4 Magento on production you could break every API service.

Guys you should at least create a quality patch for this issue.

[nimbus2300]

Upgraded to 2.4.4 over the weekend. This bug has broken all my Royal Mail Click and Connect integrations.

I'm not sure that you realise it can break whole service using API integration: this should be a P1 ticket
By launching a 2.4.3-p2 or 2.4.4 Magento on production you could break every API service.
Guys you should at least create a quality patch for this issue.

I think the fix /might/ be to ensure that:

When the magento version is upgraded to the new secure version (>=2.4.3-p2):

• Ensure that a one-time upgrade script/activity takes place, that:
  • Crypts all existing stored integration keys, and rewrites them, now encrypted, to the same row in the database.

The new version of Magento would then be able decrypt the keys successfully (rather than appearing garbled as in the screenshots in the above thread).

Needs verification from someone else though to verify this thinking.

Does Magento has a mechanism for one-time data manipulation scripts to be run on minor version upgrades?

---

Here is a screenshot showing the garbled secrets:

[onepack]

Same issue here with two clients after upgrading from 2.3 to 2.4.4-p2.
All integrations are broken.

[willbrammer]

Hi

I just want to add this issue is happening on Adobe Commerce 2.4.5-p1 too.

I have also raised a ticket with Magento Support to see if they have anything for this.

Bit surprised how long this has been open. Deleting and re-creating the integrations is not ideal for some systems

Thanks
Will

[zepgram]

Still waiting for a quality patch since July @magento.

It's just my opinion but security patch SHOULD never include a data/schema patch or any database changes.
You should talk about this with your architecture team.

The reason is simple: you may break Magento on a security patch, which is a total non-sense by doing this you are also breaking customer's confidence on security release.

At least, you may warn everyone in your release patch notes when a database patch is included in your security release.

[engcom-Hotel]

Hello @zepgram @willbrammer,

The related JIRA status is Deployment Queue, it will be merged soon with the development branch. I will keep you posted.

Thanks

[ninjacoder30]

Hello @engcom-Hotel

I am facing same issue on 2.4.4-p1 , i deleted the patch entries and still not able to fix the issue.

Also when i try to active new key its giving me below error
The attempt to post data to consumer failed due to an unexpected error. Please try again later.

Below is the trace

2022-11-04T18:11:58+00:00 INFO (6): Unable to read response, or response is empty 2022-11-04T18:11:58+00:00 INFO (6): #0 vendor/magento/framework/HTTP/ZendClient.php(52): Zend_Http_Client->request() #1 vendor/magento/module-integration/Model/OauthService.php(242): Magento\Framework\HTTP\ZendClient->request() #2 vendor/magento/module-integration/Controller/Adminhtml/Integration/TokensExchange.php(59): Magento\Integration\Model\OauthService->postToConsumer() #3 vendor/magento/framework/Interception/Interceptor.php(58): Magento\Integration\Controller\Adminhtml\Integration\TokensExchange->execute() #4 vendor/magento/framework/Interception/Interceptor.php(138): Magento\Integration\Controller\Adminhtml\Integration\TokensExchange\Interceptor->___callParent() #5 vendor/magento/framework/Interception/Interceptor.php(153): Magento\Integration\Controller\Adminhtml\Integration\TokensExchange\Interceptor->Magento\Framework\Interception\{closure}() #6 generated/code/Magento/Integration/Controller/Adminhtml/Integration/TokensExchange/Interceptor.php(23): Magento\Integration\Controller\Adminhtml\Integration\TokensExchange\Interceptor->___callPlugins() #7 vendor/magento/framework/App/Action/Action.php(111): Magento\Integration\Controller\Adminhtml\Integration\TokensExchange\Interceptor->execute() #8 vendor/magento/module-backend/App/AbstractAction.php(151): Magento\Framework\App\Action\Action->dispatch() #9 vendor/magento/framework/Interception/Interceptor.php(58): Magento\Backend\App\AbstractAction->dispatch() #10 vendor/magento/framework/Interception/Interceptor.php(138): Magento\Integration\Controller\Adminhtml\Integration\TokensExchange\Interceptor->___callParent() #11 app/code/WeltPixel/Backend/Plugin/Utility.php(76): Magento\Integration\Controller\Adminhtml\Integration\TokensExchange\Interceptor->Magento\Framework\Interception\{closure}() #12 vendor/magento/framework/Interception/Interceptor.php(135): WeltPixel\Backend\Plugin\Utility->aroundDispatch() #13 vendor/magento/module-backend/App/Action/Plugin/Authentication.php(145): Magento\Integration\Controller\Adminhtml\Integration\TokensExchange\Interceptor->Magento\Framework\Interception\{closure}() #14 vendor/magento/framework/Interception/Interceptor.php(135): Magento\Backend\App\Action\Plugin\Authentication->aroundDispatch() #15 vendor/magento/framework/Interception/Interceptor.php(153): Magento\Integration\Controller\Adminhtml\Integration\TokensExchange\Interceptor->Magento\Framework\Interception\{closure}() #16 generated/code/Magento/Integration/Controller/Adminhtml/Integration/TokensExchange/Interceptor.php(32): Magento\Integration\Controller\Adminhtml\Integration\TokensExchange\Interceptor->___callPlugins() #17 vendor/magento/framework/App/FrontController.php(245): Magento\Integration\Controller\Adminhtml\Integration\TokensExchange\Interceptor->dispatch() #18 vendor/magento/framework/App/FrontController.php(212): Magento\Framework\App\FrontController->getActionResponse() #19 vendor/magento/framework/App/FrontController.php(147): Magento\Framework\App\FrontController->processRequest() #20 vendor/magento/framework/Interception/Interceptor.php(58): Magento\Framework\App\FrontController->dispatch() #21 vendor/magento/framework/Interception/Interceptor.php(138): Magento\Framework\App\FrontController\Interceptor->___callParent() #22 vendor/magento/framework/Interception/Interceptor.php(153): Magento\Framework\App\FrontController\Interceptor->Magento\Framework\Interception\{closure}() #23 generated/code/Magento/Framework/App/FrontController/Interceptor.php(23): Magento\Framework\App\FrontController\Interceptor->___callPlugins() #24 vendor/magento/framework/App/Http.php(116): Magento\Framework\App\FrontController\Interceptor->dispatch() #25 vendor/magento/framework/App/Bootstrap.php(264): Magento\Framework\App\Http->launch() #26 pub/index.php(30): Magento\Framework\App\Bootstrap->run() #27 {main}

[engcom-Hotel]

Hello,

As I can see this issue got fixed in the scope of the internal Jira ticket AC-3752 by the internal team
Related commits: https://github.com/magento/magento2/search?q=AC-3752&type=commits

Based on the Jira ticket, the target version is 2.4.6.

Thanks

[goivvy]

Create two scripts in /path/to/magento/root

runOauth.php:

ini_set('memory_limit','12000M');
use Magento\Framework\App\Bootstrap;
require DIR.'/app/bootstrap.php';
$params = $_SERVER;
$bootstrap = Bootstrap::create(BP,$params);
$obj = $bootstrap->getObjectManager();
$state = $obj->get('Magento\Framework\App\State');
$state->setAreaCode('frontend');

try{
$encryptor = $obj->get('Magento\Framework\Encryption\Encryptor');
$tokenCollection = $obj->get('Magento\Integration\Model\ResourceModel\Oauth\Token\CollectionFactory')->create();
$tokenResourceModel = $obj->get('Magento\Integration\Model\ResourceModel\Oauth\Token');
$tokenCollection->addFieldToSelect('entity_id');
$tokenCollection->addFieldToSelect('secret');
$tokenCollection->addFieldToSelect('type');
$tokenCollection->setPageSize(100);
$connection = $tokenResourceModel->getConnection();
$pages = $tokenCollection->getLastPageNumber();
$tableName = $tokenResourceModel->getMainTable();

    for ($currentPage = 1; $currentPage <= $pages; $currentPage++) {
        $tokenCollection->setCurPage($currentPage);

        foreach ($tokenCollection as $token) {
            $existingSecret = $token->getSecret();
            $entityId = $token->getEntityId();
            $type = $token->getType() !== null ? strtolower($token->getType()) : '';

            if ($entityId && $existingSecret && $type === 'access') {
                if (strlen($existingSecret) <= 32) {
                    echo $existingSecret."\n";
                    $data = ['secret' => $encryptor->encrypt($existingSecret)];
                    $where = ['entity_id = ?' => $entityId, 'type = ?' => 'access'];
                    try {
                        $connection->update($tableName, $data, $where);
                    } catch (\Exception $exception) {
                        die($e->getMessage());
                    }
                }
            }
        }
        $tokenCollection->clear();
    }

}catch(\Exception $e){die('||'.$e->getMessage());}
`

and

runConsumer.php:

`
ini_set('memory_limit','12000M');
use Magento\Framework\App\Bootstrap;
require DIR.'/app/bootstrap.php';
$params = $_SERVER;
$bootstrap = Bootstrap::create(BP,$params);
$obj = $bootstrap->getObjectManager();
$state = $obj->get('Magento\Framework\App\State');
$state->setAreaCode('frontend');

try{
$encryptor = $obj->get('Magento\Framework\Encryption\Encryptor');
$consumerCollection = $obj->get('Magento\Integration\Model\ResourceModel\Oauth\Consumer\CollectionFactory')->create();
$consumerResourceModel = $obj->get('Magento\Integration\Model\ResourceModel\Oauth\Consumer');

$consumerCollection->addFieldToSelect('entity_id');
$consumerCollection->addFieldToSelect('secret');
$connection = $consumerResourceModel->getConnection();
$consumerCollection->setPageSize(100);
$pages = $consumerCollection->getLastPageNumber();
$tableName = $consumerResourceModel->getMainTable();

for ($currentPage = 1; $currentPage <= $pages; $currentPage++) {
$consumerCollection->setCurPage($currentPage);

   foreach ($consumerCollection as $consumer) {
   $existingSecret = $consumer->getSecret();
    $entityId = $consumer->getEntityId();

      if ($entityId && $existingSecret) {
                if (strlen($existingSecret) <= 32) {
                    $data = ['secret' => $encryptor->encrypt($existingSecret)];
                    $where = ['entity_id = ?' => $entityId];
                    try {
                        $connection->update($tableName, $data, $where);
                    } catch (\Exception $exception) {
                        die($e->getMessage());
                    }
                }
            }
        }
        $consumerCollection->clear();
  }

}catch(\Exception $e){die('||'.$e->getMessage());}
`

Then run:

php runOauth.php
php runConsumber.php

Your existing secrets should be fixed afterwards.

[cfortin]

How is this issue closed? I just upgraded to 2.4.5-p3 and am still seeing this issue.

[hostep]

@cfortin: according to a few comments earlier, you'll need to upgrade to at least Magento 2.4.6 to get it fixed. And based on the commit mentioned, it could be that the upgrade is not enough to fix it and you might need to make sure the UpgradeConsumerSecret patch gets executed again, even if it was already executed before, like mentioned in this comment

[cfortin]

Thank you very much @hostep.