Skip to content
This repository

Fix of wrong ordering logic for totals #49

Closed
wants to merge 2 commits into from

3 participants

Ivan Chepurnyi Alexander Menk
Ivan Chepurnyi

During custom development, there were discovered a bug with ordering process.

If you create custom total like this one:

<handling>
      <class>Full_Total_Class_Name</class>
      <after>shipping</after>
      <before>tax</before>
</handling>
<handling_tax>
     <class>Full_Total_Class_Name</class>
     <after>tax_shipping</after>
     <before>tax</before>
</handling_tax>

It will not be placed between tax and tax_shipping totals, the actual result was like the following:

Array
(
    [0] => nominal
    [1] => subtotal
    [2] => tax_subtotal
    [6] => freeshipping
    [7] => handling_tax
    [8] => handling
    [9] => weee
    [10] => shipping
    [11] => tax_shipping
    [12] => discount
    [14] => tax
    [17] => grand_total
    [19] => msrp
)

The problem itself in sorting method that were used. Since only close items were compared to each other and if you have total that is not next to desired one, its before and after attributes get ignored.

Apparently this commit not only fixes an issue with sorting, it also works 3 times faster then current method in core. The test was done on 1000 iteration five times and average results are the following:

  • Current core functionality 0.83539700508118 seconds
  • Patched functionality 0.26063299179077 seconds
Ivan Chepurnyi Fixed core issue related to invalid totals sorting in the quote. The …
…problem is visible if you have additional custom totals that need to be placed in between core ones.
9a3631e
Owner

@IvanChepurnyi
Thank you for yet another valuable contribution.
In order to accept it, all code changes should be supplied with the corresponding unit/integration tests.
Also, to simplify further code review, the conventional bug report would be greatly appreciated:
1. "Steps to reproduce"
2. "Expected result"
3. "Actual result".

Ivan Chepurnyi

Added integration test. It fails on 3rd data-set with non-fixed core code and on passes if my fix is applied.

The expected result is correctly sorted totals by before and after properties. The actual results and steps to reproduce already specified in original bug report.

mage2-team closed this
Owner

@IvanChepurnyi
Thank you for adding lacking tests.

The contribution has been accepted with the following deviations:

  • minimized usage of Reflections for testing
  • implemented additional tests for ambiguous situations during sorting
    • replaced testing logic to compare against expected results in order to be able to validate ambiguous cases

Again, thank you for all the effort and valuable contribution.

Changes will become available with one of the nearest merges from the internal repository. Closing the ticket.

Alexander Menk

@IvanChepurnyi Is this a toplogical sort algorithm? I am wondering what happens if the specified orders are contradictory (total foo after bar, total bar after foo). Shouldn't that be detected and an exception thrown?

Ivan Chepurnyi

It is not a good idea to throw an exception, where it was not thrown before, as far as I know the fixes from Magento 2.0 go to current version as well, and if at some Magento version conflict were resolved by some of the properties but later on an exception were thrown, then existent shops who updates via Magento connect are in danger of crashing down...

Alexander Menk

But if there a contradictions the sort order might be completely undefined ... so I think an exception is much better than an undefinied state that could cause to further problems such as wrongly calculated totals (you might sell your stuff without tax then an so on)

Ivan Chepurnyi

For this case after has always higher priority. Since grand_total gets after tax, tax goes after shipping, discount and so on, you will never get a situation of wrongly calculated taxes. It only affects custom totals that manually setting up positions for themselves.

Alexander Menk

How is the result defined if A after B , B after A was defined?

Ivan Chepurnyi

If A is processed after B then A will be placed after B, if B is processed after A, then B will be after A.

Alexander Menk

So one of the conditions is not met. For those cases also the testcase would fail (if you provide it with such "invalid input").

I still have the opinion that this can cause business critical problems that are very hard to track down and should be detected well. Everything else makes the system unreliable.

If someone updates Magento and has incompatible modules that cause such a problem, an exception would be noticed very soon (I hope nobody does an Magento update on a live system without testing).
But if the error is such a hidden one, it might go unnoticed and cause several business damage.

Ivan Chepurnyi

Sorry, Alexander, but I don't see any reason to argue with you about the exception, the patch was already submitted to core team, covered with integration tests and accepted by them into the next release. If you do not agree with my contribution, you are always free to create another pull request with your exception throw statement.

For my opinion it should not be thrown, to not create fatal error on the frontend, maybe it better just to log debug message somewhere. At least, in another core functionality with before/after sorting, there is no exception thrown if two blocks have cycling reference in before and after attributes (layouts).

As for store owners who update live version via Magento connect, it is true, and I've helped such people enormous amount of time during all the 5 years of my Magento development experience. Of course it is not recommended to update Magento version without testing on staging environment, but a lot of people just do so.

Alexander Menk

No offense. My intention is also not to argue. Its just about constructively improving your pull request.

This total sorting problem caused a lot of people lots of headaches and I am thankful that you fixed it.

But if it does not reliably fulfil the conditions giving in the XML configuration and silently ignores some of them I am just afraid that a similar issue in Magento2 will waste a lot of peoples time who have to debug this.

I also understand your point, that fatal errors for endusers are not nice. I think there is room for discussion what is better: A order that does not work, or an order with possibly wronlgy caluclates totals (if a total B that was specified to run after A is not run after A, the required total fields in the quote might not be available and the total might return 0).
Both is not good - so I think just logging an exception to exception.log might be a good compromise.

Actually I did not yet have the time to reconstruct the algorithm used in your pull request to identify the place where we could log an exception. That is one reason I did not enhance your pull request myself.

(My pull request #65 does not provide the feature to generate a "nearly correct" order - it just fails completly so I could only throw an Exception with a fatal error to the end user).

Alexander

mage2-team reopened this
Owner

We discussed the question of "to throw or not to throw" with product management and agreed on the following:

  • During checkout or anywhere on the frontend, the contradictory or ambiguous totals must not trigger errors and must not prevent customer from buying. The justification for this argument is that it is better to sell with slightly wrong price, rather than not to sell at all.
  • When totals are calculated, add validation of their declaration. The validation errors must be logged to system/error log
  • Implement an integrity test that would invoke validation of declared totals by the currently deployed Magento instance (it might include 3rd-party extensions with own totals). If ambiguous or contradictory declarations are declared -- fail the test. In such a way the developer or system administrator, when introducing an extension with new totals, will know if they break the system or not.

All the mentioned above has been implemented. Representation of totals is implemented as a graph (new library Magento_Data_Graph, but used only in validation for now). In this graph, we search for loops in various ways. If a loop is found, it means that it is logically impossible to transform it to a simple graph (or sort graph nodes).

The topological sorting algorithm should perfectly fit into the new model Magento_Data_Graph. We'll recommend it to author in the pull request #65. Changes will be rolled out later with next code updates.

mage2-team closed this
Alexander Menk

Alright - thanks for elaborating this.

"The justification for this argument is that it is better to sell with slightly wrong price, rather than not to sell at all." - makes sense to me. But I think we can not be sure, that such problems cause only slightly wrong prices... - depending on the totals to be affected.

Integrity Testing and logging to system/error.log sounds good to me and might be a good trade off. So we could make it up to the users if they conduct such tests and check the error log.

mage2-team referenced this pull request from a commit
Update as of 9/05/2012
* Implemented encryption of the credit card name and expiration date for the payment method "Credit Card (saved)"
* Implemented console utility `dev/tools/migration/get_aliases_map.php`, which generates map file "M1 class alias" to "M2 class name"
* Implemented automatic data upgrades for replacing "M1 class aliases" to "M2 class names" in a database
* Implemented recursive `chmod` in the library class `Varien_Io_File`
* Improved verbosity of the library class `Magento_Shell`
* Migrated client-side translation mechanism to jQuery
* Performance tests:
  * Improved assertion for number of created orders for the checkout performance testing scenario
    * Reverted the feature of specifying PHP scenarios to be executed before and after a JMeter scenario
    * Implemented validation for the number of created orders as a part of the JMeter scenario
    * Implemented the "Admin Login" user activity as a separate file to be reused in the performance testing scenarios
  * Implemented fixture of 100k customers for the performance tests
  * Implemented fixture of 100k products for the performance tests
    * Enhanced module `Mage_ImportExport` in order to utilize it for the fixture implementation
  * Implemented back-end performance testing scenario, which covers Dashboard, Manage Products, Manage Customers pages
* Fixes:
  * Fixed Magento console installer to enable write permission recursively to the `var` directory
  * Fixed performance tests to enable write permission recursively to the `var` directory
  * Fixed integration test `Mage_Adminhtml_Model_System_Config_Source_Admin_PageTest::testToOptionArray` to not produce "Warning: DOMDocument::loadHTML(): htmlParseEntityRef: expecting ';' in Entity" in the developer mode
* GitHub requests:
  * [#43](#43) -- implemented logging of executed setup files
  * [#44](#44)
    * Implemented support of writing logs into wrappers (for example, `php://output`)
    * Enforced a log writer model to be an instance of `Zend_Log_Writer_Stream`
  * [#49](#49)
    * Fixed sorting of totals according to "before" and "after" properties
    * Introduced `Magento_Data_Graph` library class and utilized it for finding cycles in "before" and "after" declarations
    * Implemented tests for totals sorting including the ambiguous cases
c0cf1af
mmansoorebay mmansoorebay referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 2 unique commits by 1 author.

Jul 24, 2012
Ivan Chepurnyi Fixed core issue related to invalid totals sorting in the quote. The …
…problem is visible if you have additional custom totals that need to be placed in between core ones.
9a3631e
Aug 17, 2012
Ivan Chepurnyi Added intergration test for testing totals sorting algorithm 3b17a80
This page is out of date. Refresh to see the latest.
56  app/code/core/Mage/Sales/Model/Config/Ordered.php
@@ -140,36 +140,46 @@ protected function _getSortedCollectorCodes()
140 140
         $element = current($configArray);
141 141
         if (isset($element['sort_order'])) {
142 142
             uasort($configArray, array($this, '_compareSortOrder'));
  143
+            $sortedCollectors = array_keys($configArray);
143 144
         } else {
144  
-            foreach ($configArray as $code => $data) {
145  
-                foreach ($data['before'] as $beforeCode) {
146  
-                    if (!isset($configArray[$beforeCode])) {
  145
+            $sortedCollectors = array_keys($configArray);
  146
+            // Move all totals with before specification in front of related total
  147
+
  148
+            foreach ($configArray as $code => &$data) {
  149
+                foreach ($data['before'] as $positionCode) {
  150
+                    if (!isset($configArray[$positionCode])) {
147 151
                         continue;
148 152
                     }
149  
-                    $configArray[$code]['before'] = array_unique(array_merge(
150  
-                        $configArray[$code]['before'], $configArray[$beforeCode]['before']
151  
-                    ));
152  
-                    $configArray[$beforeCode]['after'] = array_merge(
153  
-                        $configArray[$beforeCode]['after'], array($code), $data['after']
154  
-                    );
155  
-                    $configArray[$beforeCode]['after'] = array_unique($configArray[$beforeCode]['after']);
156  
-                }
157  
-                foreach ($data['after'] as $afterCode) {
158  
-                    if (!isset($configArray[$afterCode])) {
159  
-                        continue;
  153
+                    if (!in_array($code, $configArray[$positionCode]['after'], true)) {
  154
+                        // Also add additional after condition for related total,
  155
+                        // to keep it always after total with before value specified
  156
+                        $configArray[$positionCode]['after'][] = $code;
160 157
                     }
161  
-                    $configArray[$code]['after'] = array_unique(array_merge(
162  
-                        $configArray[$code]['after'], $configArray[$afterCode]['after']
163  
-                    ));
164  
-                    $configArray[$afterCode]['before'] = array_merge(
165  
-                        $configArray[$afterCode]['before'], array($code), $data['before']
166  
-                    );
167  
-                    $configArray[$afterCode]['before'] = array_unique($configArray[$afterCode]['before']);
  158
+                    $currentPosition = array_search($code, $sortedCollectors, true);
  159
+                    $desiredPosition = array_search($positionCode, $sortedCollectors, true);
  160
+                    if ($currentPosition > $desiredPosition) {
  161
+                        // Only if current position is not corresponding to before condition
  162
+                        array_splice($sortedCollectors, $currentPosition, 1); // Removes existent
  163
+                        array_splice($sortedCollectors, $desiredPosition, 0, $code); // Add at new position
  164
+                    }
  165
+                }
  166
+            }
  167
+            // Sort out totals with after position specified
  168
+            foreach ($configArray as $code => &$data) {
  169
+                $maxAfter = null;
  170
+                $currentPosition = array_search($code, $sortedCollectors, true);
  171
+
  172
+                foreach ($data['after'] as $positionCode) {
  173
+                    $maxAfter = max($maxAfter, array_search($positionCode, $sortedCollectors, true));
  174
+                }
  175
+
  176
+                if ($maxAfter !== null && $maxAfter > $currentPosition) {
  177
+                    // Moves only if it is in front of after total
  178
+                    array_splice($sortedCollectors, $maxAfter + 1, 0, $code); // Add at new position
  179
+                    array_splice($sortedCollectors, $currentPosition, 1); // Removes existent
168 180
                 }
169 181
             }
170  
-            uasort($configArray, array($this, '_compareTotals'));
171 182
         }
172  
-        $sortedCollectors = array_keys($configArray);
173 183
         if (Mage::app()->useCache('config')) {
174 184
             Mage::app()->saveCache(serialize($sortedCollectors), $this->_collectorsCacheKey, array(
175 185
                     Mage_Core_Model_Config::CACHE_TAG
202  dev/tests/integration/testsuite/Mage/Sales/Model/Config/OrderedTest.php
... ...
@@ -0,0 +1,202 @@
  1
+<?php
  2
+/**
  3
+ * Magento
  4
+ *
  5
+ * NOTICE OF LICENSE
  6
+ *
  7
+ * This source file is subject to the Open Software License (OSL 3.0)
  8
+ * that is bundled with this package in the file LICENSE.txt.
  9
+ * It is also available through the world-wide-web at this URL:
  10
+ * http://opensource.org/licenses/osl-3.0.php
  11
+ * If you did not receive a copy of the license and are unable to
  12
+ * obtain it through the world-wide-web, please send an email
  13
+ * to license@magentocommerce.com so we can send you a copy immediately.
  14
+ *
  15
+ * DISCLAIMER
  16
+ *
  17
+ * Do not edit or add to this file if you wish to upgrade Magento to newer
  18
+ * versions in the future. If you wish to customize Magento for your
  19
+ * needs please refer to http://www.magentocommerce.com for more information.
  20
+ *
  21
+ * @category    Magento
  22
+ * @package     Mage_Sales
  23
+ * @subpackage  integration_tests
  24
+ * @copyright   Copyright (c) 2012 Magento Inc. (http://www.magentocommerce.com)
  25
+ * @license     http://opensource.org/licenses/osl-3.0.php  Open Software License (OSL 3.0)
  26
+ */
  27
+
  28
+/**
  29
+ * Integration test for testing order config class
  30
+ *
  31
+ * Not possible to make as a unit test, since internally app object is called
  32
+ */
  33
+class Mage_Sales_Model_Config_OrderedTest extends PHPUnit_Framework_TestCase
  34
+{
  35
+    /**
  36
+     * Flag for checking if needed restoring of cache usage feature
  37
+     *
  38
+     * @var bool
  39
+     */
  40
+    protected $_restoreUseCache = false;
  41
+
  42
+    /**
  43
+     * Model under test
  44
+     *
  45
+     * @var Mage_Sales_Model_Config_Ordered
  46
+     */
  47
+    protected $_model = null;
  48
+
  49
+    /**
  50
+     * Disables configuration cache, sets up model
  51
+     *
  52
+     */
  53
+    protected function setUp()
  54
+    {
  55
+        $this->_restoreUseCache = Mage::app()->useCache('config');
  56
+        $this->_model = $this->getMockForAbstractClass('Mage_Sales_Model_Config_Ordered');
  57
+        Mage::app()->getCacheInstance()->banUse('config');
  58
+
  59
+    }
  60
+
  61
+    /**
  62
+     * Test total collector sorting algorithm
  63
+     *
  64
+     * @dataProvider totalCollectors
  65
+     */
  66
+    public function testGetSortedCollectorCodes($totalConfig)
  67
+    {
  68
+        $reflection = new ReflectionObject($this->_model);
  69
+        // Fill in prepared data for test
  70
+        $property = $reflection->getProperty('_modelsConfig');
  71
+        $property->setAccessible(true);
  72
+        $property->setValue($this->_model, $totalConfig);
  73
+        $property->setAccessible(false);
  74
+
  75
+        // Calling sorting method
  76
+        $method = $reflection->getMethod('_getSortedCollectorCodes');
  77
+        $method->setAccessible(true);
  78
+        $result = $method->invoke($this->_model);
  79
+
  80
+        $this->assertInternalType('array', $result, 'Result of method call is not an array');
  81
+
  82
+        // Evaluating the result
  83
+        foreach ($totalConfig as $total) {
  84
+            $totalPosition = array_search($total['_code'], $result);
  85
+
  86
+            // Walking through total after positions,
  87
+            // to check that our total really placed after them
  88
+            foreach ($total['after'] as $afterTotal) {
  89
+                $afterTotalPosition = array_search($afterTotal, $result);
  90
+                $this->assertLessThan(
  91
+                    $totalPosition, $afterTotalPosition,
  92
+                    sprintf('Total with code "%s" is not after "%s"', $total['_code'], $afterTotal)
  93
+                );
  94
+            }
  95
+
  96
+            // Walking through total before positions,
  97
+            // to check that our total really placed before them
  98
+            foreach ($total['before'] as $beforeTotal) {
  99
+                $beforeTotalPosition = array_search($beforeTotal, $result);
  100
+                $this->assertGreaterThan(
  101
+                    $totalPosition, $beforeTotalPosition,
  102
+                    sprintf('Total with code "%s" is not before "%s"', $total['_code'], $beforeTotal)
  103
+                );
  104
+            }
  105
+        }
  106
+    }
  107
+
  108
+    /**
  109
+     * Test data provider for testing totals sorting algorithm
  110
+     *
  111
+     * @return array
  112
+     */
  113
+    public function totalCollectors()
  114
+    {
  115
+        $coreTotals = array(
  116
+            // Totals defined in Mage_Sales
  117
+            'nominal'       => array('_code'  => 'nominal',
  118
+                                     'before' => array('subtotal'),
  119
+                                     'after'  => array()),
  120
+
  121
+            'subtotal'      => array('_code'  => 'subtotal',
  122
+                                     'after'  => array('nominal'),
  123
+                                     'before' => array('grand_total')),
  124
+
  125
+            'shipping'      => array('_code'  => 'shipping',
  126
+                                     'after'  => array('subtotal', 'freeshipping', 'tax_subtotal'),
  127
+                                     'before' => array('grand_total')),
  128
+
  129
+            'grand_total'   => array('_code'  => 'grand_total',
  130
+                                     'after'  => array('subtotal'),
  131
+                                     'before' => array()),
  132
+
  133
+            'msrp'          => array('_code'  => 'grand_total',
  134
+                                     'after'  => array(),
  135
+                                     'before' => array()),
  136
+            // Totals defined in Mage_SalesRule
  137
+            'freeshipping'  => array('_code'  => 'freeshipping',
  138
+                                     'after'  => array('subtotal'),
  139
+                                     'before' => array('tax_subtotal', 'shipping')),
  140
+
  141
+            'discount'      => array('_code'  => 'discount',
  142
+                                     'after'  => array('subtotal', 'shipping'),
  143
+                                     'before' => array('grand_total')),
  144
+            // Totals defined in Mage_Tax
  145
+            'tax_subtotal'  => array('_code'  => 'tax_subtotal',
  146
+                                     'after'  => array('freeshipping'),
  147
+                                     'before' => array('tax', 'discount')),
  148
+
  149
+            'tax_shipping'  => array('_code'  => 'tax_shipping',
  150
+                                     'after'  => array('shipping'),
  151
+                                     'before' => array('tax', 'discount')),
  152
+
  153
+            'tax'           => array('_code'  => 'tax',
  154
+                                     'after'  => array('subtotal','shipping'),
  155
+                                     'before' => array('grand_total')),
  156
+            // Totals defined in Mage_Wee
  157
+            'wee'           => array('_code'  => 'wee',
  158
+                                     'after'  => array('subtotal','tax','discount','grand_total','shipping'),
  159
+                                     'before' => array())
  160
+        );
  161
+        return array(
  162
+            array($coreTotals), // Test case with just core totals
  163
+            array($coreTotals + array( // Test case with custom totals
  164
+                'handling'     => array('_code' => 'handling',
  165
+                                        'after' => array('shipping'),
  166
+                                        'before' => array('tax')),
  167
+                'handling_tax' => array('_code' => 'handling_tax',
  168
+                                        'after' => array('tax_shipping'),
  169
+                                        'before' => array('tax'))
  170
+            )),
  171
+            array($coreTotals + array( // Test case with more custom totals
  172
+                                       // (this one fails with non fixed core functionality)
  173
+                'handling'     => array('_code' => 'handling',
  174
+                                        'after' => array('shipping'),
  175
+                                        'before' => array('tax')),
  176
+                'handling_tax' => array('_code' => 'handling_tax',
  177
+                                        'after' => array('tax_shipping'),
  178
+                                        'before' => array('tax')),
  179
+                'own_subtotal' => array('_code' => 'own_subtotal',
  180
+                                        'after' => array('nominal'),
  181
+                                        'before' => array('subtotal')),
  182
+                'own_total1'   => array('_code' => 'own_total1',
  183
+                                        'after' => array('nominal'),
  184
+                                        'before' => array('subtotal')),
  185
+                'own_total2'   => array('_code' => 'own_total2',
  186
+                                        'after' => array('nominal'),
  187
+                                        'before' => array('subtotal'))
  188
+            ))
  189
+        );
  190
+    }
  191
+
  192
+    /**
  193
+     * Restores cache usage options
  194
+     *
  195
+     */
  196
+    protected function tearDown()
  197
+    {
  198
+        if ($this->_restoreUseCache) {
  199
+            Mage::app()->getCacheInstance()->allowUse('config');
  200
+        }
  201
+    }
  202
+}
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.