Improve performance of `solve_strongly_connected_components` for models with named expressions #3186

Robbybp · 2024-03-07T23:47:59Z

Summary/Motivation:

Working with a small IDAES model, I noticed that solve_strongly_connected_components was much slower than I would have expected. This is for a model with about 900 variables and constraints:

Identifier                                    ncalls   cumtime   percall      %                                                                    
-------------------------------------------------------------------------------                                                                    
root                                               1    40.800    40.800  100.0                                                                    
     --------------------------------------------------------------------------                                                                    
     full model post-solve                         1     0.463     0.463    1.1                                                                    
     solve-scc                                     1    40.337    40.337   98.9                                                                    
                          -----------------------------------------------------                                                                    
                          calc-var-from-con      530    20.910     0.039   51.8                                                                    
                          generate-scc           547    11.948     0.022   29.6                                                                    
                          igraph                   1     7.371     7.371   18.3                                                                    
                          scc-subsolver           16     0.100     0.006    0.2                                                                    
                          other                  n/a     0.008       n/a    0.0                                                                    
                          =====================================================                                                                    
     other                                       n/a     0.000       n/a    0.0                                                                    
     ==========================================================================                                                                    
===============================================================================

solve_strongly_connected_components takes 40 s, half of which is in calculate_variable_from_constraint. This prompted me to review a lot of the infrastructure that solve_strongly_connected_components relies on and look for ways to speed things up. Most of the performance gains I will demonstrate come from not repeating work for the same named expressions, although reusing incidence graphs instead of walking expressions helps as well.

Changes proposed in this PR:

Add an option for solve_strongly_connected_components to avoid using calculate_variable_from_constraint. This can be beneficial for models with large, nested named expressions, which the NL writer exploits but calculate_variable_from_constraint does not.
Use IncidenceMethod.ampl_repn when constructing incidence graphs. Again, this exploits named expressions, where IncidenceMethod.standard_repn does not.
Reuse the already existing incidence graph for the block triangularization in generate_strongly_connected_components
Update _ExternalFunctionVisitor and add_local_external_functions to exploit named expressions
Add an option for TemporarySubsystemManager to temporarily remove bounds on fixed variables. This was the motivation for the suggestion in Unavoidable InfeasibleConstraintException in NL writer for variable fixed outside of bounds #3179

The timing breakdown after these improvements looks like:

Identifier                                                           ncalls   cumtime   percall      %                     
------------------------------------------------------------------------------------------------------                     
root                                                                      1     4.159     4.159  100.0                     
     -------------------------------------------------------------------------------------------------                     
     full model post-solve                                                1     0.445     0.445   10.7                     
     solve-scc                                                            1     3.713     3.713   89.3                     
                          ----------------------------------------------------------------------------                     
                          generate-scc                                  547     2.431     0.004   65.5                     
                                       ---------------------------------------------------------------                     
                                       block-triang                       1     0.033     0.033    1.4                     
                                       generate-block                   547     2.397     0.004   98.6                     
                                                     -------------------------------------------------                     
                                                     block              546     0.015     0.000    0.6                     
                                                     external-fcns      546     0.341     0.001   14.2                     
                                                     identify-vars      546     1.999     0.004   83.4                     
                                                     reference         1092     0.036     0.000    1.5                     
                                                     other              n/a     0.007       n/a    0.3                     
                                                     =================================================                     
                                       other                            n/a     0.002       n/a    0.1                     
                                       ===============================================================                     
                          igraph                                          1     0.347     0.347    9.3                     
                          scc-subsolver                                 546     0.922     0.002   24.8                     
                          other                                         n/a     0.012       n/a    0.3                     
                          ============================================================================                     
     other                                                              n/a     0.000       n/a    0.0                     
     =================================================================================================                     
======================================================================================================

We are now bottlenecked by identify_variables, of all things.

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

I agree my contributions are submitted under the BSD license.
I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

…itor

…mManager

… in solve_strongly_connected_components

…f one already exists

…tion of identify_external_functions to be a generator

…mance-2

jsiirola

Looks good. A couple questions / suggestions, but nothing that would prevent merging

pyomo/util/subsystems.py

jsiirola · 2024-03-11T14:53:57Z

FYI: Jenkins test failures appear to be real:

=================================== FAILURES ===================================
___ TestSubsystemBlock.test_external_function_with_potential_name_collision ____

self = <pyomo.util.tests.test_subsystems.TestSubsystemBlock testMethod=test_external_function_with_potential_name_collision>

    @unittest.skipUnless(find_GSL(), "Could not find the AMPL GSL library")
    def test_external_function_with_potential_name_collision(self):
>       m = self._make_model_with_external_functions()

pyomo/pyomo/util/tests/test_subsystems.py:412: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pyomo/pyomo/util/tests/test_subsystems.py:311: in _make_model_with_external_functions
    subexpr3 = m.subexpr[2] + m.v3**2
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pyomo.core.base.PyomoModel.ConcreteModel object at 0x7f889ebce6c0>
val = 'subexpr'

    def __getattr__(self, val):
        if val in ModelComponentFactory:
            return _component_decorator(self, ModelComponentFactory.get_class(val))
        # Since the base classes don't support getattr, we can just
        # throw the "normal" AttributeError
>       raise AttributeError(
            "'%s' object has no attribute '%s'" % (self.__class__.__name__, val)
        )
E       AttributeError: 'ConcreteModel' object has no attribute 'subexpr'

pyomo/pyomo/core/base/block.py:547: AttributeError
_____________ TestSubsystemBlock.test_identify_external_functions ______________

self = <pyomo.util.tests.test_subsystems.TestSubsystemBlock testMethod=test_identify_external_functions>

    @unittest.skipUnless(find_GSL(), "Could not find the AMPL GSL library")
    def test_identify_external_functions(self):
>       m = self._make_model_with_external_functions()

pyomo/pyomo/util/tests/test_subsystems.py:319: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pyomo/pyomo/util/tests/test_subsystems.py:311: in _make_model_with_external_functions
    subexpr3 = m.subexpr[2] + m.v3**2
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pyomo.core.base.PyomoModel.ConcreteModel object at 0x7f889ebbb930>
val = 'subexpr'

    def __getattr__(self, val):
        if val in ModelComponentFactory:
            return _component_decorator(self, ModelComponentFactory.get_class(val))
        # Since the base classes don't support getattr, we can just
        # throw the "normal" AttributeError
>       raise AttributeError(
            "'%s' object has no attribute '%s'" % (self.__class__.__name__, val)
        )
E       AttributeError: 'ConcreteModel' object has no attribute 'subexpr'

pyomo/pyomo/core/base/block.py:547: AttributeError
___ TestSubsystemBlock.test_local_external_functions_with_named_expressions ____

self = <pyomo.util.tests.test_subsystems.TestSubsystemBlock testMethod=test_local_external_functions_with_named_expressions>

    @unittest.skipUnless(find_GSL(), "Could not find the AMPL GSL library")
    def test_local_external_functions_with_named_expressions(self):
        m = self._make_model_with_external_functions(named_expressions=True)
>       variables = list(pyo.component_data_objects(pyo.Var))

pyomo/pyomo/util/tests/test_subsystems.py:344: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

name = 'component_data_objects'

    def __getattr__(name):
        info = _relocated.get(name, None)
        if info is not None:
            target_obj = _import_object(name, *info)
            f_globals[name] = target_obj
            return target_obj
        elif _mod_getattr is not None:
            return _mod_getattr(name)
>       raise AttributeError(
            "module '%s' has no attribute '%s'" % (f_globals['__name__'], name)
        )
E       AttributeError: module 'pyomo.environ' has no attribute 'component_data_objects'

pyomo/pyomo/common/deprecation.py:439: AttributeError
________________ TestSubsystemBlock.test_with_external_function ________________

self = <pyomo.util.tests.test_subsystems.TestSubsystemBlock testMethod=test_with_external_function>

    @unittest.skipUnless(find_GSL(), "Could not find the AMPL GSL library")
    @unittest.skipUnless(
        pyo.SolverFactory("ipopt").available(), "ipopt is not available"
    )
    def test_with_external_function(self):
>       m = self._make_model_with_external_functions()

pyomo/pyomo/util/tests/test_subsystems.py:361: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pyomo/pyomo/util/tests/test_subsystems.py:311: in _make_model_with_external_functions
    subexpr3 = m.subexpr[2] + m.v3**2
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pyomo.core.base.PyomoModel.ConcreteModel object at 0x7f889ebf2940>
val = 'subexpr'

    def __getattr__(self, val):
        if val in ModelComponentFactory:
            return _component_decorator(self, ModelComponentFactory.get_class(val))
        # Since the base classes don't support getattr, we can just
        # throw the "normal" AttributeError
>       raise AttributeError(
            "'%s' object has no attribute '%s'" % (self.__class__.__name__, val)
        )
E       AttributeError: 'ConcreteModel' object has no attribute 'subexpr'

pyomo/pyomo/core/base/block.py:547: AttributeError
______ TestSubsystemBlock.test_with_external_function_in_named_expression ______

self = <pyomo.util.tests.test_subsystems.TestSubsystemBlock testMethod=test_with_external_function_in_named_expression>

    @unittest.skipUnless(find_GSL(), "Could not find the AMPL GSL library")
    @unittest.skipUnless(
        pyo.SolverFactory("ipopt").available(), "ipopt is not available"
    )
    def test_with_external_function_in_named_expression(self):
        m = self._make_model_with_external_functions(named_expressions=True)
        subsystem = ([m.con2, m.con3], [m.v2, m.v3])
    
        m.v1.set_value(0.5)
        block = create_subsystem_block(*subsystem)
        ipopt = pyo.SolverFactory("ipopt")
        with TemporarySubsystemManager(to_fix=list(block.input_vars.values())):
            ipopt.solve(block)
    
        # Correct values obtained by solving with Ipopt directly
        # in another script.
        self.assertEqual(m.v1.value, 0.5)
        self.assertFalse(m.v1.fixed)
        self.assertAlmostEqual(m.v2.value, 1.04816, delta=1e-5)
        self.assertAlmostEqual(m.v3.value, 1.34356, delta=1e-5)
    
        # Result obtained by solving the full system
>       m_full = self._solve_ef_model_with_ipopt()

pyomo/pyomo/util/tests/test_subsystems.py:405: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pyomo/pyomo/util/tests/test_subsystems.py:351: in _solve_ef_model_with_ipopt
    m = self._make_model_with_external_functions()
pyomo/pyomo/util/tests/test_subsystems.py:311: in _make_model_with_external_functions
    subexpr3 = m.subexpr[2] + m.v3**2
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pyomo.core.base.PyomoModel.ConcreteModel object at 0x7f889ebf2800>
val = 'subexpr'

    def __getattr__(self, val):
        if val in ModelComponentFactory:
            return _component_decorator(self, ModelComponentFactory.get_class(val))
        # Since the base classes don't support getattr, we can just
        # throw the "normal" AttributeError
>       raise AttributeError(
            "'%s' object has no attribute '%s'" % (self.__class__.__name__, val)
        )
E       AttributeError: 'ConcreteModel' object has no attribute 'subexpr'

pyomo/pyomo/core/base/block.py:547: AttributeError
------------------------------ Captured log call -------------------------------
WARNING  pyomo.core:external.py:438 AMPL External function: ignoring AtReset call in external library.  This may result in a memory leak or other undesirable behavior.
WARNING  pyomo.core:external.py:438 AMPL External function: ignoring AtReset call in external library.  This may result in a memory leak or other undesirable behavior.
=========================== short test summary info ============================
FAILED pyomo/pyomo/util/tests/test_subsystems.py::TestSubsystemBlock::test_external_function_with_potential_name_collision - AttributeError: 'ConcreteModel' object has no attribute 'subexpr'
FAILED pyomo/pyomo/util/tests/test_subsystems.py::TestSubsystemBlock::test_identify_external_functions - AttributeError: 'ConcreteModel' object has no attribute 'subexpr'
FAILED pyomo/pyomo/util/tests/test_subsystems.py::TestSubsystemBlock::test_local_external_functions_with_named_expressions - AttributeError: module 'pyomo.environ' has no attribute 'component_data_objects'
FAILED pyomo/pyomo/util/tests/test_subsystems.py::TestSubsystemBlock::test_with_external_function - AttributeError: 'ConcreteModel' object has no attribute 'subexpr'
FAILED pyomo/pyomo/util/tests/test_subsystems.py::TestSubsystemBlock::test_with_external_function_in_named_expression - AttributeError: 'ConcreteModel' object has no attribute 'subexpr'
==== 5 failed, 13172 passed, 4631 skipped, 49 xfailed in 3428.47s (0:57:08) ====

mrmundt · 2024-03-11T21:27:37Z

Real failure again, @Robbybp

self = <pyomo.util.tests.test_subsystems.TestSubsystemBlock testMethod=test_local_external_functions_with_named_expressions>

    @unittest.skipUnless(find_GSL(), "Could not find the AMPL GSL library")
    def test_local_external_functions_with_named_expressions(self):
        m = self._make_model_with_external_functions(named_expressions=True)
>       variables = list(pyo.component_data_objects(pyo.Var))

/pyomo/pyomo/util/tests/test_subsystems.py:344: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

name = 'component_data_objects'

    def __getattr__(name):
        info = _relocated.get(name, None)
        if info is not None:
            target_obj = _import_object(name, *info)
            f_globals[name] = target_obj
            return target_obj
        elif _mod_getattr is not None:
            return _mod_getattr(name)
>       raise AttributeError(
            "module '%s' has no attribute '%s'" % (f_globals['__name__'], name)
        )
E       AttributeError: module 'pyomo.environ' has no attribute 'component_data_objects'

/pyomo/pyomo/common/deprecation.py:439: AttributeError

…mance-2

codecov · 2024-03-12T20:00:44Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.39%. Comparing base (b06ddea) to head (4f1e20c).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3186      +/-   ##
==========================================
- Coverage   88.39%   88.39%   -0.01%     
==========================================
  Files         846      846              
  Lines       95091    95118      +27     
==========================================
+ Hits        84058    84079      +21     
- Misses      11033    11039       +6

Flag	Coverage Δ
linux	`86.33% <80.00%> (-0.01%)`	⬇️
osx	`76.16% <80.00%> (-0.01%)`	⬇️
other	`86.53% <80.00%> (-0.02%)`	⬇️
win	`83.82% <80.00%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Robbybp · 2024-03-13T15:54:46Z

Attention: Patch coverage is 80.00000% with 9 lines in your changes are missing coverage. Please review.

My latest commits should address this.

Robbybp added 19 commits February 26, 2024 12:32

timing calls and some performance improvements

28cf069

dont repeat work for the same named expression in ExternalFunctionVis…

2b47212

…itor

scc_solver implementation using SccImplicitFunctionSolver

7aaaeed

additional timing calls in SccImplicitFunctionSolver

aaecb45

add option to remove bounds from fixed variables to TemporarySubsyste…

dd27f66

…mManager

timing calls and option to not use calculate_variable_from_constraint…

1a1c1a8

… in solve_strongly_connected_components

timing calls in generate-scc and option to reuse an incidence graph i…

3d7f2c3

…f one already exists

Merge branch 'scc-performance' into scc-performance-2

51bfe4f

remove unnecessary import

3b5aaa6

remove unnecessary imports

565a29e

pass timer to generate_subsystem_blocks

df4f7af

accept timer argument in generate_subsystem_blocks, revert implementa…

f47345c

…tion of identify_external_functions to be a generator

add tests with external functions in named expressions

0b252ae

document igraph option in generate_scc

9128679

remove redundant implementation of acceptChildResult

7299a79

comment out unnecessary walker methods

8b66e10

apply black

70f225e

Merge branch 'main' of https://github.com/pyomo/pyomo into scc-perfor…

51594d6

…mance-2

arguments on one line to keep black happy

b96d12e

Robbybp changed the title ~~Improve performance of solve_strongly_connected_components for models with large expression trees~~ Improve performance of solve_strongly_connected_components for models with named expressions Mar 8, 2024

Robbybp mentioned this pull request Mar 8, 2024

Exploit repeated named expressions when identifying variables in expressions #3187

Closed

Robbybp added 2 commits March 7, 2024 18:02

use_calc_var default should be True

ec4f733

re-add exception I accidentally deleted

cd54e6f

jsiirola approved these changes Mar 9, 2024

View reviewed changes

pyomo/util/subsystems.py Outdated Show resolved Hide resolved

pyomo/util/subsystems.py Outdated Show resolved Hide resolved

pyomo/util/subsystems.py Outdated Show resolved Hide resolved

jsiirola added the AT: PRE-TEST INSPECTED label Mar 9, 2024

pyomo-autotest removed the AT: PRE-TEST INSPECTED label Mar 9, 2024

fix attribute error

a99165c

mrmundt added the AT: PRE-TEST INSPECTED label Mar 11, 2024

pyomo-autotest removed the AT: PRE-TEST INSPECTED label Mar 11, 2024

Robbybp added 5 commits March 11, 2024 14:29

check class in native_types rather than isinstance NumericValue

b3307c9

remove unnecessary exitNode and acceptChildResult implementations

a27f90d

remove deferred todo comment

3ecd3bb

remove HierarchicalTimer use from subsystem calls

fde5eda

remove hierarchical timer usage from scc_solver module

40debe7

Robbybp added 2 commits March 11, 2024 18:03

fix typos in test

0e9af93

function args on single line

016d6d1

mrmundt added the AT: PRE-TEST INSPECTED label Mar 12, 2024

pyomo-autotest removed the AT: PRE-TEST INSPECTED label Mar 12, 2024

Robbybp added 2 commits March 12, 2024 10:30

remove outdated comment

24644ff

Merge branch 'main' of https://github.com/pyomo/pyomo into scc-perfor…

88e0e42

…mance-2

mrmundt added the AT: PRE-TEST INSPECTED label Mar 12, 2024

blnicho self-requested a review March 12, 2024 18:39

pyomo-autotest removed the AT: PRE-TEST INSPECTED label Mar 12, 2024

Robbybp and others added 2 commits March 12, 2024 15:39

fix variable assignment

de294ee

Merge branch 'main' into scc-performance-2

4f1e20c

jsiirola approved these changes Mar 13, 2024

View reviewed changes

jsiirola added the AT: PRE-TEST INSPECTED label Mar 13, 2024

pyomo-autotest removed the AT: PRE-TEST INSPECTED label Mar 13, 2024

mrmundt approved these changes Mar 13, 2024

View reviewed changes

mrmundt merged commit 1b5aa97 into Pyomo:main Mar 13, 2024
33 checks passed

Robbybp deleted the scc-performance-2 branch March 13, 2024 16:25

Robbybp mentioned this pull request Mar 15, 2024

Exploit repeated named expressions in identify_variables #3190

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve performance of `solve_strongly_connected_components` for models with named expressions #3186

Improve performance of `solve_strongly_connected_components` for models with named expressions #3186

Robbybp commented Mar 7, 2024

jsiirola left a comment

jsiirola commented Mar 11, 2024

mrmundt commented Mar 11, 2024

codecov bot commented Mar 12, 2024 •

edited

Robbybp commented Mar 13, 2024

Improve performance of solve_strongly_connected_components for models with named expressions #3186

Improve performance of solve_strongly_connected_components for models with named expressions #3186

Conversation

Robbybp commented Mar 7, 2024

Summary/Motivation:

Changes proposed in this PR:

Legal Acknowledgement

jsiirola left a comment

Choose a reason for hiding this comment

jsiirola commented Mar 11, 2024

mrmundt commented Mar 11, 2024

codecov bot commented Mar 12, 2024 • edited

Codecov Report

Robbybp commented Mar 13, 2024

Improve performance of `solve_strongly_connected_components` for models with named expressions #3186

Improve performance of `solve_strongly_connected_components` for models with named expressions #3186

codecov bot commented Mar 12, 2024 •

edited