Solver improvements via IPOPT #124

richardotis · 2017-09-14T22:22:07Z

This really big PR does a bunch of things in the core:

Adds the NLP solver IPOPT and its Cython wrapper cyipopt as dependencies (Conda-forge support for ipopt on Windows is pending)
Removes the majority of the old eqsolver core, while leaving the simplistic treatment of global minimization intact (for now)
Adds deepcopy support for CompositionSet
Creates new Problem and Solver abstractions, which will make it easier to extend the solver in the future to handle different types of problems and constraints
Tweaks minimum value constants to improve convergence
Make some equilibrium tests more restrictive

This PR has not been checked against any of the phase diagram notebooks yet.

…ofiling

…dd-remove-phases step of computation

…d reimplemented

…s. Tweak convergence criteria

…memory Conflicts: pycalphad/core/eqsolver.pyx

…were not run

…ble getting tests to finish.

bocklund

Great work.

Once these are fixed, we are just waiting on CI for this to be merged, yes?

bocklund · 2017-11-04T18:42:58Z

pycalphad/core/compiled_model.pyx

@@ -1,8 +1,13 @@
+# cython: linetrace=True


Should remove # cython: linetrace=True and # cython: profile=True for performance.

bocklund · 2017-11-05T15:48:20Z

pycalphad/core/hyperplane.pyx

+        saved_trial = argmax(smallest_fractions)
+        #print('saved_trial', saved_trial)
+        if smallest_fractions[saved_trial] < -num_components:
+            print('hyperplane: No further progress is possible')


Remove commented out print statements (lines 127-129) and also the 'hyperplane: No further progress is possible' on line 134

In addition lines 142, 143, 145, 147, 160.

Feel free to uncomment and print if verbose=True for any of these that you would like.

bocklund · 2017-11-05T15:50:28Z

pycalphad/core/hyperplane.pyx

        for i in range(num_components):
            trial_simplices[i, i] = min_df
        if lowest_df[0] > -1e-8:
            break
-    if lowest_df[0] < -1e-8:
-        raise ValueError('Max hull iterations exceeded. Remaining driving force: ', lowest_df[0])
+    #if lowest_df[0] < -1e-8:


Should this be commented? Remove if necessary.

bocklund · 2017-11-05T15:58:10Z

pycalphad/core/problem.pyx

+            constraint_offset += 1
+        return np.array(constraint_jac)
+
+    #def hessian(self, x, lagrange, obj_factor):


Do we need lines 192-214?

bocklund · 2017-11-05T19:51:24Z

pycalphad/core/compiled_model.pxd

@@ -46,13 +46,13 @@ cdef public class CompiledModel(object)[type CompiledModelType, object CompiledM
                                double *eval_row, double[:] parameters) nogil


I think cymem is dead now so the following needs to be cleaned up

compiled_model.pxd

cdef public Pool mem in line 39.

As well as the import in line 2.

In compiled_model.pyx:

self.mem = Pool()(line 56)

import in line 2

inst.mem in line 1071

Pool() in line 1018

In addition the

cymem.pyx and pxd

cymem license

cymem cythonize in setup.py

bocklund · 2017-11-08T10:30:23Z

pycalphad/core/solver.py

+        length_scale = max(length_scale, 1e-9)
+        # nlp.addOption(b'derivative_test', b'first-order')
+        # nlp.addOption(b'check_derivatives_for_naninf', b'yes')
+        MAX_SOLVE_DRIVING_FORCE = 1e-4


Should the MAX_SOLVE_DRIVING_FORCE and other tolerances here be set in the constants module? That way it could be reasonably found and changed at compile time. I think you mentioned moving towards being able to set these at run time, which would be more easily supported by that kind of change.

I could see advanced users (and myself) wanting any easy way to tweak the solver performance and recompile.

bocklund · 2017-11-08T10:33:04Z

pycalphad/core/solver.py

+                converged = False
+        elif info['status'] < 0:
+            if self.verbose:
+                print('Calculation Failed: ', cur_conds, info['status_msg'])


Should calculation failures be noisier as in the current solver? Or is this a different kind of error?

This error means that ipopt failed to find a solution, which should be pretty rare now.

bocklund · 2017-11-08T11:47:46Z

pycalphad/tests/test_equilibrium.py

-    assert np.all(np.abs(mass_error) < 0.01)
+    assert_allclose(eq.GM.values, -105871.20, atol=0.1)
+    assert_allclose(eq.MU.values.flatten(), [-104655.532294, -142591.644379,  -82905.085459], atol=0.1)
+    #assert_allclose(eq.GM.values, -105871.54, atol=1.0)


Can we remove the commented out assertion?

bocklund · 2017-11-08T11:48:59Z

pycalphad/tests/test_equilibrium.py

@@ -217,6 +221,9 @@ def test_eq_issue43_chempots_misc_gap():
                     {v.X('AL'): .1246, v.X('CR'): 1e-9, v.T: 1273, v.P: 101325},
                     verbose=True)
    chempots = 8.31451 * np.squeeze(eq['T'].values) * np.array([[[[[-19.47631644, -25.71249032,  -6.0706158]]]]])
+    mass_error = np.nansum(np.squeeze(eq.NP * eq.X), axis=-2) - \
+                 [0.1246, 1e-9, 1-(.1246+1e-9)]
+    print('Mass error', mass_error)


Should we be doing some assertion here with the mass error or can it be removed?

bocklund · 2017-11-08T11:52:40Z

setup.py

@@ -35,7 +36,7 @@ def read(fname):
    long_description=read('README.rst'),
    url='https://pycalphad.org/',
    install_requires=['matplotlib', 'pandas', 'xarray!=0.8', 'sympy', 'pyparsing', 'Cython>=0.24',
-                      'tinydb', 'scipy', 'numpy>=1.9', 'dask[complete]>=0.10', 'dill'],
+                      'tinydb', 'scipy', 'numpy>=1.9', 'dask[complete]>=0.10', 'dill', 'ipopt'],


I think the PyPI ipopt is a little behind cyipopt. Do we need to be involved in maintaining ipopt bindings PyPI as well?

The friendly @moorepants is currently maintaining the bindings on PyPI, so I think if we sell upstream on Windows support, the release will follow shortly.

The previous developer has given me access to the ipopt pypi but i've never updated it, just been using conda-forge's version which is a fork. I think the right move is to start using the name cyipopt on pypi and push from the fork: https://github.com/matthias-k/cyipopt, and abandone the ipopt name on pypi.

@moorepants I support you on the name change. Moving forward with it now would be great since we could write it in before pycalphad's next major release (T-minus ~1 week).

Someone needs to make a PR with the name changes to: https://github.com/matthias-k/cyipopt. There is already an issue open about it. We agreed that the distribution name and the package name (the import call) should be cyipopt. I am unlikely to have time in a week to do that and I don't have merge rights on the repo.

…cy handling

…PLAY on CI

bocklund · 2017-11-13T10:20:34Z

Just waiting on cyipopt for Windows. Otherwise I think this is ready to merge.

… API

This really big commit does a bunch of things in the core: - Adds the NLP solver IPOPT and its Cython wrapper cyipopt as dependencies - Removes the majority of the old eqsolver core, while leaving the simplistic treatment of global minimization intact (for now) - Adds deepcopy support for CompositionSet - Creates new `Problem` and `Solver` abstractions, which will make it easier to extend the solver in the future to handle different types of problems and constraints - Tweaks minimum value constants to improve convergence - Make some equilibrium tests more restrictive * WIP: Initial memory cleanup work * WIP: Profiling and memory and speed perf attempts * WIP: eqsolver constraints and add_new_phases: perf improvements by profiling * WIP: eqsolver: Remove debugging code * WIP: nogil in eval_energy * WIP: eval_energy nogil * WIP: _eval_disordered_energy and _eval_energy_gradient nogil * WIP: revert _eval_energy_gradient nogil * WIP: _eval_energy_gradient nogil fixed with calloc * WIP: eval_energy_gradient nogil * WIP: eval_energy_hessian and update nogil * WIP: eqsolver: type declaration cleanup * ENH: eqsolver: Add more line search iterations for performance * WIP: eval_energy: change to pointer for out argument * WIP: eval_energy_gradient and eval_energy fully release gil * WIP: eval_energy_hessian fully release gil * FIX: eqsolver: Add guard for singular matrices during solve * FIX: eqsolver: Convergence at lower point density by fixing bugs in add-remove-phases step of computation * FIX: eqsolver: Performance and stability improvements * WIP: Snapshot with all but one test passing * REF: Create Solver class and abstract away some of the details from the main equilibrium solve loop * ENH: CompositionSet: Add GM to CompositionSet.__str__ * FIX: hyperplane: Memory corruption during phase fraction calculation * FIX: eqsolver: Inf norm calculation * FIX/WIP: solver: Tweak complementarity tol to get convergence on one test, but break test_rose_nine. * TST/FIX: Add cyipopt to dependencies * FIX: Remove NumPyPrinter due to sympy 1.1 incompatibility. Other sympy 1.1 import changes. Fixes gh-105. * MAINT: eqsolver: Remove dead code * FIX: add_new_phases: Misalignment of phase fractions array due to non-deterministic sorting of simplex vertex index array * WIP: add_new_phases: Force endmembers to always be candidates for inclusion * WIP: Continued attempts at global minimization improvement. * WIP: add_new_phases: Add all stable phases plus the phase with the largest driving force. * ENH: CompositionSet: deepcopy support * WIP: eqsolver: Clean up and simplify global minimization code. Doesn't yet work completely but many tests pass and the performance hit is improved. * FIX/WIP: Get tests passing at pdens=500 with ipopt. Use a two-step approach to improve ipopt convergence. * WIP: eqsolver: For pdens=50, pass all tests with solver tweaks for cases near boundary. * DOC/FIX: BinaryExamples: Bump results and fix error in energy plot. * FIX: equilibrium: Reset default pdens to 500 to address ternary miscibility gap detection * BLD/FIX: appveyor: Bump miniconda to fix broken conda * MAINT: core: Remove obsolete cymem-related code * MAINT: compiled_model: Remove cython debugging directives * MAINT: hyperplane: Remove extraneous print statements * MAINT: Problem: Remove extraneous comments on hessian implementation * MAINT: solver: Remove extraneous commented-out print statements * TST/MAINT: test_equilibrium: Clean up assertions * BLD/TST: travis: Drop py34 and disable Dockerization to ease dependency handling * MAINT: core: Refactor solver constants to be in core.constants * FIX/TST: Use matplotlib 'Agg' backend in tests to avoid issues no DISPLAY on CI * BLD: appveyor: Add msys2 and pycalphad channels to enable ipopt * BLD/WIP: appveyor: Fix conda channel add * FIX: equilibrium: dask module rename compat * FIX: calculate: Remove fast_concat due to reliance on xarray internal API * FIX: custom_autowrap: Import correctly on Windows * FIX: solver: Increase ipopt max_iter to fix Windows * BLD: appveyor: Bump py27 to 64-bit

…rties (pycalphad#124) * xfail a test for dataset checking species * Test for non-equilibrium thermochemical data * Fix for searching for non-equilibrium thermochemical data * Move update_phase_records * Add WIP test for thermochemical error * WIP: approximate equilibrium is working * Fix phase records * Set up weights * Add test for equilibrium thermochemical data with species * Fix bug for multiple potentials * Add test for property outside of standard properties * Raise a value error if trying to calculate a per-phase property * Fix propagating changed parameters through * EmceeOptimizer should now use the equilibrium thermochemical error * Fix for case where there's no equilibrium thermochemical data * Basic dataset checking for equilibrium thermochemical datasets * Fix for symbols in _eqcalculate that were converted to SymPy Symbols * Try pinning cyipopt version to a valid build * Check that a dataset is one of equilibrium, zpf, or activity. * Support data_weights in equilibrium thermochemical error * Cleanup * Bail out early for equilibrium thermochemical data if infinity or calculation error * Revert "Try pinning cyipopt version to a valid build" This reverts commit 497b18b70655643b4eb9896a290c5a46e9207a9d. * Debug commit for CI * pip install with no deps * pin ipopt for correctness issue * Revert "Debug commit for CI" This reverts commit f4fd55228223075661fb590901335f03118edec7. * Cleanup unused packages and dead comments. * Update build_eqpropdata types and docstring * Update get_equilibrium_thermochemical_data docstring and types * Docstring and types calc_prop_differences * Update docstring/types and remove error kwarg from calculate_equilibrium_thermochemical_probability

richardotis and others added 30 commits May 19, 2017 18:27

WIP: Initial memory cleanup work

922d137

WIP: Profiling and memory and speed perf attempts

813ba0b

WIP: eqsolver constraints and add_new_phases: perf improvements by pr…

4dffa3c

…ofiling

WIP: eqsolver: Remove debugging code

7c0d71a

WIP: nogil in eval_energy

d1cfc5b

WIP: eval_energy nogil

ea0db25

WIP: _eval_disordered_energy and _eval_energy_gradient nogil

962b3b0

WIP: revert _eval_energy_gradient nogil

40df868

WIP: _eval_energy_gradient nogil fixed with calloc

bf89e78

WIP: eval_energy_gradient nogil

ec3264c

WIP: eval_energy_hessian and update nogil

7d45404

WIP: eqsolver: type declaration cleanup

b046204

ENH: eqsolver: Add more line search iterations for performance

599880c

WIP: eval_energy: change to pointer for out argument

43dcecc

WIP: eval_energy_gradient and eval_energy fully release gil

955baef

WIP: eval_energy_hessian fully release gil

5a995f7

FIX: eqsolver: Add guard for singular matrices during solve

b174eee

FIX: eqsolver: Convergence at lower point density by fixing bugs in a…

be80820

…dd-remove-phases step of computation

FIX: eqsolver: Performance and stability improvements

580ef69

WIP: eqsolver: Incomplete attempt at getting Hillert's two-step metho…

563e407

…d reimplemented

ENH/WIP: eqsolver: Add constant-potential refinement to add_new_phase…

38ef883

…s. Tweak convergence criteria

Merge branch 'fix_memory' of github.com:pycalphad/pycalphad into fix_…

9de0ced

…memory Conflicts: pycalphad/core/eqsolver.pyx

FIX: eqsolver: Snapshot with tests passing

e56fd2f

WIP: Snapshot with passing tests and two conv failures on Al-Zn

f5910a5

WIP: Snapshot with good Al-Zn but many single-phase failures

02c0e2a

WIP: Snapshot with no wrong answers but lots of conv failures. Tests …

72dba20

…were not run

WIP: Snapshot of really slow but seemingly robust solver. Having trou…

e9e4ffc

…ble getting tests to finish.

WIP: eqsolver: Add initial remove_degenerate_phases() back

62470e8

WIP: Snapshot with all but one test passing

7a8504c

WIP: py27 compat for add_new_phases

c56d5ac

BLD/FIX: appveyor: Bump miniconda to fix broken conda

4ef8a7a

bocklund requested changes Nov 8, 2017

View reviewed changes

richardotis and others added 11 commits November 12, 2017 16:36

WIP: Merge latest develop

4790d65

Merge branch 'develop' into fix_memory

147094d

MAINT: core: Remove obsolete cymem-related code

16ccba3

MAINT: compiled_model: Remove cython debugging directives

d21a36b

MAINT: hyperplane: Remove extraneous print statements

2169906

MAINT: Problem: Remove extraneous comments on hessian implementation

eea3d1c

MAINT: solver: Remove extraneous commented-out print statements

7fe6f42

TST/MAINT: test_equilibrium: Clean up assertions

eeeb854

BLD/TST: travis: Drop py34 and disable Dockerization to ease dependen…

a861daf

…cy handling

MAINT: core: Refactor solver constants to be in core.constants

bde216c

FIX/TST: Use matplotlib 'Agg' backend in tests to avoid issues no DIS…

fe7e0db

…PLAY on CI

bocklund force-pushed the fix_memory branch from 89f9984 to fe7e0db Compare November 13, 2017 09:48

bocklund approved these changes Nov 13, 2017

View reviewed changes

richardotis and others added 8 commits November 24, 2017 11:57

BLD: appveyor: Add msys2 and pycalphad channels to enable ipopt

eae81ff

Merge branch 'develop' into fix_memory

5c3a778

BLD/WIP: appveyor: Fix conda channel add

1ae47d5

FIX: equilibrium: dask module rename compat

f1f868e

FIX: calculate: Remove fast_concat due to reliance on xarray internal…

12f485a

… API

FIX: custom_autowrap: Import correctly on Windows

83d1e1c

FIX: solver: Increase ipopt max_iter to fix Windows

533b580

BLD: appveyor: Bump py27 to 64-bit

ee5cda8

richardotis merged commit 40973ee into develop Nov 25, 2017

bocklund deleted the fix_memory branch November 30, 2017 22:48

richardotis added this to Waiting in Roadmap Feb 26, 2018

richardotis removed this from Waiting in Roadmap Feb 26, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Solver improvements via IPOPT #124

Solver improvements via IPOPT #124

richardotis commented Sep 14, 2017 •

edited

bocklund left a comment

bocklund Nov 4, 2017

bocklund Nov 5, 2017

bocklund Nov 5, 2017

bocklund Nov 8, 2017

bocklund Nov 5, 2017

bocklund Nov 5, 2017

bocklund Nov 5, 2017

bocklund Nov 8, 2017

bocklund Nov 8, 2017

bocklund Nov 8, 2017

richardotis Nov 13, 2017

bocklund Nov 8, 2017

bocklund Nov 8, 2017

bocklund Nov 8, 2017

richardotis Nov 13, 2017

moorepants Nov 13, 2017

richardotis Nov 13, 2017

moorepants Nov 13, 2017

bocklund commented Nov 13, 2017

		@@ -46,13 +46,13 @@ cdef public class CompiledModel(object)[type CompiledModelType, object CompiledM
		double *eval_row, double[:] parameters) nogil

Solver improvements via IPOPT #124

Solver improvements via IPOPT #124

Conversation

richardotis commented Sep 14, 2017 • edited

bocklund left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bocklund commented Nov 13, 2017

richardotis commented Sep 14, 2017 •

edited