[line-search] implement Armijo line search algorithm #925

HenKlei · 2020-05-18T15:46:35Z

When solving nonlinear problems for several different parameters, finding a proper value for the relax parameter of the Newton solver that works for every parameter is quite difficult. Thus, I implemented a (very) simple line search algorithm that, at least in my application, led to convergence of the Newton method for every parameter.

To check the original Armijo-Goldstein condition, one needs the gradient of the objective function. Thus, if no gradient is provided, a weaker version of this condition is checked; namely that the function value is decreasing, no matter to what extend.

To be able to compute the gradient of the error function in the Newton method, we switched from error_norm to error_product in the Newton. Thus, we can derive the gradient for any error_product. Certainly, this restricts us to the Hilbert space case; non-Hilbert norms are no longer supported.

Furthermore, we had to make some adjustments in the tests of the Newton method; the Jacobian of the MonomOperators was not implemented properly. We added a test case for the line search algorithm, in which the Newton without line search starts oscillating between two values.

codecov · 2020-05-18T17:00:19Z

Codecov Report

Merging #925 into master will increase coverage by 0.16%.
The diff coverage is 95.45%.

Impacted Files	Coverage Δ
src/pymor/algorithms/newton.py	`90.90% <94.44%> (+4.37%)`	⬆️
src/pymor/algorithms/line_search.py	`96.15% <96.15%> (ø)`
src/pymor/operators/interface.py	`85.91% <0.00%> (+0.70%)`	⬆️
src/pymor/operators/constructions.py	`84.28% <0.00%> (+0.95%)`	⬆️
src/pymor/vectorarrays/numpy.py	`84.55% <0.00%> (+0.98%)`	⬆️
src/pymor/tools/formatrepr.py	`87.80% <0.00%> (+1.21%)`	⬆️

src/pymor/algorithms/line_search.py

src/pymor/algorithms/newton.py

TiKeil · 2020-05-19T13:39:31Z

Hi Hendrick,

this already looks very good. Could you add a reference for the Armijo-Goldstein line search?

Also, I would be very happy if you could change the names of the arguments in the armijo rule to a more general setting. Please have a look at the requested changes.

Tim

HenKlei · 2020-05-19T13:41:59Z

Hi Tim, thank you for your suggestions! I will have a look.

sdrave

Hi @HenKlei, thanks a lot for this PR. This is looking very good already .. - You should add an entry to AUTHORS.md ..

src/pymor/algorithms/line_search.py

src/pymor/algorithms/newton.py

sdrave

Apart from minor issues, I think this can be merged soon. @HenKlei, you should still add an entry to AUTHORS.md ...

src/pymor/algorithms/line_search.py

src/pymor/algorithms/newton.py

sdrave

This is all looking very nice now. Only one thing remains: there should be some form of test for the new code. There is already something very basic in pymortests.algorithms.stuff. Could you add something there?

HenKlei · 2020-05-27T12:58:58Z

From my point of view, there isn't anything missing and this is ready to get merged.

@TiKeil, @sdrave Any objections or further comments?

HenKlei · 2020-05-27T18:37:20Z

This is all looking very nice now. Only one thing remains: there should be some form of test for the new code. There is already something very basic in pymortests.algorithms.stuff. Could you add something there?

I have an idea for a test, I will work on that.

HenKlei · 2020-05-28T13:38:57Z

I added a small test in de2747b, but actually, I am a bit astonished about the behavior of the Newton algorithm in this case. Using 0.5 as initial value does not work without using the line search method. Certainly, as far as I can see, the algorithm should converge even when using 0.5 as initial guess. Maybe, there's something wrong with the monomials, or I didn't get the idea of the tests in stuff.py at all?

sdrave · 2020-05-29T12:25:46Z

@HenKlei, turns out the implementation of MonomOperator.jacobian is wrong. The right implementation would be:

    def jacobian(self, U, mu=None):
        return NumpyMatrixOperator(self.derivative(U.to_numpy()).reshape((1,1)))

Can you fix it in this PR?

HenKlei · 2020-05-29T13:07:21Z

Alright, I'm going to fix this and have another look at the tests. Thanks for the clue.

HenKlei · 2020-05-29T19:49:31Z

@sdrave After applying the check you suggested, the test is not passing. This makes completely sense when looking at the problem: for instance the monom of degree 2, i.e. x^2, reaches the desired absolute tolerance of 1e-15 already for values like x=1e-8. Hence, comparing this value of x to 0 will lead to a failing test, although the correct root is approximated. I think, this test is maybe not the best for the Newton algorithm at all, at least in the way it is currently performed. Do you have any suggestions on how to proceed?

sdrave · 2020-06-02T16:11:37Z

Hence, comparing this value of x to 0 will lead to a failing test, although the correct root is approximated. I think, this test is maybe not the best for the Newton algorithm at all, at least in the way it is currently performed.

Not at all ...

Do you have any suggestions on how to proceed?

Maybe the easiest way would be to check whether the solution actually has a residual smaller than atol?

sdrave · 2020-06-03T13:37:23Z

@HenKlei, I see you have used float_cmp, which is also fine. Shall we merge this now?

HenKlei · 2020-06-04T06:34:21Z

Now, the test should work as expected.

@sdrave Maybe you could have another look. From my point of view, this can be merged.

HenKlei · 2020-06-04T07:32:27Z

The test_lincomb_op is broken due to the changes on the MonomOperator. I will have another look.

HenKlei · 2020-06-05T08:26:34Z

I had another look at test_lincomb_op in pymortests/operators.py. This is the first part of the method:

def test_lincomb_op():
    p1 = MonomOperator(1)
    p2 = MonomOperator(2)
    p12 = p1 + p2
    p0 = p1 - p1
    x = np.linspace(-1., 1., num=3)
    vx = p1.source.make_array((x[:, np.newaxis]))
    assert np.allclose(p0.apply(vx).to_numpy(), [0.])
    assert np.allclose(p12.apply(vx).to_numpy(), (x * x + x)[:, np.newaxis])
    assert np.allclose((p1 * 2.).apply(vx).to_numpy(), (x * 2.)[:, np.newaxis])
    assert almost_equal(p2.jacobian(vx).apply(vx), p1.apply(vx) * 2.).all()

To me, the last line does not really make sense. I think, p2.jacobian(vx).apply(vx) should be some kind of list of directional derivatives at the points given by vx in the directions given by vx. On the other hand, p1.apply(vx) * 2. should be a list of directional derivatives at the points given by vx in direction 1 in each component. I think p2.jacobian(vx).apply(vx) should be replaced by something like p2.jacobian(vx).apply(ones), where ones is a vector containing 1 in each entry.

Nevertheless, I did not get the test to work again so far. The dimensions of the vectors and operators do not seem to fit. @sdrave Are you sure that the reshape in return NumpyMatrixOperator(self.derivative(U.to_numpy()).reshape((1,1))) is correct, especially when the Jacobian should be used in test_lincomb_op in the way it is at the moment?

sdrave · 2020-06-05T09:34:26Z

Nevertheless, I did not get the test to work again so far. The dimensions of the vectors and operators do not seem to fit. @sdrave Are you sure that the reshape in return NumpyMatrixOperator(self.derivative(U.to_numpy()).reshape((1,1))) is correct, especially when the Jacobian should be used in test_lincomb_op in the way it is at the moment?

jacobian should only get a VectorArray of length 1. You cannot get more than one Jacobian with a single call. So the test is wrong, and an assertion in MonomOperator.jacobian might be helpful as well.

HenKlei · 2020-06-05T09:38:30Z

jacobian should only get a VectorArray of length 1. You cannot get more than one Jacobian with a single call. So the test is wrong, and an assertion in MonomOperator.jacobian might be helpful as well.

Alright, then I should be able to fix that. Thank you!

HenKlei · 2020-06-06T12:29:53Z

@sdrave The tests pass. Just two small questions:
Should we rename stuff.py to newton.py? Do you have any improvements or simplifications for test_lincomb_op?

sdrave · 2020-06-08T07:31:53Z

@HenKlei, thanks for all the additional work!

Should we rename stuff.py to newton.py?

Probably. Although I would do it in a separate PR.

Do you have any improvements or simplifications for test_lincomb_op?

I think this is not our nicest test, but for now I would leave it as is. @renefritze, is currently changing a lot in the test suite and I would wait a bit until things have settled down.

I would say, let's merge this PR.

HenKlei · 2020-06-09T06:54:43Z

Alright, sounds good!

TiKeil reviewed May 19, 2020

View reviewed changes

sdrave reviewed May 19, 2020

View reviewed changes

sdrave requested changes May 26, 2020

View reviewed changes

sdrave added this to the 2020.1 milestone May 26, 2020

sdrave reviewed May 27, 2020

View reviewed changes

sdrave approved these changes May 27, 2020

View reviewed changes

HenKlei mentioned this pull request May 28, 2020

Residual norm of exactly 0 in Newton throws error #927

Closed

HenKlei and others added 9 commits June 3, 2020 09:45

[line-search] implement Armijo line search algorithm

b9c45e1

[line-search] rename some variables and error types

4fc4118

[line-search] added reference for the Armijo condition

5b334da

[line-search] log usage of line search

5f3c6e3

[line-search] allow 'armijo' as relax parameter, add line search params

3bd9353

[line-search] use error-product instead of error-norm

e6fa0b0

[AUTHORS] updated authors

55e38b1

[line-search] fixed some minor issues

b1415e5

[line-search] added small test

de5e7d9

HenKlei force-pushed the line-search branch from de2747b to de5e7d9 Compare June 3, 2020 07:49

[pymortests] fixed test for Newton method

4735a41

HenKlei added 2 commits June 6, 2020 09:35

[pymortests] fixed test for lincomb of monom operators

beb71a1

[pymortests] added test for Newton with unknown line search method

afce012

sdrave merged commit 9ad6fef into pymor:master Jun 9, 2020

HenKlei deleted the line-search branch August 31, 2020 09:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[line-search] implement Armijo line search algorithm #925

[line-search] implement Armijo line search algorithm #925

HenKlei commented May 18, 2020 •

edited

Loading

codecov bot commented May 18, 2020 •

edited

Loading

TiKeil commented May 19, 2020

HenKlei commented May 19, 2020

sdrave left a comment

sdrave left a comment

sdrave left a comment

HenKlei commented May 27, 2020

HenKlei commented May 27, 2020

HenKlei commented May 28, 2020

sdrave commented May 29, 2020

HenKlei commented May 29, 2020

HenKlei commented May 29, 2020

sdrave commented Jun 2, 2020

sdrave commented Jun 3, 2020

HenKlei commented Jun 4, 2020

HenKlei commented Jun 4, 2020

HenKlei commented Jun 5, 2020

sdrave commented Jun 5, 2020

HenKlei commented Jun 5, 2020

HenKlei commented Jun 6, 2020

sdrave commented Jun 8, 2020

HenKlei commented Jun 9, 2020

[line-search] implement Armijo line search algorithm #925

[line-search] implement Armijo line search algorithm #925

Conversation

HenKlei commented May 18, 2020 • edited Loading

codecov bot commented May 18, 2020 • edited Loading

Codecov Report

TiKeil commented May 19, 2020

HenKlei commented May 19, 2020

sdrave left a comment

Choose a reason for hiding this comment

sdrave left a comment

Choose a reason for hiding this comment

sdrave left a comment

Choose a reason for hiding this comment

HenKlei commented May 27, 2020

HenKlei commented May 27, 2020

HenKlei commented May 28, 2020

sdrave commented May 29, 2020

HenKlei commented May 29, 2020

HenKlei commented May 29, 2020

sdrave commented Jun 2, 2020

sdrave commented Jun 3, 2020

HenKlei commented Jun 4, 2020

HenKlei commented Jun 4, 2020

HenKlei commented Jun 5, 2020

sdrave commented Jun 5, 2020

HenKlei commented Jun 5, 2020

HenKlei commented Jun 6, 2020

sdrave commented Jun 8, 2020

HenKlei commented Jun 9, 2020

HenKlei commented May 18, 2020 •

edited

Loading

codecov bot commented May 18, 2020 •

edited

Loading