Add MoreThuente Linesearcher

[vladimir-ch]

I like the names DecreaseFactor and CurvatureFactor. I would vote for using them also in the other Linesearchers.

As I mentioned before, CG is the method that most benefits from this Linesearcher, so I set it as the default. Maybe we should remake the CG tests, there are already many combinations and the Linesearchers add an extra dimension.

PTAL @btracey

[btracey]

Shouldn't these be in functions? Even if specialized I think we/someone will find it useful to have an easily accessible set of functions. We could provide a one-D wrapper here or something.

[vladimir-ch]

Indeed. I will move them there.

[vladimir-ch]

Done. PTAL @btracey

[btracey]

How about this is "ErrLinesearchBound". Some linesearchers have a minimum step, and this may as well cover both?

[vladimir-ch]

Changed to ErrLinesearcherBound. At the moment I will leave it as an ordinary error but I can imagine that storing the step and the bounds could be potentially useful.

[vladimir-ch]

Addressed comments. I also added a check that Linesearcher.Init returns an evaluating operation and changed the invalid operation panic to a regular t.Errorf. Sorry for changing the code outside of the comments but it seemed merited to me. PTAL @btracey

[btracey]

characteristics such as?

[vladimir-ch]

It's hard for me to describe it in one sentence. Please take a look at what I have there now.

[btracey]

Sorry, I do understand why Max and Min steps are absolute, but I don't understand why StepTolerance should be relative, especially when the other Step numbers are all absolute.

[btracey]

Since this only has two values, should it be a boolean?

[vladimir-ch]

In fact it has three values because I (ab)use the zero value of stage to check that Init has been called. If you don't like it or if such stringent checking is deemed too pedantic, I can make it a bool.

[btracey]

No, that's fine.

[btracey]

I think absolute is easier to think about. There's no clear definition of "relative" here (though I know you provided one). Additionally, the tolerance, even absolutely, is already scaled no ||dir||. Our "coordinates" have a step of size ||dir|| equal to 1.

[vladimir-ch]

Additionally, the tolerance, even absolutely, is already scaled no ||dir||. Our "coordinates" have a step of size ||dir|| equal to 1.

Yes, that's why I offered the alternative to measure the interval width in real, non-scaled-on-dir units. However, for that we would have to change the Linesearcher interface and that does not feel worthy. I don't have a strong argument for a relative tolerance, so I will change it to an absolute tolerance and see what it does with the tests.

[btracey]

Same here.

[btracey]

LGTM. Thanks.

[vladimir-ch]

The discussion about the relative step tolerance has been left open-ended but I will merge anyway. I will open an issue and copy the discussion there so that it is together in one place.