MAINT: Refactor linear programming solvers #9145

Kai-Striega · 2018-08-15T04:04:20Z

Early implementation of changes discussed in #8942 and #9076. It's far from complete but I think it's worth discussing the changes so far.
Major changes:

Move functions related to pre and post processing into _linrpog.py
Remove redundant code in _linrpog_simplex(…).
Update tests to account for pre-solve.
Move simplex solver into own module

The effects on the current issues:

Linprog(method='simplex') produces incorrect starting basis. #9008 will no longer be relevant and can be closed.
scipy.optimize.linprog _sometimes_ ignores restrictions #7237 now raises an error as expected.
linprog returns solution that violates inequality constraint #5400 and linprog failure on provided input #8174 still raise warnings rather than errors.

For 5400 and 8174 the simplex method pivots with a tolerance value very close to zero. The solution then becomes (wrongly) infeasible with a status of 0 (success). A warning is raised during the simplex method if this occurred. An error should be raised instead. Post-process catches this but returns a status of 4 and an error message instead of an error, should this be changed to a ValueError?

There are more less important changes, most of which are added as comments:

References / docstrings need to be updated
How do we handle input for presolve
How to test for callback functions with the interior-point
etc..

@mdhaber again this is far from complete, but how does this look so far?

Move ``_clean_inputs(...)``, ``_presolve(...)``, ``_get_Abc(...)`` and ``_post_process(...)`` from module ``_linprog_ip`` into ``_linprog``. Remove duplicate behavior from ``_linprog_simplex(...)`` and modify return values to match those required in pre and postsolve functions.

mdhaber · 2018-08-15T15:31:34Z

Excited about this! I'll try to look today.

mdhaber

Cool! Tough to follow because of how GitHub detects additions/deletions but I think it all makes sense and doesn't look too dangerous. I wrote a bunch of comments about what I think was changed and why. These are just to confirm that I checked the changed and tried to make sense of them, but if any are wrong, would you let me know?

I didn't check the test changes very carefully. Anything substantial, besides relaxing some of the tests that previously were not applied to simplex?

Only thing I saw that I was a little concerned about was poping options. I don't understand why get isn't sufficient.

mdhaber · 2018-08-16T12:04:27Z

scipy/optimize/_linprog.py

+    tol = options.get('tol', default_tol)
+
+    # This is an undocumented option for unit testing sparse presolve
+    _sparse_presolve = options.pop('_sparse_presolve', False)


Why pop it (instead of get it)?

_sparse_presolve applies only to this part of the program (changing A to sparse for testing). Previously this was done in _linprog_ip(...) with _sparse_presolve passed as a (undocumented) option. The conversion now occurs in _linprog(...). _linprog_ip(...) no longer requires _sparse_presolve to be passed as an option, and has been removed. If options still contains _sparse_presolve it will be passed to _check_unknown_options(...), not be recognized as an option (as it no longer is) and raise an error.

mdhaber · 2018-08-16T12:06:57Z

scipy/optimize/_linprog.py

+
+    # Solve trivial problem, eliminate variables, tighten bounds, etc...
+    c0 = 0  # we might get a constant term in the objective
+    if options.pop('presolve', True):


Again why are we modifying options? This is a reference to the dictionary passed in by the user, not our own local copy, right? So this would modify the user's object, which we wouldn't want to do.

Same reasoning as for _sparse_presolve.

I considered making a local copy of options, however the user's dictionary would not be used anywhere else. Unless you think keeping it as a reference is worth doing I don't see a benefit to keeping a duplicate copy.

mdhaber · 2018-08-16T12:07:16Z

scipy/optimize/_linprog.py

+
+    # Solve trivial problem, eliminate variables, tighten bounds, etc...
+    c0 = 0  # we might get a constant term in the objective
+    if options.pop('presolve', True):


Again why are we modifying options here and anywhere else rather than just reading it? This is a reference to the dictionary passed in by the user, not our own local copy, right? So this would modify the user's object, which we wouldn't want to do.

mdhaber · 2018-08-16T12:11:18Z

scipy/optimize/_linprog_ip.py

@@ -1754,11 +759,9 @@ def eta(g=gamma):

 def _linprog_ip(
        c,
-        A_ub=None,


These deletions make sense because the problem is already in standard form.

mdhaber · 2018-08-16T12:11:57Z

scipy/optimize/_linprog_ip.py

-        A_eq=None,
-        b_eq=None,
-        bounds=None,
+        c0=0,


Makes sense because presolve might have added constant to objective - which is just needed in here for display, right?

Correct, c0 will only be used for display.

mdhaber · 2018-08-16T12:51:01Z

scipy/optimize/_linprog_simplex.py

@@ -0,0 +1,496 @@
+"""


_linprog_simplex, _solve_simplex, _apply_pivot, _pivot_row, and _pivot_col simply moved from _linprog.py

mdhaber · 2018-08-16T12:53:16Z

scipy/optimize/tests/test_linprog.py

-                sup.filter(OptimizeWarning, "Solving system with option...")
-                res = linprog(c=cost, A_eq=A_eq, b_eq=b_eq, bounds=bounds,
-                              method=self.method, options=self.options)
+                            method=self.method, options=self.options)


I think the indentation of this line is off? Should match with ( on previous line.

I'll fix it in the next commit

mdhaber · 2018-08-16T12:56:12Z

scipy/optimize/tests/test_linprog.py

@@ -874,7 +885,7 @@ def cb(xk, **kwargs):
        res = linprog(c, A_ub=A_ub, b_ub=b_ub, callback=cb, method=self.method)

        assert_(callback_complete[0])
-        assert_allclose(last_xk[0], res.x)
+        assert_allclose(last_xk[0][:res.x.size], res.x)


Was there a bug in this test before?

The simplex method sees the artificial variables / variable substitutions from pre processing as "true" variables expecting them to appear in the solution. The variables are instead removed during _postprocess and therefore do not appear.

mdhaber · 2018-08-16T12:57:02Z

scipy/optimize/tests/test_linprog.py

        res = linprog(
            c, A_ub, b_ub, A_eq, b_eq, method=self.method,
-            bounds=bounds, options={'tol': low_tol}
+            bounds=bounds, options=self.options


Not digging into this - can you explain why this changed?

mdhaber · 2018-08-16T12:58:48Z

scipy/optimize/tests/test_linprog.py

-        if self.method == "simplex":
-            # Including the callback here ensures the solution can be
-            # calculated correctly.
+


Cool that several of these tests pass with simple now.

mdhaber · 2018-08-16T13:06:32Z

CI failure is due to a doctest in _linprog.py line 1322. Just that the number of iterations is different. Any idea why that would have changed?

AppVeyor finding some assertion errors in simplex. Not sure what's going on there.

mdhaber · 2018-08-16T13:21:07Z

Maybe also check whether test_issue_6139 and most of BaseTestLinprogIP can be moved into the common tests?

Kai-Striega · 2018-08-16T14:07:53Z

I think the number of iterations are different because of using presolve, at least that seems logical.

The relevant output from AppVoyeur:

    assert_(isinstance(basis, np.ndarray))
> assert_(basis.dtype == np.int_)
E  AssertionError
    basis      = array([5, 6, 7], dtype=int64)

on my system:

>>> import numpy as np

>>> a = np.array([5, 6, 7], dtype=np.int64)
>>> a.dtype == np.int_
True

So it shouldn't really be happening. Maybe it has to do with only being on python 2.7?

ilayn

I've tried to make it at least halfway. I would really recommend checking out previous code to stay coherent with the remaining parts of the library.

And mostly for the logical comparisons we can be a bit more concise.

ilayn · 2018-08-16T13:50:58Z

scipy/optimize/_linprog.py

 from warnings import warn
 from .optimize import OptimizeResult, OptimizeWarning, _check_unknown_options
+from scipy.optimize._remove_redundancy import _remove_redundancy


You might want to group these imports with ( on the first one and ) on the last over multiple lines.

ilayn · 2018-08-16T13:51:09Z

scipy/optimize/_linprog.py

-        regardless of magnitude).
+    c : array_like
+        Coefficients of the linear objective function to be minimized.
+    A_ub : array_like, optional


This can simply use the [...] such that A_ub @ x gives the upper bound constraints or something similar. Also - is not required for dimension notation to be consistent with the rest.

This was simply copied from _linporg_ip. The description has NOT been consistent between additions. Standardising the descriptions will need to be done at some point, preferably before anything new is added.

ilayn · 2018-08-16T13:51:14Z

scipy/optimize/_linprog.py

+        1-D array of values representing the RHS of each equality constraint
+        (row) in ``A_eq``.
+    bounds : sequence, optional
+        ``(min, max)`` pairs for each element in ``x``, defining


Does it have to be tuple? list of tuples work? Also the explanation might use some explicit example :

[...] [(a1, b1), (a2, b2) ...] defining upper bounds on the parameter. By default all as are set to 0 and all bs to None hence modeling the interval [0, inf]

I agree these SHOULD be a different form. This is how they were implemented as standard @mdhaber and myself have already noted that it should be changed, but in a separate PR.

ilayn · 2018-08-16T13:51:18Z

scipy/optimize/_linprog.py

-
+    c : 1-D array
+        Coefficients of the linear objective function to be minimized.
+    A_ub : 2-D array


Same as above with dimension dashes and A_ub@x. Why do you explain the outputs if they are going to be checked anyways.

ilayn · 2018-08-16T13:51:22Z

scipy/optimize/_linprog.py

-    bland : bool
-        If True, use Bland's rule for selection of the row (if more than one
-        row can be used, choose the one with the lowest variable index).
+    try:


This looks pretty nonstandard array checking. You can already use the misc.utils functions for checking valid input arraylikes. You don't want to chase exception types but trust on np.arrayability. Those functions will complain anyways.

Didn't know about those when I wrote it. It's quite thorough and works very well, so I hope we don't have to overhaul this right now. See history of such comments in #7123.

ilayn · 2018-08-16T14:22:15Z

scipy/optimize/_linprog.py

+    ub_newub = ubs[ub_some]
+    n_bounds = np.count_nonzero(ub_some)
+    A_ub = vstack((A_ub, zeros((n_bounds, A_ub.shape[1]))))
+    b_ub = np.concatenate((b_ub, np.zeros(n_bounds)))


hstack maybe ? Since you have vstack elsewhere?

ilayn · 2018-08-16T14:24:53Z

scipy/optimize/_linprog.py

+
+    # add slack variables
+    A2 = vstack([eye(A_ub.shape[0]), zeros((A_eq.shape[0], A_ub.shape[0]))])
+    A = hstack([A1, A2])


ah you have hstack afterall

ilayn · 2018-08-16T14:25:17Z

scipy/optimize/_linprog.py

-    where:  -inf <= x[0] <= inf
+    Parameters
+    ----------
+    x : 1-D array


Same comments as above.

ilayn · 2018-08-16T14:25:42Z

scipy/optimize/_linprog.py

+             1 : Iteration limit reached
+             2 : Problem appears to be infeasible
+             3 : Problem appears to be unbounded
+             4 : Serious numerical difficulties encountered


What is not "serious" in terms of difficulties?

Difficulties that have been resolved using a more robust, albeit less efficient, solver.

ilayn · 2018-08-16T14:26:15Z

scipy/optimize/_linprog.py

+        optimal objective value for original problem
+    slack: 1-D array
+        The (non-negative) slack in the upper bound constraints, that is,
+        ``b_ub - A_ub * x``


You know I just learned about that a few weeks ago?

mdhaber · 2018-08-16T15:02:13Z

@ilayn Thanks for all these checks but can we wait for a separate PR to fix existing issues in the code? Most of the stuff that shows up in green is not new; it is just moved from elsewhere.

Kai-Striega · 2018-08-16T15:11:30Z

@ilayn Thanks for having such a thorough look through it! I'm going to respond to most of the reviews within a single comment. I have not written most of the code. The PR moves existing code from _linporg_ip.py into _linprog.py and creates a new module for the simplex method in an attempt to remove code duplication between simplex and interior-point solvers. The existing presolve code has already been discussed elsewhere (see @mdhaber's comment on #9076) and will be addressed once the two have been merged.

ilayn · 2018-08-16T20:05:01Z

Ah I was going through only the green lines and didn't pay too much attention to the existing code. In that case indeed let's keep it as only the refactoring.

But we at least need the stacklevel arguments while we are at it. They turn out to be quite important for CI blocking issues.

Kai-Striega · 2018-08-18T03:48:52Z

Based on the discussion with @mdhaber and @ilayn, here is what I think we should be looking at for merging the solvers:

Modifying `options`:

The user may provide an options storing any additional solver options. My change modifies options by using pop to access presolve and _sparse_presolve.

These parameters previously only applied to the interior-point method. Now presolving occurs in the top-level linear programming module. Causing me to remove them from _linprog_ip. presolve and _sparse_presolve are now unknown options and will cause _check_unknown_options(...) to fail.

I agree changing the user provided options is not the best way to go. Further the docs define options to be:

options : dict, optional

    A dictionary of solver options. All methods accept the following generic options:

        maxiter : int

            Maximum number of iterations to perform.
        disp : bool

            Set to True to print convergence messages.

    For method-specific options, see show_options('linprog').

Which is now no longer correct. Maybe chaning presolve and _sparse_presolve to keyword arguments will provide a more accurate approach?

Tests:

The interior-point solver does not implement callback functions. This is currently tested for. However linprog should only raise this error if the interior-point method is actually used i.e. ``method='interior-point' and the problem is not solved in presolve.
Check if more tests from BaseTestLinprogIP and issue 6139 and can be moved into common tests.

Documentation

Edit parameter descriptions to be consistent between different functions.
Move docstrings describing presolve from _linprog_ip into linprog
Change reference numbers to be consistent between ALL linear programming modules (maybe?)

mdhaber · 2018-08-18T04:10:12Z

@Kai-Striega more to come, but why don't we just change it so that the options are not unknown? We should also add presolve to the documentation. They should be options and not keyword arguments.

mdhaber · 2018-08-18T04:13:56Z

Also the thoughts about docstrings sound great.

Kai-Striega · 2018-08-18T04:16:24Z

Is there a way to add presolve as an option without adding to _linprog_simplex and _linprog_ip? I don't like passing variables to a function if they are not going to be used anywhere.

mdhaber · 2018-08-18T20:05:52Z

@Kai-Striega It doesn't bother me. If it really bothers you, you could copy the dictionary.

options = {"a":1, "b":2, "presolve":10} # example
used_options = {"rr", "presolve", "_sparse_presolve"}
remaining_options= {key:val for key, val in options.items() if key not in used_options} #unfortunately options.items() creates a list in Python 2, but I don't think it's worth it to have separate commands for Python 2/3
print(remaining_options)

Or perhaps someone else has a more clever solution. But I think it's more trouble than it's worth.

Changes to the ``simplex`` method now provide a more robust simplex solver. Tests previously only passing for ``interior-point`` now also pass for ``simplex``. The tests from ``BaseTestLinprogIP`` have been moved into ``LinprogCommonTests`` to reflect this.

``test_issue_6139`` incorrectly failed for ``TestLinprogIPDense`` due to the tolerance being set aribtrarily low. The commit increases the ``tol`` value inside ``_postprocess(...)`` by a factor of 10.

By default linprog attempts to presolve a problem. If the problem is solved linprog avoids using a solver at all. However the test expects a NotImplementedError to be raised regardless of if a callback is actually required. The test now expects a NotImplementedError only if a callback would actually have been used.

mdhaber · 2018-08-25T00:02:23Z

Hey Kai, this is pretty close? Let me know when you'd like me to take another look.

Kai-Striega · 2018-08-25T02:07:51Z

@mdhaber I'm getting close. I've been busy with uni giving me less time to work on this.

The doc string for linprog(...) now contains the section describing presolve with the references standardised between solver (maybe double check these?). LinprogCommonTests now tests for all tests from BaseTestLinprogIP and all pass.

I still haven't changed how linprog handles the option values and to get the doc tests passing. I've also left most the improvements by @ilayn (for now). They are not directly related to the merging, although I want to keep them in the discussion as I think they are worthwhile but should be a new PR.

Ideally I'll have time to get the above working today or tomorrow. If you would like to have a look at it those changes will be independent of the changes so far.

rgommers · 2018-09-03T01:56:28Z

For now it would just have the sort of information as the one returned by linprog but it is flexible enough to add more information later. How about that?

that sounds good to me.

Kai-Striega · 2018-09-03T04:36:24Z

@mdhaber @rgommers In summary, the signature for callback should be changed to require an OptimizeResult object only. I could change this fairly quickly, however, I would prefer any further changes to be a separate PR.

EDIT: Adding only an OptimizeResult will break backwards compatibility. The current callback(...) function requires the current tableau and is used in linprog_verbose_callback(...). An OptimizeResult object does not contain the tableau.

- Add ``# may vary`` to output slack arguments - Add unit test for docstring example - Extend docstring for status 4 to be more descriptive

…into merge_linprog

mdhaber · 2018-09-04T18:55:43Z

@Kai-Striega Please review the comments. As you can see, we were reluctant to ask for this change, as we know this PR was intended only to refactor code. However, the PR as it stood broke backwards compatibility silently by changing the meaning of the information passed to the callback function without changing the signature. Unfortunatley, this was considered unacceptable. We iterated on how the situation should be handled, and decided that the best course of action - easiest for now and sufficiently flexible for the future - would be to pass an OptimizeResult into the callback function to ensure that callback functions relying on the old callback signature would raise an error. This OptimizeResult should contain, at a minimum, an array x representing the current solution to the original problem. Thoughts on obtaining that (eliminating any phase one artificial variables and calling _postprocess) are above.

You are welcome to modify linprog_verbose_callback as necessary to avoid reliance on the tableau. We might decide in the future to provide the tableau to the current problem - whenever we are ready to also provide information about how the variables of the current problem correspond with those in the original problem. We can always add these things to an OptimizeResult later.

Thanks!

Recent changes to ``linprog`` silently break the (undefined) behaviour of ``callback``. The commit changes the callback signature to require an ``OptimizeResult`` rather than the existing ``xk``, ``kwargs`` by refactoring ``_postprocess`` into multiple, smaller, functions in a new module ``_linprog_util``.

mdhaber

@Kai-Striega Looks great! I don't see very much I'd change. I really like the callback test's way of verifying that the callback function is receiving the information it is supposed to. It is good that problem chosen for the callback test has inequality constraints because then it is known to have variables added in conversion to standard form, so checking, for instance, that the values of x inside the callback are equal to those reported at completion is meaningful.

I might have checked all the fields of the OptimizeResult and chosen a problem that would also have been modified in presolve, and I think that the name of _T_o/T_o is descriptive enough.

The only thing I have to ask for is that the documentation of the callback function argument for linprog and linprog(method='simplex') be modified. They still specify the old signature and reference the tableau.

mdhaber · 2018-09-09T15:58:58Z

@rgommers See anything else that needs to be changed, or can this be merged once the documentation is updated? Regarding the documentation, can you suggest how to call attention to the compatibility break (in 1.2, presumably) and the rationale behind it?

pv · 2018-09-09T16:02:39Z

Notices of backward-compatibility issues should be added to the release notes.
The draft release notes can be edited here: https://github.com/scipy/scipy/wiki/Release-note-entries-for-SciPy-1.2.0 --- the notice should be added there before release.

mdhaber · 2018-09-10T00:40:26Z

Draft release notes edited to reflect this change. @Kai-Striega You're welcome to edit if they could be more clear; just wanted to help.

Kai-Striega · 2018-09-10T04:13:11Z

@mdhaber I've edited the docstrings as requested, plus added more tests for fun returned by the callback. Your release notes cover everything I can think of and I appreciate the help =).

I've done some further refactoring by moving the private functions from _linprog into the linprog_util module. _linprog's module doc string states A top-level linear programming interface. I feel that if we already implement a utility module including the private functions seems appropriate.

Setting the solver_options, _clean_inputs(...) and sparsity checking are now wrapped in _parse_linprog. As mentioned by @ilayn this should help make the internals more concrete.

mdhaber · 2018-09-10T04:17:09Z

@Kai-Striega Took a quick look; it appears that you've updated the docstrings and moved things around in a way that makes sense. Looking forward to the checks passing! If so, are you done?

Kai-Striega · 2018-09-10T04:20:27Z

Yes. This should be it. I think the PR is getting too unwieldly anyway so any extra changes should really be in a new PR

mdhaber · 2018-09-10T17:56:12Z

@rgommers Think this is ready? Do the draft release notes adequately describe the change?

rgommers · 2018-09-10T18:20:22Z

Release note update looks good to me. Go ahead and merge if code is good to go.

rgommers · 2018-09-11T18:48:06Z

Awesome!

Could we close the issues that are now solved? Not quite sure if there's anything left to do in, e.g., gh-8942 or gh-9076.

Kai-Striega added 6 commits August 14, 2018 16:28

Update tests to include presolve.

b326be4

Move simplex into new module and edit docstrings

98e3081

Add simplex method module

f44f313

Add test for simplex without presolve

ac5b5a9

Update linprog tests.

4949184

Kai-Striega force-pushed the merge_linprog branch from 83f328e to 4949184 Compare August 15, 2018 05:00

tylerjereddy added scipy.optimize maintenance Items related to regular maintenance tasks labels Aug 15, 2018

mdhaber reviewed Aug 16, 2018

View reviewed changes

ilayn reviewed Aug 16, 2018

View reviewed changes

Kai-Striega added 5 commits August 21, 2018 14:58

Remove - from array docs

3f408d5

Match references in function docstrings

052b44a

Move simplex method specific tests to common tests

4e235b3

``test_issue_6139`` incorrectly failed for ``TestLinprogIPDense`` due to the tolerance being set aribtrarily low. The commit increases the ``tol`` value inside ``_postprocess(...)`` by a factor of 10.

Kai-Striega added 2 commits September 3, 2018 20:44

Improve linprog docstrings

c3144b7

- Add ``# may vary`` to output slack arguments - Add unit test for docstring example - Extend docstring for status 4 to be more descriptive

Merge branch 'merge_linprog' of https://github.com/Kai-Striega/scipy …

22b8c61

…into merge_linprog

Kai-Striega force-pushed the merge_linprog branch from 651e4f4 to 42c9779 Compare September 9, 2018 09:17

Kai-Striega force-pushed the merge_linprog branch from 42c9779 to 246dc22 Compare September 9, 2018 12:43

mdhaber requested changes Sep 9, 2018

View reviewed changes

mdhaber approved these changes Sep 10, 2018

View reviewed changes

Further code refactoring

c99aff9

Kai-Striega force-pushed the merge_linprog branch from 666363f to c99aff9 Compare September 10, 2018 04:55

ilayn approved these changes Sep 10, 2018

View reviewed changes

mdhaber merged commit f3e0dcf into scipy:master Sep 11, 2018

rgommers added this to the 1.2.0 milestone Sep 11, 2018

This was referenced Sep 12, 2018

Linprog(method='simplex') produces incorrect starting basis. #9008

Closed

MAINT: Refactor _linprog.py and _linprog_ip.py to remove duplicate behaviour. #8942

Closed

mdhaber mentioned this pull request Sep 12, 2018

Bug fixes for #5400, #7237 and #8174 in linprog(method='simplex') #9076

Closed

Kai-Striega deleted the merge_linprog branch September 12, 2018 10:32

mdhaber mentioned this pull request Sep 14, 2018

ENH: linprog in SciPy 1.2+ discussion #9269

Closed

22 tasks

mdhaber mentioned this pull request Apr 22, 2019

Clean _clean_inputs in optimize.linprog #10083

Merged

MAINT: Refactor linear programming solvers #9145

MAINT: Refactor linear programming solvers #9145

Conversation

Kai-Striega commented Aug 15, 2018

mdhaber commented Aug 15, 2018

mdhaber left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdhaber commented Aug 16, 2018 • edited Loading

mdhaber commented Aug 16, 2018

Kai-Striega commented Aug 16, 2018 • edited Loading

ilayn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdhaber Aug 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdhaber commented Aug 16, 2018

Kai-Striega commented Aug 16, 2018

ilayn commented Aug 16, 2018 • edited Loading

Kai-Striega commented Aug 18, 2018

Modifying options:

Tests:

Documentation

mdhaber commented Aug 18, 2018

mdhaber commented Aug 18, 2018

Kai-Striega commented Aug 18, 2018

mdhaber commented Aug 18, 2018

mdhaber commented Aug 25, 2018

Kai-Striega commented Aug 25, 2018 • edited Loading

rgommers commented Sep 3, 2018

Kai-Striega commented Sep 3, 2018 • edited Loading

mdhaber commented Sep 4, 2018 • edited Loading

mdhaber left a comment • edited Loading

Choose a reason for hiding this comment

mdhaber commented Sep 9, 2018

pv commented Sep 9, 2018 • edited Loading

mdhaber commented Sep 10, 2018

Kai-Striega commented Sep 10, 2018 • edited Loading

mdhaber commented Sep 10, 2018

Kai-Striega commented Sep 10, 2018

mdhaber commented Sep 10, 2018

rgommers commented Sep 10, 2018

rgommers commented Sep 11, 2018

mdhaber commented Aug 16, 2018 •

edited

Loading

Kai-Striega commented Aug 16, 2018 •

edited

Loading

mdhaber Aug 16, 2018 •

edited

Loading

ilayn commented Aug 16, 2018 •

edited

Loading

Modifying `options`:

Kai-Striega commented Aug 25, 2018 •

edited

Loading

Kai-Striega commented Sep 3, 2018 •

edited

Loading

mdhaber commented Sep 4, 2018 •

edited

Loading

mdhaber left a comment •

edited

Loading

pv commented Sep 9, 2018 •

edited

Loading

Kai-Striega commented Sep 10, 2018 •

edited

Loading