Yield stack trace information in resilient mode model_selection warnings

[GregoryMorse]

Fixes #15621

Now a stack trace will be included in the FitFailedWarning so that not having early termination will still allow proper analysis of errors.

Note: this change could break functionality where the warning message is parsed and assumes a 2 line message with the warning tag line and the specific Python exception detail.

The new format would be warning tag line + stack trace + specific Python exception detail. If we want backward compatibility, then a flag would need to be added. However aside from test code which is being fixed, I think generally this could be an upgrade requirement, rather than over-cluttering the class parameters for every class which uses this - may be more than just GridSearchCV.

[rth]

Thanks please add a unit test.

[jnothman]

Should we just use format_exc?

[GregoryMorse]

I will see about the unit test shortly, I realized the Exception object does not have the traceback which needs to be retrieved so it was giving errors. Also format_exception is certainly better and simply not trimming it to line 0.

[GregoryMorse]

It seems the functionality is basically tested already through test_validation.py. However _utility.py assert_warns_message and it would be pretty contrived to put the exact stack trace in. However that macro-like function could be changed to support a stacktrace injected into the middle of the message which will clear the errors that are occurring now via regular expressions perhaps if making it not overly specific. Does this require such a precise stacktrace specific test? It would be highly unmaintainable since when line numbers change, so would the message need to change. For now I am trying to just make the errors vanish while having a stacktrace which I think is near achieved.

[GregoryMorse]

I edited above for this important detail:

Note: this change could break functionality where the warning message is parsed and assumes a 2 line message with the warning tag line and the specific Python exception detail.

The new format would be warning tag line + stack trace + specific Python exception detail. If we want backward compatibility, then a flag would need to be added. However aside from test code which is being fixed, I think generally this could be an upgrade requirement, rather than over-cluttering the class parameters for every class which uses this - may be more than just GridSearchCV.

[GregoryMorse]

It seems to be passing now. However some of the build systems seem to have an out of date exceptions.py or some bizarre newline detail is there. I think it's because they are not getting the version from the PR but from master. Passed on 5, failed on 4...

[GregoryMorse]

Now with unit test and attempt at making it as backward compatible as possible.

[jnothman]

Lgtm if tests pass

[GregoryMorse]

I don't understand what they changed in doctest in Python 3.8 but its making getting straight forward things like this difficult. All the tests passed and the ellipsis should work just fine in all.

[jnothman]

I don't think it's a change in py 3.8 ... We don't run doctest in all CI

I hope one of us can help soon

[GregoryMorse]

.. We don't run doctest in all CI

The error message shows that it does not compare. But I was able to resolve the Py 3.8 doctest before using '\\\\n' instead of '\\n' so I am almost sure its some intricacy with newlines and ellipsis. I am not sure the doctest version on Py3.8 vs the rest. I have not yet even installed Py3.8 even on my own system.

[GregoryMorse]

=================================== FAILURES ===================================
________________ [doctest] sklearn.exceptions.FitFailedWarning _________________
[gw1] linux -- Python 3.8.0 /usr/share/miniconda/envs/testvenv/bin/python
125     Examples
126     --------
127     >>> from sklearn.model_selection import GridSearchCV
128     >>> from sklearn.svm import LinearSVC
129     >>> from sklearn.exceptions import FitFailedWarning
130     >>> import warnings
131     >>> warnings.simplefilter('always', FitFailedWarning)
132     >>> gs = GridSearchCV(LinearSVC(), {'C': [-1, -2]}, error_score=0, cv=2)
133     >>> X, y = [[1, 2], [3, 4], [5, 6], [7, 8]], [0, 0, 1, 1]
134     >>> with warnings.catch_warnings(record=True) as w:
Expected:
    FitFailedWarning('Estimator fit failed. The score on this train-test
    partition for these parameters will be set to 0.000000.
    Details: \n...ValueError: Penalty term must be positive; got (C=-2)\n'
    ...)
Got:
    FitFailedWarning('Estimator fit failed. The score on this train-test partition for these parameters will be set to 0.000000. Details: \nTraceback (most recent call last):\n  File "/home/vsts/work/1/s/sklearn/model_selection/_validation.py", line 515, in _fit_and_score\n    estimator.fit(X_train, y_train, **fit_params)\n  File "/home/vsts/work/1/s/sklearn/svm/_classes.py", line 230, in fit\n    raise ValueError("Penalty term must be positive; got (C=%r)"\nValueError: Penalty term must be positive; got (C=-2)\n')

Really I have no idea what changed in doctest 3.8 as I already checked https://docs.python.org/3/library/doctest.html and there are no significant changes since Python 3.4. Obviously the new line and ellipsis have some details that must have changed somehow.

A section What About Exceptions? implies tracebacks are handled specially. But in our case we catch the exception and merely yield the string for comparison so none of that should apply at all.

[GregoryMorse]

I have tried being very creative even helping guide the pattern matching along. And everything always passes fine in Py3.5/3.7. As for pylatest, nothing yet has worked.

[GregoryMorse]

Thank you very much for this fix. I don't exactly understand still what changed in Python 3.8 that is causing such peculiarities with new lines.

[thomasjpfan]

The issue stemmed from "..." being a greedy regex:

Reference: https://docs.python.org/3.7/library/doctest.html#doctest.ELLIPSIS

(I was using 3.7 for testing this locally)

[GregoryMorse]

The issue stemmed from "..." being a greedy regex:
Reference: https://docs.python.org/3.7/library/doctest.html#doctest.ELLIPSIS
(I was using 3.7 for testing this locally)

I had known about it and tried to work around it - why I put ValueError in there twice but alas I failed to think about it enough. Regardless, appreciate your efforts - the PR is looking good now.

[thomasjpfan]

Please add an Enhancement to the change log at doc/whats_new/v0.23.rst. Like the other entries there, please reference this pull request with :pr: and credit yourself (and other contributors if applicable) with :user:

[thomasjpfan]

Thank you @GregoryMorse !