-
-
Notifications
You must be signed in to change notification settings - Fork 25.8k
Yield stack trace information in resilient mode model_selection warnings #15622
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Thanks please add a unit test. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we just use format_exc?
I will see about the unit test shortly, I realized the Exception object does not have the traceback which needs to be retrieved so it was giving errors. Also format_exception is certainly better and simply not trimming it to line 0. |
It seems the functionality is basically tested already through test_validation.py. However _utility.py assert_warns_message and it would be pretty contrived to put the exact stack trace in. However that macro-like function could be changed to support a stacktrace injected into the middle of the message which will clear the errors that are occurring now via regular expressions perhaps if making it not overly specific. Does this require such a precise stacktrace specific test? It would be highly unmaintainable since when line numbers change, so would the message need to change. For now I am trying to just make the errors vanish while having a stacktrace which I think is near achieved. |
I edited above for this important detail: Note: this change could break functionality where the warning message is parsed and assumes a 2 line message with the warning tag line and the specific Python exception detail. The new format would be warning tag line + stack trace + specific Python exception detail. If we want backward compatibility, then a flag would need to be added. However aside from test code which is being fixed, I think generally this could be an upgrade requirement, rather than over-cluttering the class parameters for every class which uses this - may be more than just GridSearchCV. |
It seems to be passing now. However some of the build systems seem to have an out of date exceptions.py or some bizarre newline detail is there. I think it's because they are not getting the version from the PR but from master. Passed on 5, failed on 4... |
Now with unit test and attempt at making it as backward compatible as possible. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lgtm if tests pass
I don't understand what they changed in doctest in Python 3.8 but its making getting straight forward things like this difficult. All the tests passed and the ellipsis should work just fine in all. |
I don't think it's a change in py 3.8 ... We don't run doctest in all CI I hope one of us can help soon |
The error message shows that it does not compare. But I was able to resolve the Py 3.8 doctest before using |
Really I have no idea what changed in doctest 3.8 as I already checked A section What About Exceptions? implies tracebacks are handled specially. But in our case we catch the exception and merely yield the string for comparison so none of that should apply at all. |
I have tried being very creative even helping guide the pattern matching along. And everything always passes fine in Py3.5/3.7. As for pylatest, nothing yet has worked. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you very much for this fix. I don't exactly understand still what changed in Python 3.8 that is causing such peculiarities with new lines.
The issue stemmed from "..." being a greedy regex: Reference: https://docs.python.org/3.7/library/doctest.html#doctest.ELLIPSIS (I was using 3.7 for testing this locally) |
I had known about it and tried to work around it - why I put |
Co-Authored-By: Thomas J Fan <thomasjpfan@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add an Enhancement
to the change log at doc/whats_new/v0.23.rst. Like the other entries there, please reference this pull request with :pr:
and credit yourself (and other contributors if applicable) with :user:
Thank you @GregoryMorse ! |
Fixes #15621
Now a stack trace will be included in the FitFailedWarning so that not having early termination will still allow proper analysis of errors.
Note: this change could break functionality where the warning message is parsed and assumes a 2 line message with the warning tag line and the specific Python exception detail.
The new format would be warning tag line + stack trace + specific Python exception detail. If we want backward compatibility, then a flag would need to be added. However aside from test code which is being fixed, I think generally this could be an upgrade requirement, rather than over-cluttering the class parameters for every class which uses this - may be more than just GridSearchCV.