Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Override submit method in StatefulBrowser #233

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Override submit method in StatefulBrowser #233

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ada520b
stateful_browser.py: rename submit_selected to submit
hemberger Aug 29, 2018
f681e7b
stateful_browser.py: improve submit backwards compatibility
hemberger Aug 29, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ a DuckDuckGo search:
# Fill-in the search form
browser.select_form('#search_form_homepage')
browser["q"] = "MechanicalSoup"
browser.submit_selected()
browser.submit()

# Display the results
for link in browser.get_current_page().select('a.result__a'):
Expand Down
9 changes: 9 additions & 0 deletions docs/ChangeLog.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,15 @@ Release Notes
Version 1.0 (in development)
============================

Main changes:
-------------

* **Breaking Change:** The method ``StatefulBrowser.submit_selected`` has been
renamed to :func:`StatefulBrowser.submit`. The original name remains usable
for backwards compatibility. This is a breaking change _only_ if you use the
:func:`Browser.submit` method from a ``StatefulBrowser`` instance (this is
not typical), since it is now overridden by :func:`StatefulBrowser.submit`.

Bug fixes
---------

Expand Down
2 changes: 1 addition & 1 deletion docs/tutorial.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ used to list the fields)::
Assuming we're satisfied with the content of the form, we can submit
it (i.e. simulate a click on the submit button)::

>>> response = browser.submit_selected()
>>> response = browser.submit()

The response is not an HTML page, so the browser doesn't parse it to a
BeautifulSoup object, but we can still see the text it contains::
Expand Down
2 changes: 1 addition & 1 deletion examples/example.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
browser.select_form('#login form')
browser["login"] = args.username
browser["password"] = args.password
resp = browser.submit_selected()
resp = browser.submit()

# Uncomment to launch a web browser on the current page:
# browser.launch_browser()
Expand Down
2 changes: 1 addition & 1 deletion examples/expl_duck_duck_go.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
# Fill-in the search form
browser.select_form('#search_form_homepage')
browser["q"] = "MechanicalSoup"
browser.submit_selected()
browser.submit()

# Display the results
for link in browser.get_current_page().select('a.result__a'):
Expand Down
2 changes: 1 addition & 1 deletion examples/expl_google.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
browser["q"] = "MechanicalSoup"
# Note: the button name is btnK in the content served to actual
# browsers, but btnG for bots.
browser.submit_selected(btnName="btnG")
browser.submit(btnName="btnG")

# Display links
for link in browser.links():
Expand Down
2 changes: 1 addition & 1 deletion examples/expl_httpbin.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,5 +23,5 @@
# Uncomment to display a summary of the filled-in form
# browser.get_current_form().print_summary()

response = browser.submit_selected()
response = browser.submit()
print(response.text)
2 changes: 1 addition & 1 deletion mechanicalsoup/form.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -306,7 +306,7 @@ def choose_submit(self, submit):
browser.open(url)
form = browser.select_form()
form.choose_submit('form_name_attr')
browser.submit_selected()
browser.submit()
"""
# Since choose_submit is destructive, it doesn't make sense to call
# this method twice unless no submit is specified.
Expand Down
25 changes: 22 additions & 3 deletions mechanicalsoup/stateful_browser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
import sys
import re
import bs4
import warnings


class _BrowserState:
Expand Down Expand Up @@ -60,6 +61,10 @@ def __init__(self, *args, **kwargs):
self.__verbose = 0
self.__state = _BrowserState()

# Aliases for backwards compatibility
# (Included specifically in __init__ to suppress them in Sphinx docs)
self.submit_selected = self.submit

def set_debug(self, debug):
"""Set the debug mode (off by default).

Expand Down Expand Up @@ -207,7 +212,7 @@ def select_form(self, selector="form", nr=0):

return self.get_current_form()

def submit_selected(self, btnName=None, *args, **kwargs):
def submit(self, btnName=None, *args, **kwargs):
"""Submit the form that was selected with :func:`select_form`.

:return: Forwarded from :func:`Browser.submit`.
Expand All @@ -216,6 +221,19 @@ def submit_selected(self, btnName=None, *args, **kwargs):
to :func:`Form.choose_submit` on the current form to choose between
them. All other arguments are forwarded to :func:`Browser.submit`.
"""
# Temporarily allow calling the old inherited Browser.submit directly
# in case we can detect with certainty that the call is an old style.
# Note: Browser.submit also accepts a bs4.element.Tag with name="form",
# but we cannot assume this is an old-style call since there could be
# a submit button with name="form".
if isinstance(btnName, Form):
warnings.warn("This usage of StatefulBrowser.submit is deprecated."
" Please see the documentation for this function to "
"upgrade to the new interface.",
DeprecationWarning)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, forbidding this is not only a backward-compatibility problem, but also a violation of Liskov's Substitution Principle. The base .submit() accepts a btnName instance of Form, so all derived methods should also accept at least that (and possibly more), so that anything doable with a Browser is also doable with a StatefulBrowser.

Also, having the same positional argument be btnName in one class and form in another is very misleading. LGTM is right to complain here IMHO.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before I invest too much more time into this, I just wanted to check if you are generally opposed to this PR, or if you can imagine a modified PR that you would consider accepting. Even better if you already have an idea in mind! ;)

My preference is always to make StatefulBrowser the best class it can be. The Browser class sometimes gets in the way of that, such as in this case. I believe StatefulBrowser would benefit from a submit method, so if we can't override Browser.submit in the proposed way, the first two alternatives that come to mind (setting aside the issue of backwards compatibility for a moment) are renaming Browser.submit to either a) Browser._submit or b) Browser.submit_form.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I were to redo this from scratch, I'd consider using composition instead of inheritance, i.e. consider that a StatefulBrowser contains a Browser, not that it is a Browser. It'd be hard to change that without breaking backward compatibility, though.

Renaming Browser.submit is also problematic with respect to backward compatibility. Perhaps not many people call submit on a StatefulBrowser, but everyone who wrote code before StatefulBrowser was added does call submit on a Browser.

There's another option: add warnings for suspicious cases, and allow the user to disable the warnings as needed. We could warn on calls to StatefulBrowser.submit, or if we want to go further, on creation of a Browser that isn't a StatefulBrowser. Probably not as a deprecation warning like "what you're doing won't work anymore soon", but just like "uh, are you sure you want that?".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A warning for suspicious use is a great idea, but I'm not sure that it addresses the main issues here (and might be just as much of an annoyance as a modified interface to some users). Out of curiosity, though, did you have a design in mind for warning toggling?

Okay, two more possibilities:

  1. Simply add a comment to the Browser.submit docstring saying something like:

If you are calling this method from a StatefulBrowser instance, consider using StatefulBrowser.submit_selected instead.

  1. Override Browser.submit in a Liskov-compliant way, e.g.
def submit(self, form=None, url=None, **kwargs, btnName=None):

where form and url default to the current form/url if None. It's a bit of a clunky interface, but might be the best we can do if we want to make overriding work within our constraints. Or even just

def submit(self, form=None, url=None, **kwargs):

to avoid complications with btnName.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@moy I'd like to get version 0.11 released as soon as possible, but I want to address this issue in some way before I do, even if it's just a minor addition to the documentation.

How about I make the documentation change proposed above for 0.11, and then we can reassess if we want to make any code changes for version 1.0 later?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The doc change is obviously a good step forward.

Your option 2. is probably doable too. I won't have time to look at this in details soon, but if you're confident enough I trust you to do the right thing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback! For the 0.11 release I'll just change the docstring, unless you recommend otherwise.

As I think more about it, overriding submit seems increasingly important. One of the biggest issues (that I didn't appreciate until just now) is that if you call submit on a StatefulBrowser, the browser state becomes stale. I would argue that this is a pretty dangerous inconsistency for a method that looks like the most obvious way to submit a form.

return super(StatefulBrowser, self).submit(btnName, *args,
**kwargs)

self.get_current_form().choose_submit(btnName)

referer = self.get_url()
Expand All @@ -225,8 +243,9 @@ def submit_selected(self, btnName=None, *args, **kwargs):
else:
kwargs['headers'] = {'Referer': referer}

resp = self.submit(self.__state.form, url=self.__state.url,
*args, **kwargs)
resp = super(StatefulBrowser, self).submit(self.__state.form,
url=self.__state.url,
*args, **kwargs)
self.__state = _BrowserState(page=resp.soup, url=resp.url,
request=resp.request)
return resp
Expand Down
15 changes: 15 additions & 0 deletions tests/test_stateful_browser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -621,5 +621,20 @@ def test_refresh_error():
browser.refresh()


def test_deprecated_submit(recwarn):
"""Check that calling StatefulBrowser.submit forwards to the base class
with a deprecation warning when a deprecated call is detected."""
expected_post = [('diff', 'Review Changes'),
('text', 'All I know is my gut says maybe')]
browser, url = setup_mock_browser(expected_post=expected_post)
browser.open(url)
form = browser.select_form('#choose-submit-form')
form.choose_submit(expected_post[0][0])
form[expected_post[1][0]] = expected_post[1][1]
res = browser.submit(form, browser.get_url())
assert issubclass(recwarn.pop().category, DeprecationWarning)
assert(res.status_code == 200 and res.text == 'Success!')


if __name__ == '__main__':
pytest.main(sys.argv)