A Python library for automating interaction with websites. MechanicalSoup automatically stores and sends cookies, follows redirects, and can follow links and submit forms.
pip install MechanicalSoup
WWW-Mechanize and Mechanize are similar Perl and Ruby libraries. Unfortunately, their Python sister project has become inactive, and is now incompatible with recent Python versions (Python 3.x). This library provides a similar API, built on Python giants Requests (for http sessions) and BeautifulSoup (for document navigation).
example.py, code to log into the GitHub website:
import mechanicalsoup browser = mechanicalsoup.Browser() # request github login page login_page = browser.get("https://github.com/login") # find login form login_form = login_page.soup.select("#login").select("form") # specify username and password login_form.select("#login_field")['value'] = "username" login_form.select("#password")['value'] = "password" # submit form page2 = browser.submit(login_form, login_page.response.url) # verify we are now logged in assert page2.soup.select(".logout-form") # verify we remain logged in (thanks to cookies) as we browse the rest of the site page3 = browser.get("https://github.com/colonelpanic/MechanicalSoup") assert page3.soup.select(".logout-form")
For an example with a more complicated form (with checkboxes, radio buttons and textareas), read