activesoup

A simple library for interacting with the web from python

Description

activesoup combines familiar python web capabilities for convenient headless "browsing" functionality:

• Modern HTTP support with requests - connection pooling, sessions, ...
• Convenient access to the web page with an interface inspired by beautifulsoup - convenient HTML navigation.
• Robust HTML parsing with html5lib - parse the web like browsers do.

Full documentation can be found at https://activesoup.dev.

Use cases

activesoup aims to provide just enough functionality for basic web automation / crawler tasks. Consider using activesoup when:

• You've already checked out requests-html
• You need to actively interact with some web-page from Python (e.g. submitting forms, downloading files)
• You don't control the site you need to interact with (if you do, just make an API).
• You don't need javascript support (you'll need selenium or phantomjs).

Usage examples

In the example below, we'll load a page with a simple form, enumerate the fields, and make a submission:

>>> import activesoup

>>> # Start a session
>>> d = activesoup.Driver()

>>> page = d.get("https://httpbin.org/forms/post")

>>> # conveniently access elements, inspired by BeautifulSoup
>>> form = page.form

>>> # get the power of raw xpath search too
>>> form.find('.//input[@name="size"]')
BoundTag<input>
>>> # any element, searching by attribute
>>> form.find('.//*', name="size")
BoundTag<input>
>>> # or just search by attribute
>>> form.find(name="size")
BoundTag<input>

>>> # inspect element attributes
>>> print([i['name'] for i in form.find_all('input')])
['custname', 'custtel', 'custemail', 'size', 'size', 'size', 'topping', 'topping', 'topping', 'topping', 'delivery']

>>> # work actively with objects on the page
>>> r = form.submit({"custname": "john", "size": "small"})

>>> # responses parsed and ready based on content type
>>> r.keys()
dict_keys(['args', 'data', 'files', 'form', 'headers', 'json', 'origin', 'url'])
>>> r['form']
{'custname': 'john', 'size': 'small', 'topping': 'mushroom'}

>>> # access the underlying requests.Session too
>>> d.session
<requests.sessions.Session object at 0x7f283dc95700>

>>> # log in with cookie support
>>> d.get('https://httpbin.org/cookies/set/foo/bar')
>>> d.session.cookies['foo']
'bar'