Easier wrangling of web data.
Switch branches/tags
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
docs
.gitignore
.travis.yml
CHANGES.md
LICENSE
MANIFEST.in
README.md
requirements-rtd.txt
requirements-travis.txt
setup.py
soupy.py
test_soupy.py

README.md

Soupy

Latest Version License Build Status Coverage Status

Soupy is a wrapper around BeautifulSoup that makes it easier to build complex queries when wrangling web data.

Here's an example of a Soupy query.

from soupy import Soupy, Q

html = """
<div id="main">
  <div>The web is messy</div>
  and full of traps
  <div>but Soupy loves you</div>
</div>"""

print(Soupy(html).find(id='main').children
      .each(Q.text.strip()) # extract text from each node, trim whitespace
      .filter(len)          # remove empty strings
      .val())               # dump out of Soupy

# ['The web is messy', 'and full of traps', 'but Soupy loves you']

The same query using BeautifulSoup:

 from bs4 import BeautifulSoup, NavigableString

html = """
<div id="main">
  <div>The web is messy</div>
  and full of traps
  <div>but Soupy loves you</div>
</div>"""

result = []
for node in BeautifulSoup(html).find(id='main').children:
    if isinstance(node, NavigableString):
        text = node.strip()
    else:
        text = node.text.strip()
    if len(text):
        result.append(text)

print(result)

For more information, see the Soupy Documentation

Installation

pip install soupy

Dependencies

six and BeautifulSoup4

Soupy is supported on Python 2.6+ and 3.3+