I'm getting the following error:

[emilyanncr]

Great script. I'm getting the following error however even after installing the requirements:

emily@kali:/Tools/pagodo$ sudo pip3 install -r requirements.txt
[sudo] password for emily:
Requirement already satisfied: beautifulsoup4>=4.6.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 1))
Collecting google>=2.0.1 (from -r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/c8/b1/887e715b39ea7d413a06565713c5ea0e3132156bd6fc2d8b165cee3e559c/google-2.0.1.tar.gz
Requirement already satisfied: numpy>=1.13.3 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 3))
Requirement already satisfied: requests>=2.18.4 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 4))
Building wheels for collected packages: google
Running setup.py bdist_wheel for google ... done
Stored in directory: /root/.cache/pip/wheels/b3/6d/94/ad59f018e26ad1987116a8eda758a4dd4285fcb0b4daf7c50d
Successfully built google
Installing collected packages: google
Successfully installed google-2.0.1
emily@kali:/Tools/pagodo$ python ghdb_scraper.py -n 5 -x 3875 -s -t 3
Traceback (most recent call last):
File "ghdb_scraper.py", line 15, in
from bs4 import BeautifulSoup
File "/usr/local/lib/python2.7/dist-packages/bs4/init.py", line 30, in
from .builder import builder_registry, ParserRejectedMarkup
File "/usr/local/lib/python2.7/dist-packages/bs4/builder/init.py", line 314, in
from . import _html5lib
File "/usr/local/lib/python2.7/dist-packages/bs4/builder/_html5lib.py", line 70, in
class TreeBuilderForHtml5lib(html5lib.treebuilders._base.TreeBuilder):
AttributeError: 'module' object has no attribute '_base'

Suggestions? Thanks.

[opsdisk]

Hi @emilyanncr - thanks for reaching out.

What is the output of this command?

python -V

It looks like you are calling Python2 instead of Python3. You can try python3 -V to see if it's installed. If so, the script would be called as:

python3 ghdb_scraper.py -n 5 -x 3875 -s -t 3

I've only tested it on Python3 since Python2 is on it's way out. Let me know if that works for you.

[opsdisk]

Did you get it working @emilyanncr ?

[opsdisk]

Closing this out. Let me know if you have any other questions.