GitHub - afcarl/vocabulary-builder: Simple Python tool for selecting most used words from specific web-sites.

afcarl / vocabulary-builder Public

forked from mushketyk/vocabulary-builder

Notifications You must be signed in to change notification settings
Fork 0
Star 0

Simple Python tool for selecting most used words from specific web-sites.

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 8 Commits
README		README
builder.py		builder.py
config.py		config.py

Repository files navigation

vocabulary-builder is a simple script that collects words that occur the most frequently on a set of a crawled web-pages.

It works as follows: start on a set of web-pages, strips HTML tags, count occurrence of each word on a page, collect all links from the page, repeat the same process for all pages that had links from the first page.

This script can be helpful for collecting set of words that can be used later as inputs for neural networks.

Used libraries:
* stemming 1.0 - for word stemming
* BeautifulSoup - for HTML parsing