Scrapy Phrases and Make Them Yours

I'm a computer science student at the University of Arizona and it's actually been only a year since I came to the states. Still struggling with English but feel like I actually enjoy learning the foreign language! It is true that I'm exposed to the foreign environment so improving my speech bit by bit but I'm not going to be too pleased to stay this level. Anyway, I made this repository just for memorizing vocabularies. All the files here are simply just for the format and have no special things (but might seem like a pretty dirty in a way I used selectors because I'm not even much familiar with the world of web) so that beginners can modify them easily (hopefully..). For memorizing words even outside, you may import an excel file into an app such as Lexilize what I'm going to use. Yea, I totally forgot there are a bunch of useful vocabulary apps that's why I coded crawlers. However, I'm pretty sure my own words crawler can give me power not to give up easily.

* A Screenshot of Lexilize.

Four Crawlers

1. kyphrase.py (ENG) Idiom # the use of the item pipeline.
2. inlingua.py - (ENG) 200-common-phrasal-verbs # the simplest crawler-1.
3. urbandictionary.py - (ENG) Slang # the multiple subpages crawler.
4. spanish.py - (SPN) 1000-most-common-spanish-words # the simplest crawler-2.

Example

Usage

To scrape the site and save the data:

$ cd kyphrase
$ scrapy crawl kyphrase

Tutorial

1. Choose a website you want to crawl! If you want to check out the allowance for the access, append 'robots.txt' at the end of the domain.

    ex) [https://knowyourphrase.com/robots.txt](https://knowyourphrase.com/robots.txt)
   

2. To start scrapy:

    $ scrapy startproject kyphrase
   

3. Make your spider file (e.g., kyphrase.py) under the spiders directory.

    ├── kyphrase
    │   ├── __init__.py
    │   ├── __pycache__
    │   ├── items.py
    │   ├── middlewares.py
    │   ├── pipelines.py
    │   ├── settings.py
    │   └── spiders
    │       ├── __init__.py
    │       ├── __pycache__
    │       └── kyphrase.py
    └── scrapy.cfg
   

4. Copy the code below into the spider file (e.g., kyphrase.py) and change URL, indexlist, the class name, and name under the class to fit your website where you want to crawl.

import scrapy

URL = "https://[YOUR_DOMAIN]/{index}"
indexlist = ["hello world", 1, 2, 3]

class KyphraseSpider(scrapy.Spider):
        name = "kyphrase"
        start_urls = [URL.format(index=index) for index in indexlist]

        def parse(self, response):
        """implement parsing here"""

        import ipdb; ipdb.set_trace() # recommend use the ipdb, an IPython debugger, but not required

* To install the IPython debugger(ipdb):

        $ pip install ipdb

5. Figure out what html tags are surrounding the information what you want to extract. To view the source of the web page, append view-source: at the beginning of the url. In my case:

    view-source:https://knowyourphrase.com/a
   

6. Code the parsing part using the debugger or a web developer tool. To note about selectors, items, and pipelines, please refer to the link below.

   • Selectors
   • Items
   • Item Pipeline

Error Handling

1. AttributeError: 'TelnetConsole' object has no attribute 'port'

• add a line "TELNETCONSOLE_PORT = None" into the settings.py or comment off this:

    EXTENSIONS = {
    'scrapy.extensions.telnet.TelnetConsole': None,
    }
  

2. If your crawler is blocked, you should check the allowance by appending robots.txt at the end of the domain and modify settings.py.

Author

Sooyoung Moon / @symoon94