Documentation

Documentation

Single Threaded Automations

Here is an example of a single threaded automation. First, we will import all the appropriate pieces.

from time import sleep

# Database models used to interact with databases if needed.
from Project import Models

# The environment variable loader.  These variables can be set in the .env file.
# This is important if we want to create configurable web automations / scrapers.
from Config.Environment import env, env_driver

# Controllers utilize a `WebDriver` instance in order to control the various web 
# pages that need to be scraped.
from SiteAutomations.Examples import GoogleExample, BingExample

Controllers are kept in the SiteAutomations folder.

# The quitting contexts helps to `close()` and `quit` the WebDriver instance if 
# something goes wrong.
from Helpers.Contexts import quitting
from selenium.webdriver.support.wait import WebDriverWait

To get a hold of our WebDriver instance, we need to use the env_driver and env functions. env('BROWSER') will return the name of the browser set in the .env file and env_driver takes the name of the browser, and returns the appropriate WebDriver instance. The quitting function is used to open the WebDriver instance the same way you would open a file using with. When you add all of this together, you get:

# This could be written as:
#
# browser = env("BROWSER")
# web_driver = env_driver(browser)
# with quitting(web_driver()) as driver:
#     pass
with quitting(env_driver(env("BROWSER"))()) as driver:
    # Do stuff.

Now that we have a valid WebDriver instance, we can instantiate our Controllers and do some work.

    # Get an instance of `WebDriverWait`.
    wait = WebDriverWait(driver, 30)

    # Pass the web driver to the site automation along with anything
    # else it might need to do its job. This could include an
    # instance of `WebDriverWait`, and even the collection of
    # Models.
    google_search = GoogleExample.GoogleSearch(driver, wait, Models)
    bing_search = BingExample.BingSearch(driver, wait, Models)

    # Do stuff with your controllers.
    google_search.do_search('google wiki')
    sleep(5)
    bing_search.do_search('bing wiki')
    sleep(5)

Controllers

There are 2 types of controllers. The first type is really just a class that controls a WebDriver instance. The other is a class that inherits from IndependentController, and controls a WebDriver instance. The only difference is instances of IndpendentController attach their instance of WebDriver after they're instantiated using it's attach_driver method. This facilitates the use of the ThreadedCommandFactory and CommandFactory objects. A basic controller might look something like:

class Google(object):
    def __init__(self, driver, wait):
        self.driver = driver
        self.wait = wait

    def do_search(self, search_term):
        self.driver.get('https://google.com')

        # Type search
        search_input = self.driver.find_element_by_name('q')
        search_input.send_keys(search_term)

        # Click search button.
        search_button = self.driver.find_element_by_name('btnG')
        search_button.click()
        self.wait.until(lambda the_driver: the_driver.find_element_by_id('resultStats').is_displayed())
        return self

ThreadedCommandFactory and CommandFactory