Webscrapping

• Install Anisible on local machine or virtual machine

  Ubuntu

    $ sudo apt update
    $ sudo apt install software-properties-common
    $ sudo apt-add-repository --yes --update ppa:ansible/ansible
    $ sudo apt install ansible
  

  CentOs

    $ sudo yum install python3-pip
    $ sudo pip install --upgrade pip
    $ sudo pip3 install ansible
  

• Ansible will connect to AWS using boto SDK. So we have to install boto and boto3 ackages on our local machine or VM

    $ sudo pip3 install boto boto3
  

• Install AWS CLI on the VM

    $ pip3 install awscli --upgrade --user
  

• Go to AWS console and create an IAM User. Add it to a group each having AmazonEC2FullAccess permissions.

• Copy the user's AWS Access Key ID and AWS Secret Access Key

• Return to Virtual Machine. Configure AWS with the user keys and set the default region as per requirement

    $ aws configure
  

• Also create an .aws/credentials file and copy the keys and default region to it so that ansible can access the keys accordingly

• Create a keypair and ensure the keypair.pem file is accessible to virtual machine.

• Create an EC2 Instance(CentOs machine) using Ansible playbook and add host to group 'just_created' with variable foo=42

    - name: Create an EC2 instance
        hosts: local
        connection: local
        gather_facts: False
        tasks:
            - name: Launch instance
             ec2:
                key_name: ansible-lab
                group: ansible-node
                instance_type: t2.micro
                image: ami-02354e95b39ca8dec
                wait: true
                region: us-east-1
                # aws_access_key: "{{ lookup('env', 'AWS_ACCESS_KEY') }}"
                # aws_secret_key: "{{ lookup('env', 'AWS_SECRET_KEY') }}"
            register: ec2
            - name: Print all ec2 variables
            debug: var=ec2
            - name: Get the Ip address
            debug: var=ec2.instances[0].public_dns_name
  
             - name: add host to group 'just_created' with variable foo=42
                add_host:
                name: "{{ ec2.instances[0].public_dns_name }}"
                groups: ec2_hosts
                ansible_host: "{{ ec2.instances[0].public_dns_name }}"
                ansible_ssh_user: ec2-user
                ansible_ssh_private_key_file: /vagrant_data/ansible-lab.pem
  

  Replace the following values in the playbook:

    hosts: localhost
    key_name: name of the key-pair you will use to ssh to ec2 instance.
    group: security group of ec2 (ssh port should be open for the security group used)
    instance_type, image : as per the ec2 instance you want to create.
    region: as per the requirement.
  

  Make sure that the full path to .pem key file is profided to the "ansible_ssh_private_key_file" parameter

• Install packages onthe EC2 instance: This step can be done manually after sshing to the EC2 instance or automatically through ansible playbook. First method: Execute the below commands based on the OS of EC2 instances for installing following packages: 1. python

  Ubuntu

    $ sudo apt-get update && sudo apt-get upgrade -y
    $ sudo apt-get install python3.7
  

  CentOs

    $ sudo yum install -y https://repo.ius.io/ius-release-el7.rpm
    $ sudo yum update -y
    $ sudo yum install -y python36u python36u-libs python36u-devel python36u-pip
  

  2. pip

  CentOS-7 and higher

  Install pip in CentOS, using yum and python3 package manager:

    $ sudo yum install python3-pip
    $ sudo pip install --upgrade pip
  

  Ubuntu

  Install pip in Ubuntu, using apt-get package manager:

    $ sudo apt-get update -y
    $ sudo apt-get install python3-pip
    $ sudo pip install --upgrade pip
  

  3. git

  CentOs

    $ sudo yum install git
  

  Ubuntu

    $ sudo apt-get install git
  

  4. mariadb server

  CentOs

    $ sudo yum install mariadb-server
  

  For starting the mysql server.

    $ sudo systemctl start mariadb
    $ sudo systemctl status mariadb
    $ sudo echo -e "\n\nroot\nroot\n\n\nn\n\n " | mysql_secure_installation 2>/dev/null
  

  Note: -e enable interpretation of the following backslash escapes 2>/dev/null will filter out the errors so that they will not be output to your console. In more detail: 2 represents the error descriptor, which is where errors are written to. ... /dev/null is the standard Linux device where you send output that you want ignored.

  Ubuntu

    $ sudo apt update
    $ sudo apt install mariadb-server
    $ sudo echo -e "\n\nroot\nroot\n\n\nn\n\n " | mysql_secure_installation 2>/dev/null
  

  4. BeautifulSoup4

  CentOs

    $ sudo pip3 install bs4
  

  Ubuntu

    $ sudo apt-get update -y
    $ sudo apt-get install -y python3-bs4
    $ sudo apt-get install -y python-beautifulsoup
  

  5. Requests

  CentOs

    $ sudo pip3 install requests
  

  Ubuntu

    $ sudo apt-get update -y
    $ sudo apt-get install -y python3-requests
  

  Second Method: Using Ansible Tasks to install all above packages:

    name: Install packages into ec2 hosts
    hosts: ec2_hosts
    become: yes
    tasks:
        - yum: pkg=python3 state=latest
        - yum: pkg=python3-pip state=latest
        - yum: pkg=git state=installed
        - yum: pkg=mariadb-server state=installed
        - shell: sudo systemctl start mariadb
        - shell: echo -e "\n\nroot\nroot\n\n\nn\n\n " | mysql_secure_installation 2>/dev/null
        - shell: sudo pip3 install requests bs4
  

• After the packages are installed successfully, clone the git repository in which the webscrapper python script is present to the EC2 instance home directory.

    $ git clone https://github.com/amohan14/Webscrapping.git
  

• Run the Webscrapper python file and it shuld successfully outputs a .csv file containing all the reviews and their corresponding details.

    $ python3 Webscrapping/yelp_reviews_scrapping.py
  

To automate the above 2 points, we can add task to ansible playbook as follows:

tasks:
    - shell: git clone https://github.com/amohan14/Webscrapping.git        
    - shell: python3 Webscrapping/yelp_reviews_scrapping.py

You can find the complete Ansible Playbook