Skip to content

Latest commit

 

History

History
170 lines (137 loc) · 9.71 KB

README_selfsetup.md

File metadata and controls

170 lines (137 loc) · 9.71 KB
Raw

CV-Inspector Self Setup

This provides full instructions on how to set up CV-Inspector.

CV-Inspector was developed and used in the paper: CV-Inspector: Towards Automating Detection of Adblock Circumvention.

Visit our CV-Inspector Project page for more information, including datasets that we utilized in the paper.

Citation

If you create a publication (including web pages, papers published by a third party, and publicly available presentations) using CV-Inspector, please cite the corresponding paper as follows:

@inproceedings{le2021cvinspector,
  title={{CV-Inspector: Towards Automating Detection of Adblock Circumvention}},
  author={Le, Hieu and Markopoulou, Athina and Shafiq, Zubair},
  booktitle={The Network and Distributed System Security Symposium (NDSS)},
  url = {https://dx.doi.org/10.14722/ndss.2021.24055},
  doi = {10.14722/ndss.2021.24055},
  year={2021}
}

Contact

We also encourage you to provide us (athinagroupreleases@gmail.com) with a link to your publication. We use this information in reports to our funding agencies.

Setup Overview

In order to properly run CV-Inspector, your environment must be completely setup. This setup was tested using Amazon Machine Image: Ubuntu Server 18.04 LTS (HVM), SSD Volume Type

  1. Install all basic dependencies
    1. npm, mongodb, xvfb, Python 3.6+, python-pip, virtualenv
    2. Create a virtual environment for python3: we will call this cvinspector
      1. virtualenv --python=python3 [path_to_your_envs]/cvinspector
    3. Next see Install Google-Chrome 78
    4. Next see Install Chrome Driver 78
  2. Set up CV-Inspector Adblock Plus Chrome Extension
    1. Follow instructions from that repo to build the extension
    2. Keep track of where the devenv.chrome directory is within that project
  3. git clone CV-Inspector
    1. Install CV-Inspector python module: See Installing CV-Inspector
    2. Build chrome extensions: See Build CV-Inspector Chrome Extensions
  4. Install the necessary MongoDB collections: See Setup MongoDB
  5. Setup the chrome profiles you will need. CV-Inspector relies on two main cases: (A) No Adblocker, (B) With Adblocker. Along with those, we also can decide whether to consider the anti-cv list or only the easylist.
    1. See Setup Chrome Profiles
  6. Install npm packages for ad-block postprocessing.
    1. See Setup Ad-block Postprocessing

Your environment is now ready. Now proceed to Starting CV-Inspector

Installing CV-Inspector

  1. cd to the root project
  2. Activate virtualenv if necessary
    1. source [path_to_your_envs]/cvinspector/bin/activate
  3. As a development package: pip install -e .

Install Google-Chrome 78

CV-Inspector uses google-chrome 78. To make sure it is compatible, do the following:

  1. wget https://www.slimjet.com/chrome/download-chrome.php?file=files%2F78.0.3904.97%2Fgoogle-chrome-stable_current_amd64.deb -O google-chrome-stable_current_amd64.deb
    1. Taken from https://www.slimjet.com/chrome/google-chrome-old-version.php
  2. sudo dpkg -i google-chrome-stable_current_amd64.deb
  3. sudo apt-get -f install

Install Chrome Driver 78

  1. wget https://chromedriver.storage.googleapis.com/78.0.3904.105/chromedriver_linux64.zip
  2. unzip chromedriver_linux64.zip
  3. mv chromedriver [CV-Inspector]/chromedriver/chromedriver78

Build CV-Inspector Chrome Extensions

  1. cd to the root project
  2. cvinspector_buildextensions --extension_path chromeext
  3. if this fails, you may need to update your nodejs and npm versions
    1. To update nodejs:
       sudo npm cache clean -f
   sudo npm install -g n
   sudo n stable
    1. Then update npm: sudo npm install -g npm
    2. See if both are updated by looking at their versions:
       npm -v
   node -v

Setup MongoDB

  1. Install MongoDB
  2. Create the necessary db and collections:
    1. cd to the external_scripts/ directory of this project
    2. Open a new terminal and start the client side of mongodb: mongo
    3. Create a new db: use anticircumvention
    4. Add all necessary collections using script: load("setup_mongo_collections.js")
    5. Verify collections are there: show collections
    6. You are done: quit()

Adblock Plus Proxy

The proxy serves the version of EasyList and Anti-CV list that we want.

  1. Start a screen: screen -S abp_proxy
  2. Activate the cvinspector virtual env: source [path_to_your_envs]/cvinspector/bin/activate
  3. Go to the root of CV-Inspector: cvinspector_abp_proxy --filter_list_directory filter_lists

Setup Chrome Profiles

Automatically create four default profiles. For now, the chrome profiles are hardcoded to be in chromeprofiles within CV-Inspector root directory. First scenario: We need No Adblocker case and With Adblocker case (with EasyList only) Second scenario: We need No Adblocker case and With Adblocker case (with EasyList + Anti-CV list)

  1. Run the Adblock Plus proxy first: See Adblock Plus Proxy
  2. Then within a different screen, activate the cvinspector virtualenv
    1. source [path_to_your_envs]/cvinspector/bin/activate
  3. Run the command:

The below is an example of how you can call cvinspector_create_chrome_profiles.

   cvinspector_create_chrome_profiles --chrome_driver_path chromedriver/chromedriver78 --chrome_adblockplus_ext_abs_path /home/ubuntu/github/adblockpluschrome/devenv.chrome

Setup Ad-block Postprocessing

Unfortunately, we cannot use nodeJS v14 to build the ad-block package.

To build this AND run CV-Inspector, you must have nodeJS v10 (assuming our environment is what we described in Setup Overview).

  1. Purge what we have before: sudo apt-get purge nodejs npm
  2. curl -sL https://deb.nodesource.com/setup_10.x | sudo bash -
  3. sudo apt-get install -y nodejs
  4. Check that your npm -v should be 6 and node -v should be 10.
  5. Go to the root directory of CV-Inspector
  6. npm install --save fs path ad-block

Starting CV-Inspector

Your environment must be setup already. See Setup Overview

Note 1: If you already have the screen necessary, then re-use them. Note 2: You must have the node version as described in Setup Ad-block Postprocessing

A. We need to start the local proxy that serves the static list of easylist and anti-cv list.

  1. Start a new screen : screen -S abp_proxy
  2. Activate the virtualenv: source [path_to_your_envs]/cvinspector/bin/activate
  3. Go to root directory of CV-Inspector
  4. start the proxy: cvinspector_abp_proxy --filter_list_directory filter_lists
  5. Detach from the screen: CTRL+a then press d

B. Now we can run the script for CV-Inspector

  1. Start a new screen : screen -S cvinspector
  2. Activate the virtualenv: source [path_to_your_envs]/cvinspector/bin/activate
  3. Go to root directory of CV-Inspector
  4. The main script is cvinspector_monitor. It has many parameters to pass in, so use cvinspector_monitor --help if need be.
  5. While the script is running, you can detach from screen if necessary using CTRL+a then press d.

An example here: (this example can be used as is to run the example file misc_data/example_label_input.csv)

cvinspector_monitor --anticv_on False --trials 4 --beyond_landing_pages true --filter_list_paths filter_lists/easyprivacy.txt,filter_lists/disconnectme_abp.txt,filter_lists/getadmiral-domains.txt,filter_lists/antiadblockfilters.txt --classifier_path model/rf_model.sav --classifier_features_file_path model/features.txt --start_index 0 --end_index 2 --sites_csv misc_data/example_label_input.csv --output_directory /home/ubuntu/temp_output/detection_output/example_monitor/ --output_directory_ts /home/ubuntu/temp_output/detection_output/example_monitor_ts/ --output_suffix test_label --chrome_driver_path chromedriver/chromedriver78 --chrome_adblockplus_ext_abs_path /home/ubuntu/github/adblockpluschrome/devenv.chrome --by_rank false --log_level INFO

When the script finishes, it will print out where it outputs the last CSV with the label results. Use the cv_detect column from the CSV to know whether it predicted 0 = No Circumvention or 1 = Has Circumvention

Important parameters to notice:

  • --anticv_on: whether you want CV-Inspector to load the anti-cv list. If false, it will only rely on EasyList
  • --filter_list_paths: path to filterlists that you want to use to filter out traffic that you DO NOT care about
  • --sites_csv: the file that you want CV-Inspector to run on. An example is in misc_data/example_label_input.csv. Formatting must match that file
  • --start_index and --end_index: How many sites of the given file from --sites_csv do you want to crawl? For example, if the csv file has 100 sites and you only want to first test the first 10, then use --start_index 0 --end_index 10.
  • --output_directory: where the output will be
  • --beyond_landing_pages: if you want it to find a subpage to crawl as well.
  • --beyond_landing_pages_only: Given a URL, crawl an existing subpage only, while skipping the given URL.
  • --chrome_driver_path: Path to your chrome driver, this should be in chromedriver/chromedriver78
  • --chrome_adblockplus_ext_abs_path: Path to the CV-Inspector custom adblock plus. See Setup Overview