Skip to content
Code used for collecting and saving the Top Stories and Trending Stories in Apple News via Appium.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.
data_from_paper Delete .DS_Store Aug 6, 2019
LICENSE Fresh commit Aug 6, 2019 Update Aug 6, 2019 Fresh commit Aug 6, 2019

Apple News Scraper

This repository provides code and data used in the following paper:

Bandy, Jack and Nicholas Diakopoulos. "Auditing News Curation Systems: A Case Study Examining Algorithmic and Editorial Logic in Apple News." To Appear in Proceedings of the Fourteenth International AAAI Conference on Web and Social Media (ICWSM 2020).

Installation and Setup Instructions

Install Appium

Download appium-desktop:

(You can try the brew/npm installation - - but those releases have been buggier in my experience)

Install apple-news-scraper

After cloning this repository onto your computer,

  1. Run instruments -s devices in your terminal
  2. Choose a device, something like iPhone XS Max (12.1) [XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX] (Simulator)
  3. Open and replace the first few lines with your device information. Afterwards, it may look something like:
# user-defined variables
device_name_and_os = 'iPhone XS Max (12.1)'
device_os = '12.1'
  1. Change the output folder, where you want to save the data:
# easy relative path, keep data in repository
output_folder = 'data_output/'


# put data in a folder on the desktop
output_folder = '~/apple_news_data/'


Execution should be as easy as python

To run repeatedly, I recommend cron. Just make sure you use absolute paths. For example, to run collection every five minutes, add something like this to your crontab:

*/5 * * * * /usr/local/bin/python /Users/jack/dev/apple-news-scraper/

If you're in a hurry, you can also just hack out a shell script:

while true
        python &
        sleep 300
You can’t perform that action at this time.