This repository contains materials from the workshop Web Scraping for Economists. The aim is to equip economists with the skills needed to gather data from websites for analysis.
I am an economist. In all my papers, I have used web scraping to assemble unique datasets, enabling me to explore questions that would otherwise remain inaccessible. I believe web scraping provides a powerful tool for assembling datasets and uncovering new insights, particularly in areas where traditional data sources are limited.
I have provided workshops with practical takeaways from web scraping for economists and decided to share the content of those workshops here.
Here is a list of my research projects where I employed web scraping as a key tool for data collection. Feel free to explore these works for examples of practical applications:
- A New Order? Digital Disruption and Entrepreneurial Opportunities
- Immigration and Political Realignment
- Understanding Multi-Layered Sanctions: A Firm-Level Analysis
- From Streets to Ballots: The Impact of Climate Protests on Public Awareness and Electoral Outcomes (with Azhar Hussain and Ranjana Sinha)
While web scraping is a powerful tool, it comes with responsibilities and nuances that must not be overlooked:
- Legal Considerations: Always ensure compliance with website terms of service, intellectual property laws, and data privacy regulations.
- Ethical Concerns: Respect the integrity and intentions of the data owner and avoid harmful practices like overloading servers or scraping sensitive information.
- Practical Limitations: Web scraping can sometimes produce incomplete or biased datasets. Proper validation and supplementary data sources are often necessary.
- Technical Nuances: Web scraping requires familiarity with HTML, CSS, and website structures, as well as awareness of potential pitfalls like dynamic content or anti-bot mechanisms.
Although I have applied web scraping in a practical sense for my research, it is crucial to approach it with care. I strongly recommend consulting additional resources to deepen your understanding and ensure that your work adheres to the highest standards of legality and ethics.
I welcome your comments, suggestions, and questions! If you’d like to discuss web scraping or my research further, feel free to reach out. Collaboration and knowledge-sharing are integral to advancing the field, and I am happy to connect.
Happy scraping! 🚀