a.k.a. "Using Playwright to scrape pages that require you to fill out forms"
Jonathan Soma, Knight Chair in Data Journalism at Columbia Journalism School
Contact: jonathan.soma@gmail.com / @dangerscarf / jonathansoma.com
Playwright is a next-generation browser automation tool that allows you to use Python or JavaScript to scrape almost any web page. It can assist in downloading pages of government documents, capturing tweets before they get deleted, or simply breaking past the cookie consent banner. Beyond the basics, it can also easily take screenshots, monitor and log network requests, and even fit right into your traditional BeautifulSoup scraping approach.
Note for Windows: For some reason only one particular version of one particular thing works on Windows, so you need to run
pip install ipykernel==6.28.0before running Jupyter. The explanation is long and complicated. Also, maybe doing that breaks some things? It honestly isn't clear.
- Syllabus books: selectors, pagination
- Texas Tow Trucks Licenses: dropdowns
- Iowa Appraisal Management Companies: dropdowns, 'next page' buttons
- North Dakota Oil Wells: dropdowns, using every dropdown option
- Maryland Locksmiths: text boxes, inspecting pages, lists of inputs (zip codes), back button
- Texas Medical Board Actions Details: text fields, clicking links, downloading files, changing browsers, lists of inputs (license numbers)
If you'd like a general-purpose introduction to Playwright try this one, and if you want to know how to break CAPTCHAs, here you go!
The Playwright documentation is also pretty good.