Download LIHKG threads (aka posts) and pages in json format.
The approach is selenium will connect to the local Chrome browser via Chrome DevTools Protocol and "overheard" the traffic between the browser and the LIHKG server, so that the very same API response which is used to render the LIHKG web app could be seen by selenium. Image urls will be parsed at the end of the script and downloaded separately.
- Install required packages using
pip install -r requirements.txt
- Install Google Chrome (if not yet installed)
- (Optional) To scrape member-only contents (eg. threads in 成人台 / 留言嘅加密部分), log in to LIHKG in Google Chrome
- (Optional) To override the default config properties defined in
config/config.default.ini
, create a copy of it and rename the new file toconfig/config.ini
, and edit the values in the new file
- Quit Google Chrome if it is opened
- Open Google Chrome with
--remote-debugging-port=9222
passed into the command line arguments.- The port
9222
may be changed to other available ports. Please also editdebugger_address
in the config file if another port is used
- The port
- Run
python scrape.py -t <thread-id> [-p <page-numbers>] [-o <output-folder>] [--remove-me]
-t <thread-id>
is required-p <page-numbers>
is optional, will scrape all pages in the thread if not provided
Examples:-p 1
/-p 1-5,8,11-13
-o <output-folder>
is optional, will save the output tooutput
if not provided--remove-me
is optional, will remove the logged in user data (if any) from the output
To view the output in a human-readable manner, use the web app LIHKG Snapshot Viewer (repository: lihkg-snapshot-viewer)