A XenForo forum downloader written in Node.js:
- Scrapes content from forum pages
- For each thread, downloads attachments and saves messages in text files
- Supports downloading a single thread or all threads in a forum
- Supports continuing from previous download
Since the downloader works through scraping, it is not guaranteed to work with all XenForo forums. I created the downloader for my data-hoarding needs targeting a handful of sites, so it might be limited in what it can scrape. But feel free to raise issues.
First, install Node.js.
Then, in a terminal, run the following command:
npm i -g xenforo-dl
$ xenforo-dl [OPTION]... URL
Pattern: <forum_site_url>/threads/<title_slug>.<thread_id>[/page-<num>]
Download all messages and attachments shown on page. If content spans multiple pages, download from subsequent pages as well.
If page-<num>
is present in URL, then download will begin with the specified page.
Pattern: <forum_site_url>/forums/<title_slug>.<forum_id>[/page-<num>]
Download all threads listed on page. If the forum has threads spanning multiple pages, download from subsequent pages as well.
If page-<num>
is present in URL, then download will begin with the specified page.
For URLs not matching the above patterns, xenforo-dl
will scrape for forum links and download from them. It is your responsibility to ensure the given URL is a valid XenForo link.
Option | Description |
---|---|
-h , --help |
Display usage guide |
-k , --cookie |
(string) Cookie to set in requests. See Cookies. |
-o , --out-dir |
(string) Path of save directory. Default: current working directory. |
-d , --dir-structure |
Combination of flags controlling the output directory structure of downloaded threads:
Default: |
-w , --overwrite |
Overwrite existing attachment files |
-l , --log-level |
Log level: info , debug , warn or error ; set to none`` to disable logging. Default: info` |
-s , --log-file |
(string) Save logs to specified path |
-r , --max-retries |
(number) Maximum retry attempts when a download fails. Default: 3 |
-c , --max-concurrent |
(number) Maximum number of concurrent downloads for attachments. Default: 10 |
-p , --min-time-page |
(number) Minimum time, in milliseconds, to wait between page fetch requests. Default: 500 |
-i , --min-time-image |
(number) Minimum time, in milliseconds, to wait between download requests for attachments. Default: 200 |
--continue |
Continue from previous download |
-y , --no-prompt |
Do not prompt for confirmation to proceed |
Cookies allow you to download content that would otherwise be inaccessible due to lack of user credentials. To obtain a cookie for passing to xenforo-dl
through the --cookie
option, do the following:
- In a browser, sign in to the target forum site.
- Press
F12
to bring up Developer Tools. - Select
Network
tab, followed byHTML
filter. - Press
F5
to refresh the page. Select one of the entries that appear under theNetwork
tab. - Under
Headers
->Request Headers
, you should see theCookie
entry. Copy the value of that entry and pass it toxenforo-dl
.
Cookies should remain valid until they expire or you sign out of the forum site.
v1.0.0
- Initial release
MIT