Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
base_xenforoforum adapter sites
Three sites are supported using base_xenforoforum adapters:
Forums are not really designed to host stories, but people use them that way anyway. When I started reading stories from the Worm fandom (an original web serial by Wildbow), the largest collection of stories I found for it was on a couple of near identical forum sites: SB & SV.
(QQ was later added by request because it is also nearly identical. QQ is the only one that supports user/password login to access its NSFW sections.)
The biggest problem with downloading stories posted to a forum is picking out the 'chapter' posts from all the other posts, since most stories on these sites combine chapters and reader comments. Plus, there's more than one way such stories are organized.
Supported story forms
I've identified and supported three different story forms:
Threadmarks First, a story thread can have 'Threadmarks' which link to specific posts. If a thread URL is given, and the thread has 2 or more Threadmarks, each Threadmarked post is considered a chapter--a number of stories I found had a single Threadmark for an 'Index' post. It's not uncommon to also have an Index post (see below), but Threadmarks take priority. You can give FanFicFare the full thread URL, but it will remember and use the 'normalized' URL without the thread title. Examples:
First Post Index Second, a story thread can have an 'Index' in the first post containing links to chapter posts. If a thread URL is given (and there are fewer than
minimum_threadmarksThreadmarks), it's assumed that the first post of the thread is the Index. As an added complication, sometimes the Index post also contains the first chapter. The first post/index is included as the first chapter titled 'First Post', unless it's also linked to explicitly in the post itself. Examples:
Post Index URL Third, a story can have an Index post that isn't the first post of the thread. If a specific post URL is given, that post is used as the Index and included as 'First Post' if not explicitly linked. Or you can give an Index post URL to explicitly not use the Threadmarks, for example if Omakes are listed in the index post but not in threadmarks. As with thread URLs, you can give a full post URL, but it will be normalized by FanFicFare. Examples:
One-shot Post A post URL can also be used to download a one-shot or single chapter story, as are common in the Discussion/Idea threads. But because they take the thread title by default, you'll have to set the title manually afterwards or setup an Individual Story Section. Examples:
In all cases, the title of the story is taken from the thread title and the author from the first post.
In all Index post cases, an Index post can contain links to chapter posts in other threads. Only links to posts on the same site will be included. Example: https://forums.spacebattles.com/threads/309577/
As of this writing (Oct 2016), SB has reinstated thread length limits, forcing arbitrary thread splits. I don't believe threadmarks can be used to point to posts in previous threads. So stories may be forced to abruptly split. Example: https://forums.spacebattles.com/threads/426988/
Unsupported story forms
In addition to the supported story forms, there are some common, but unsupported story forms:
No Index I've seen story threads that contain multiple chapter posts but lack either Threadmarks or an Index post. These stories are not supported because of the difficulty of distinguishing chapter posts from comment posts. Example:
Story Only (without Index) 'Story Only' threads aren't supported unless they have an index because FanFicFare has no way to know that it really is a Story Only thread. The potential for abuse (downloading lengthy discussion threads) is just too high to provide a version that can download all posts.
Normalized URLs FanFicFare 'normalizes' both the story URLs and chapter URLs for base_xenforoforum adapter sites. This is because the URLs as shown in your browser, which are meant to be more human-readable, can change if the thread title is changed or if a post is moved from one thread to another.
Update Notices I have yet to find an effective 'update notice' email mechanism for these sites. 'Watching' a thread works for story threads that have fallen dormant, but it's common for discussion (and derailments) to continue for weeks after the last chapter update. Email notice on OP update may work for you, depending on the activity of the thread.
Dates are a bit uncertain.
datePublishedis set to oldest Threadmark date, or posted date of the Index post.
dateUpdatedis set to newest Threadmark date, or updated date of the Index post.
numWordsare not set from site data. In the Calibre Plugin version, there are two options for counting words after downloading: FanFicFare Config->Basic->Calculate Word Count (preferred) and FanFicFare Config->Count Pages->Word Count (legacy). The common metadata
categoryare also empty. Instead,
tagsfromtitleare included in subject/tags.
In-line Post Index URL I don't have the example off hand, but I've seen at least one story that included the most up-to-date index in the same post as a later chapter without linking to it. FanFicFare has no way to detect that, which will place that chapter out of order in First Post.
Useful INI Setting for
There are a number of
[base_xenforoforum] specific INI settings and changes to common default settings.
strip_chapter_numbers:falseis explicitly set for
[base_xenforoforum]because of how many chapters labeled simply '1.1', '1.2', etc I saw. So even if you set
[defaults], it will be false for
[base_xenforoforum]unless you also set it in
Some fairly complex
replace_metadatamagic is included by default for
[base_xenforoforum]to find 'tags' in the thread title and include them as subject/tags and on the Title Page, but not in the
title. It also removes 'Thread 5' etc. This is instead of the usual
categorymetadata we collect from most sites.
Way of the Open Hand (Worm/D&D)becomes Title:
Way of the Open Handwith Tags:
SB & SV now include thread Tags, but not all threads use them. By default, those tags will be populated in the metadata field
forumtagsand included as subject/tags when the index was found in Threadmarks. You can set
always_use_forumtags:trueto use them without Threadmarks. By default,
forumtagsare automatically capitalized. Set
capitalize_forumtags:falseif you prefer the original lowercase.
descriptionis set from the content of the first post or Index post.
description_limit:500is set by default for
[base_xenforoforum]adapters because the first/Index post is often also the first chapter.
Even when using the first post as the chapter list, the first post/chapter will not be updated in the normal operation of the Update EPUB feature because Update assumes past chapters don't change. You can force it to be updated by setting
always_include_first_post:truecan be set to force inclusion of the first post. This is useful if there are Threadmarks, but the first post isn't threadmarked or if index post is not the first post of the thread but is linked in the post itself. There's a chance this can cause the post to be included more than once.
The title 'First Post' can be changed with INI parameter
The number of Threadmarks required to use threadmarks for chapters can be adjusted with INI parameter
Because base_xenforoforum adapters can pull chapter URLs from human written posts, the odds of errors in the chapter URLs are vastly higher than for most sites. Usually, a bad chapter URL will abort the entire download. You can set
continue_on_chapter_error:trueto continue after failing to download a chapter and instead record an error message in the ebook for that chapter.
truewill normalize all links in chapter texts, if they match chapter URLs. This will both fix partial URLs that wouldn't otherwise work and allow for the next setting:
truewill search all links in chapter texts and, if they match any included chapter URLs, replace them with links to the chapter in the download. Only works with epub and html output formats. base_xenforoforum adapters should also use
normalize_text_linkswith this. Now all those links in the index post correctly link to the chapters in your epub instead of going back to the site.
Settings per Individual Story
Given the diversity of forum stories, you may want to use different settings for different stories. This is possible using Individual Story URL Sections. Note that you should use Normalized URLs and match the thread or post URL exactly.
Here are some examples:
## Change story title, add categories. [https://forums.spacebattles.com/threads/426988/] replace_metadata: title=>.*=>The Battleshippening extracategories:Kancolle, Alternate Universe, Cute Battleships Doing Cute Battlethings ## Change story title and author, add categories. [https://forums.spacebattles.com/posts/20867703/] replace_metadata: title=>.*=>Belated Battleships: An Old Cat author=>.*=>Skywalker_T-65 authorUrl=>.*=>https://forums.spacebattles.com/members/skywalker_t-65.318921/ extracategories:Kancolle, Alternate Universe, Cute Battleships Doing Cute Battlethings ## Change First Post title, add tags. [https://forums.spacebattles.com/threads/346640/] always_include_first_post:true first_post_title:Prologue and Chapter 1 extratags: Original, Sci-Fi, AI ## replace_br_with_p for this story only. [https://forums.spacebattles.com/threads/384594/] replace_br_with_p:true