HTML Souce - Lack of Docs! Help!

[DiagonalArg]

I'm having some trouble figuring out the options for HTML sources. For example:

1. Does the "Sort by file names instead of full links" switch choose between the "Custom APK Link Filter by Regular Expression" and the "Filter APKs by Regular Expression"?

2. How does "Filter for an Intermediate Link to Visit First" work?

3. And how do I use "Version Extraction"?

4. What does "Match Group to use" refer to?

5. Why are the choices for Version Detection greyed out other than "No version detection"?

My immediate concern is the 3rd question for Linphone, where the version is part of the filename.

[ImranR98]

Yes, some settings can be confusing - I'm hoping to flesh out the Wiki page at some point.

Note: you should know what Regular Expressions are before using any of the filtering options.

The HTML Source works in this way:

1. It looks for links on an HTML page.
2. It filters out certain links. By default, this is any link that doesn't end in .apk but you can use "Custom APK Link Filter" to specify your own filter.
3. It sorts the remaining links. This sorting is alphanumeric sorting on the whole link, but you can choose to sort by only the last segment of the link. This last segment is usually the filename but may not be if you used your own filter in step 2. It really should be called "Sort by last segment of link" or something.
4. It applies yet another optional filter on all remaining links. I'm not sure there's a good use case for this, but this setting ("Filter APKs by Regular Expression") is a more general filter that all Sources have - it is inherited from the parent AppSource class. It's probably easier to use this in some situations rather than having a super complicated filter in step 2.
5. Of the remaining links, it picks the first one (or the last one if you enabled the reverse option).
6. Now that we have the final APK link, we need a unique release ID to go with it so that when the ID changes, we know the app has an update available. For other Sources, the unique release ID is the app version, but for the HTML Source, it might not be possible to extract a version string. So by default, this is just a hash of the link.
   • However, links often have version strings embedded within them. Obtainium can't know how to extract these on its own (different websites would have different ways of doing it), so the user can choose to specify a regular expression that can be applied to the link in order to extract the version - this is what the "Version Extraction" field is for.
   • But often it can be difficult to come up with a regular expression that accurately matches the version while excluding extra characters. For this, we have the "Match Group" option that lets the user specify which group in the regular expression we should use as a version.
   • The version extraction feature isn't really necessary - using link hashes is easier and more reliable. Some users might just want it (Add new version detection using REGEX for HTML sources #861) as having the real version looks nicer/more accurate and it allows Obtainium, in most cases, to use version detection.

As for the "Intermediate Link" filter, if this is used, the HTML Source works as follows (see #820 for a situation where this is useful):

1. It looks for links on an HTML page.
2. It filters out any links that don't match the intermediate link filter.
3. It grabs the first remaining link (no reverse option here), and then feeds that link as the input for the normal HTML Source process described previously.

[ImranR98]

So for Linphone, we have a bunch of different links on the page:

• linphone-android-x.y.z.apk
• linphone-freecodecs-video-x.y.z.apk
• linphone-nonfreecodecs-video-x.y.z.apk
• maven_repository/

So:

1. We want to get rid of all links that are not of the form linphone-android-x.y.z.apk.
   • We could do all this in a single regular expression in "Custom APK Link Filter" like this: linphone-android-.*\.apk
   • But having 2 separate filters makes it easier:
     • The default APK link filter already filters out anything not ending in .apk (in this case, maven_repository/)
     • We can then use a simpler second APK filter to get rid of the APKs we don't want: linphone-android
2. We could just stop here.
   • No need for an intermediate link filter (we are already on the page with the final APKs on it).
   • No real need for version extraction either.
3. If you do want version extraction, you can:
   • Use this regex in the "Version Extraction" filter: android-(.*)\.apk
     • Set "Match Group" to the default of (1). In the above regex, the first group is .* (each group is in brackets).
   • If you are sure that the version is always 3 sets of digits separated by dots, and that this pattern will never appear more than once in the link, you could avoid using groups entirely by using this regex with "Match Group" set to 0 (meaning "use the whole match"): [0-9]+\.[0-9]+\.[0-9]+

[ImranR98]

Verified that the above instructions work. In the process I found a couple of bugs that will be fixed in the next release. Namely that the match group needs to explicitly be input by the user (the default value doesn't work), and that it's better to have a default of 0 than 1.

[DiagonalArg]

Fantastic! Thank you so much for your thorough explanation. (And for your work on this useful piece of software.)