Allow automating ClearUrl, redirection, query removal and pattern checking without user interaction by default.

[Rtizer-9]

I would like to start by saying this app is a dream come true; I so badly wanted these features in other similar apps but couldn't find any.

So, coming to this app the only major issue I find is that although in a way it saves time by giving you the original link instead of going through intermediate tracking links; the need for user interaction to click those modules one by one kinda counterbalance that saved time and even increases in almost 99% of cases when the redirection is clearly needed.

I wanted to do it myself but found it already implemented in one of your repository fork - https://github.com/Huhni/UrlChecker

• This might look like many "lines of code" changes but almost 90% of this already works with the apk I tried making with that fork repo changes.

What I think should be ideal is that;

• Redirection, ClearUrl, Query Removal and Pattern checking (especially HTTPS redirection) should be implemented by default without requiring user to press apply/check.

• All the modules should be triggered by default every time a redirection takes place arising the need of applying one or more modules one more time.

• Although the modifications in the fork repo already implements this amazingly it only does this once when in some cases its needed twice or more. Eg: Links in FB, instagram are first obfuscated with l/lm.facebook/instagram.com redirection and some cases even that link comes out to be trib.al which again needs a redirection. User interaction in such cases should not be required by default.

• Unlike the fork repo the history should allow viewing the originally clicked url in all the cases where the modules are applied automatically so that in problematic cases user can see the actual url and go back to it. I've personally found some links where redirection results in some weird links. If you think this is gonna cause problems to the average users then introduce a separate option to automate so that at least users who are comfortable can use it.

• Query removal should also include amp and lite which is common if you click links in google app. You can see them more clearly by using clicking through news articles in google app. It should include exceptions as the query removal in youtube links completely remove the part after v=.

• Allow apps set to open dedicated links by default like twitter, youtube like the fork repo has implemented for twitter links. It's kinda cumbersome to first click apply on all modules then choose which app you wanna open it with. Ideally default apps should be able to open links automatically just like they do before installing urlchecker and rest with the default browser.
  Again for the exceptional cases which can cause problem, a whitelist or a dedicated toggle can be an alternative.

Thanks.

[TrianguloY]

Thanks 😊, the idea of the app was to become some sort of firewall, giving you the option to analyze the url before doing anything, but it's true that it's becoming useful by itself and even though I still think the automatic redirections should be off by default, giving the user an option to enable them is needed.

That fork does have some of the features, but they are hardcoded and isn't as easy as just copy/paste, a good solution should have a toggle in every module (where it makes sense) to enable the automatic redirection, which should happen in the modules order (next version allows to reorder them) until no module 'fires' a new url, testing for loops just in case. To be honest that work is partially done, it just needs to be completed (currently if a module tries to change the url while it is being analyzed it is just ignored, but that can be changed). And of course allowing to not fire it if needed (the history module will need this flag, so that going back doesn't trigger it).

The autoOpen for some apps is also a good idea, but different to the automatic redirections feature. Mind to open a separate issue for that?

[Rtizer-9]

Great, I completely agree about turning automation off by default as I'd already mentioned above that a large part of users can get affected if the url is problematic (google drive/account authentication/download url is one example). The toggle option as you said will be perfect.

I'm not sure if I understood the hardcoded modification thing correctly; if you are saying they aren't exactly implemented the way you just described above then I get it otherwise I've personally downloaded the latest git code of your repo and implemented those changes in studio by just copy-pasting and they works for my taste at the moment. I was surprised by that module order thing - great feature I didn't think about. It works perfectly in the apk I build. I can provide it if you want just to see the behavior or just for the sake of it. Obviously you can do it already, you wrote the code after all (I'm just saying if you want) .I'm hesitating to upload it already and provide a url because it's sort of your copyright material.

I'll open another issue for autoOpen as you said.

[TrianguloY]

Yes, I meant that the changes are directly made without testing (if the modules are disabled the app will crash, for example). A proper solution requires more code, but will also be more customizable and allow both the old and new features.

[TrianguloY]

The next version has ClearUrl automatic option (disabled by default). Not happy how I ended implementing it, but for now it should be enough. I didn't add the other modules yet, but they will probably be implemented in the future.

I keep this issue open for future discussion.

[ig9000]

I found this issue to be similar to what I came here to suggest, so I'm going to present that here, but will happily open a new issue if it's not on-topic enough.

My thought after some use of this (much appreciated) app was that it could use a whitelisting system to stay out of the way for known cases where no interaction is desired. Basically, if URL matches an expression, take the configured actions without any prompting. Further actions could include processing by modules or not, then handing over the URL to the configured app. This would actually be a great alternative to the (not at all flexible) native "default app" functionality in Android.

Hope that makes sense, and thanks for making this!

[TrianguloY]

Hmm, I think that's better as a new issue, this one is more for automatically apply existing modules (something partially implemented).

In fact what you are suggesting is probably very specific and maybe hard to implement with the 'modular' design (not saying it is impossible, just that it will probably require very different logic).
In any case I think #18 is a very similar and probably easier to implement similar thing. Feel free to comment there or create a new issue in any case.

[TrianguloY]

From the original post suggestions: the only modules that now doesn't have automatic options are Status Code and Queries remover.

Queries remover is just extremely dangerous, even as opt-in I don't think it is a very good idea, parameters in urls are used a lot so those that enable it will probably need to 'undo' it more often than not. Solutions like ClearURL that removes only the problematic ones are better. Feel free to correct me if I'm not right.

The automatic status code is suggested on #25.

For these reasons I'm closing this issue. If there was a suggestion I missed or you want to discuss some of the decisions, don't hesitate to write a comment, or even create a new issue if required.

[Ilithy]

Is it possible (in the future) that the "Url Scanner" module can benefit from this automation option?

This module could very well be applied automatically it seems to me (I mean without risk of causing problems, like the "Queries Remover")

[TrianguloY]

From a code perspective it could be implemented as "pressing the button" when the url changes, although it is a background process so it requires some more checking to avoid issues with concurrency (maybe even a refactor to avoid checking urls that will change from other modules instantaneously).

From a user perspective, even if it is implemented, I would recommend not to enable it. That will mean that every url is sent to virustotal, which according to it's privacy policy means that all those urls can be accessed by others. If you click a link on an email and there are some private parameters, those will be sent.

In any case, an option (with a warning) is possible and it's a valid and planned feature to add.

[Ilithy]

From a user perspective, even if it is implemented, I would recommend not to enable it. That will mean that every url is sent to virustotal, which according to it's privacy policy means that all those urls can be accessed by others. If you click a link on an email and there are some private parameters, those will be sent.

I hadn't seen it that way, it's an excellent point, this makes me doubt the usefulness of automating the scanning of urls...

[Ilithy]

From the original post suggestions: the only modules that now doesn't have automatic options are Status Code and Queries remover.

Not quite, the Patterns Checker module was forgotten 😉

Would it be possible to add automation for the Patterns Checker module please?
This would be very practical, whether to correct the lack of security layer of the Hypertext protocol or to better exploit new ability to perform privacy redirections (it is extremely practical, thank you @TrianguloY and @Raj9039852537 for this very nice addition ❤️).
Many thanks 🙏

[Raj9039852537]

Pattern checker already support automatic redirection.
see code available at wiki (gist link) there is one entry called wikiless fix and it have that feature enabled.
what it does is add language code to pages that does not have that already.

for example
https://wikipedia.org/wiki/Clearnet_(networking)
converted to
https://farside.link/wikiless/wiki/Clearnet_(networking)?lang=
Should be
https://farside.link/wikiless/wiki/Clearnet_(networking)?lang=en
This link doesn't start with lang code so after changing it to wikieds URL there is no language code with URL that can be mitigated with adding "en" at the end since the previous regex have no item at lang code position.

"Wikipedia": { "regex": "^https?:\\\/\\\/([a-z0-9-])*?(\\.)?wikipedia\\.org\\\/(.*)", "replacement": "https:\/\/farside.link\/wikiless\/$3?lang=$1", "enabled": "true" }, "Wikiless fix": { "regex": "^https?:\\\/\\\/farside\\.link\\\/wikiless(.*)=$", "replacement": "https:\/\/farside.link\/wikiless$1=en", "automatic": "true", "enabled": "true" }