lists with outdated URL's... #7

dnmTX · 2018-08-21T05:45:47Z

@funilrys there are three lists so far that i've encounter which moved their repos to GitLab, very much active with at least one weekly update and here and in dead-hosts you still using the old URL's from GitHub which are outdated to say the least.If you don't mind look in to it.I'd rather download those lists from here but knowing that they're outdated is a big NO NO for me:

https://github.com/Ultimate-Hosts-Blacklist/quidsup_notrack_trackers
https://github.com/Ultimate-Hosts-Blacklist/quidsup_malicious-sites
New URL:
https://gitlab.com/quidsup/notrack-blocklists

https://github.com/Ultimate-Hosts-Blacklist/ZeroDot1_CoinBlockerLists
New URL:
https://gitlab.com/ZeroDot1/CoinBlockerLists

Thank you.

P.S. Another thought...on active lists like the one mentioned above or any other the domain.list needs to be updated more often,like at least once weekly(don't want to push it but twice weekly sounds even better 🙂)

The text was updated successfully, but these errors were encountered:

funilrys · 2018-08-22T15:37:44Z

Hi @dnmTX thanks for reporting will update the links asap.

About the domain.list, if the upstream link is update it is also updated...

Have a nice day/night.

dnmTX · 2018-08-22T18:50:11Z

About the domain.list, if the upstream link is update it is also updated...

@funilrys that is a nice feature,let's hope it's working as intended.I'll be checking on it to make sure it does once you update the URL's.Thank you.

P.S. For CoinBlockerLists please add HOSTS and HOSTS_BROWSER
Thank you !!!!

This patch update the link as mentioned by Ultimate-Hosts-Blacklist/dev-center#7

funilrys · 2018-08-25T17:38:40Z

Hi @dnmTX,
As referenced, this issue is partially fixed.

dnmTX · 2018-08-25T18:22:03Z

Thank you @funilrys .You know....on a second thought(as previously discussed with @ZeroDot1) many of the domains on his lists are going offline for a short time and then coming back to life(i guess some tactic of a sort).So i'd say you leave the dead hosts filtering to him,He does check and removes really dead ones but he checks them for a longer time before taking any decisions.
On the other hand(based so far on my observations),whoever maintains the quidsup lists he hardly ever removes any,just adding more and more.And considering how big the lists is i can't believe that they're all active,so those two lists need filtering for sure.

funilrys · 2018-08-25T20:20:59Z

Hi @dnmTX,

If we match them as INACTIVE we move them into inactive_db.json and after x days (defined by info.json but by default 1 day), we retest them and if they became ACTIVE, we include them into the list of ACTIVE which actually means clean.list.

mitchellkrogza · 2018-08-26T06:42:13Z

I think we need some logic in PyFunceble that if a domain was inactive once and became active again we mark it as suspicious and keep it on active?

dnmTX · 2018-09-07T16:20:56Z

@funilrys i think the clean.list in quidsup_malicious-sites is not updating.
Any info on when this commit will be merged or it's still in test? I still see the GitHub's URL's on both of them.

funilrys · 2018-09-09T13:51:17Z

Hi @dnmTX,
if there is no changes in the tests it's normal that it is not updating.

@mitchellkrogza

This patch fix #12. Explanation of the SUSPICIOUS status: This status is filled only if we delete an element which is VALID from the inactive database. Renaming: * Of the `HTTP_Analytic` directory to `Analytic`. * Of every analytic index in the configuration file. * Of every occurences of http_analytic in the code. * Of `Generate._analytic_file()` to `Generate.analytic_file()` Introduction: * Of the `SUSPICIOUS` index in the configuration file. * Of the `SUSPICIOUS` logic. * Of the automatic update of the directory structure file. * Of the link the psl database into the configuration file. * Of a safety check for the HTTP Code before we generate any types of file. Review: * of typo into README.rst * Of the way we construct the right analytic directory. * Of the way we test the `Database()` class. Update: * Of the version number. * Of dir_structure_production.json in order to fill the new structure. * Of the way we update the directory structure map from the configuration file. * Of the way we reference link to download. * Of the colomns documentation. * Of the configuration documentation. * Of the information documentation. * Of the `What can we do ?` section of the README.rst Deprecation: * Of all version which are less or equal to `0.102.0` Contributors: * @mitchellkrogza (#12 && Ultimate-Hosts-Blacklist/dev-center#7 (comment))

funilrys · 2018-09-21T20:58:53Z

https://github.com/Ultimate-Hosts-Blacklist/ZeroDot1_CoinBlockerLists_optional and https://github.com/Ultimate-Hosts-Blacklist/ZeroDot1_CoinBlockerLists_browser created.

Closing.

dnmTX · 2018-09-21T23:02:49Z

@funilrys consider changing CoinBlockerLists_optional with hosts. I think the optional one is the one that is more aggressive and have more false positives.The hosts one covers wider area and it gives you better protection then the browser one in general(i personally using it on my end).

P.S. You told @quidsup that you are informed of the changes but here the URL's are still from GitHub.What's up with that?

funilrys · 2018-09-22T07:08:06Z

In Detail @dnmTX.

Please report to

The system will do its work automatically and correctly once the repository will have an execution time from Travis CI.

Repository	Next session in
https://github.com/Ultimate-Hosts-Blacklist/quidsup_notrack_trackers	7h
https://github.com/Ultimate-Hosts-Blacklist/quidsup_malicious-sites	7h

I do not understand what you say.

Repo	Tested
https://github.com/Ultimate-Hosts-Blacklist/ZeroDot1_CoinBlockerLists	https://zerodot1.gitlab.io/CoinBlockerLists/list.txt
https://github.com/Ultimate-Hosts-Blacklist/ZeroDot1_CoinBlockerLists_browser	https://gitlab.com/ZeroDot1/CoinBlockerLists/raw/master/list_browser.txt
https://github.com/Ultimate-Hosts-Blacklist/ZeroDot1_CoinBlockerLists_optional	https://gitlab.com/ZeroDot1/CoinBlockerLists/raw/master/list_optional.txt

funilrys · 2018-09-22T07:27:16Z

Also, our whole organization (Ultimate-Hosts-Blacklist) - except for special repositories - has for purpose to test input source and provide the clean.list. The whole process is done automatically.

Understand that this organization is only the backend of the main repository mitchellkrogza/Ultimate.Hosts.Blacklist.

Everyday at T14:06:39Z the generation of the unified hosts file is started. What it does it getting the list of all repository of this organization from the GitHub API then get clean.list or domains.list if clean.list is not found.

As we need many IP to have consistent test for long list, we choose by design to use Travis CI which is great as it's offer a free usage for open repository. To understand why the repository are not updated right in the way, you have to understand that Travis CI only allow us to run 5 instances at the same time.

We are aware that our system provide some stress to the Travis CI team and infrastructure as our system is maybe one of those rare who run almost 24/24/7 that's why we accept those 5 instances limitation.

In order to allow bigger list to run stright away when they are under test, we implemented a code side "switch".
Indeed, as some upstream may not be update their list every day we also have the days_until_next_test in order to stop the system right before it test if the number of day < days_until_next_test.

Therefor if you find that days_until_next_test is not realistic for a repository let me know, I will check and update if needed.

I hope that this clarify what was misunderstood (if something was actually misunderstood).

Cheers,
Nissar

dnmTX · 2018-09-22T08:12:23Z

@funilrys this is what misled me:

Makes the impression that still the old URL's are in use.Obviously my mistake,sorry for the confusion,basically disregard my last post.Thank you for that in detail explanation.
Cheers!

funilrys · 2018-09-22T08:24:30Z

Fixed.

@mitchellkrogza

This patch fix #12. Explanation of the SUSPICIOUS status: This status is filled only if we delete an element which is VALID from the inactive database. Renaming: * Of the `HTTP_Analytic` directory to `Analytic`. * Of every analytic index in the configuration file. * Of every occurences of http_analytic in the code. * Of `Generate._analytic_file()` to `Generate.analytic_file()` Introduction: * Of the `SUSPICIOUS` index in the configuration file. * Of the `SUSPICIOUS` logic. * Of the automatic update of the directory structure file. * Of the link the psl database into the configuration file. * Of a safety check for the HTTP Code before we generate any types of file. Review: * of typo into README.rst * Of the way we construct the right analytic directory. * Of the way we test the `Database()` class. Update: * Of the version number. * Of dir_structure_production.json in order to fill the new structure. * Of the way we update the directory structure map from the configuration file. * Of the way we reference link to download. * Of the colomns documentation. * Of the configuration documentation. * Of the information documentation. * Of the `What can we do ?` section of the README.rst Deprecation: * Of all version which are less or equal to `0.102.0` Contributors: * @mitchellkrogza (#12 && Ultimate-Hosts-Blacklist/dev-center#7 (comment))

funilrys self-assigned this Aug 22, 2018

funilrys added bug Something isn't working enhancement New feature or request labels Aug 22, 2018

funilrys added a commit to Ultimate-Hosts-Blacklist/quidsup_notrack_trackers that referenced this issue Aug 25, 2018

Update of info.json

22adedf

This patch update the link as mentioned by Ultimate-Hosts-Blacklist/dev-center#7

funilrys added a commit to Ultimate-Hosts-Blacklist/quidsup_malicious-sites that referenced this issue Aug 25, 2018

Update of info.json

61ff4fa

This patch update the link as mentioned by Ultimate-Hosts-Blacklist/dev-center#7

funilrys mentioned this issue Aug 31, 2018

Suspicious domains funilrys/PyFunceble#12

Closed

funilrys closed this as completed Sep 21, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lists with outdated URL's... #7

lists with outdated URL's... #7

dnmTX commented Aug 21, 2018 •

edited

Loading

funilrys commented Aug 22, 2018

dnmTX commented Aug 22, 2018 •

edited

Loading

funilrys commented Aug 25, 2018

dnmTX commented Aug 25, 2018 •

edited

Loading

funilrys commented Aug 25, 2018

mitchellkrogza commented Aug 26, 2018

dnmTX commented Sep 7, 2018

funilrys commented Sep 9, 2018

funilrys commented Sep 21, 2018

dnmTX commented Sep 21, 2018 •

edited

Loading

funilrys commented Sep 22, 2018

funilrys commented Sep 22, 2018

dnmTX commented Sep 22, 2018

funilrys commented Sep 22, 2018

lists with outdated URL's... #7

lists with outdated URL's... #7

Comments

dnmTX commented Aug 21, 2018 • edited Loading

funilrys commented Aug 22, 2018

dnmTX commented Aug 22, 2018 • edited Loading

funilrys commented Aug 25, 2018

dnmTX commented Aug 25, 2018 • edited Loading

funilrys commented Aug 25, 2018

mitchellkrogza commented Aug 26, 2018

dnmTX commented Sep 7, 2018

funilrys commented Sep 9, 2018

funilrys commented Sep 21, 2018

dnmTX commented Sep 21, 2018 • edited Loading

funilrys commented Sep 22, 2018

funilrys commented Sep 22, 2018

dnmTX commented Sep 22, 2018

funilrys commented Sep 22, 2018

dnmTX commented Aug 21, 2018 •

edited

Loading

dnmTX commented Aug 22, 2018 •

edited

Loading

dnmTX commented Aug 25, 2018 •

edited

Loading

dnmTX commented Sep 21, 2018 •

edited

Loading