All URLs in the Global List Should be Active URLs.

[jakubd]

I realize that there is an intention to keep some URLs that are dead (404ing, parked pages,etc) on the local lists because this repo is intended to measure censorship and this may sometimes last longer than the site itself. I believe that this logic does not hold as well when talking about the global list. Is there any objection to checking the consistency of the global list and making sure that all the global list URLs are active non-dead URLs?

[sneft]

I agree. Given the frequency with which global list URLs are tested there's an extra incentive to keep it fresh, and knowing the history of that list I'm certain there's some redundant URLs.

At the same time I don't think it needs to be constantly updated - maybe there could be a monthly or quarterly check for dead URLs? It would also probably be useful to have a bit of sanity checking before removing, to prevent removing a URL as a result of a short-term technical issue.

[hellais]

This PR has a discussion that is also relevant to this topic I think: #127

[bact]

What about the update of HTTP addresses with their HTTPS ones? (if they are automatically redirected) .. should we keep the HTTP ones as well, or shouldn't we?

• This may not applied well for country list, as censorship can go specifically with HTTP address and not HTTPS (and vice versa).
• But for the global list discussed here, I think it could be relevant.

Examples are those Wikipedia links.