URLs Blacklist for Squid
Switch branches/tags
Nothing to show
Clone or download
Latest commit 464e1b5 Sep 12, 2018
Permalink
Failed to load latest commit information.
bw Update 09/12/2018 Sep 12, 2018
bwupdate Update 09/12/2018 Sep 12, 2018
LICENSE Update 10/21/2016 Oct 21, 2016
README.md Update 09/12/2018 Sep 12, 2018
blackweb.sh Update 08/13/2018 Aug 13, 2018

README.md

Blackweb

License GitHub version

Blackweb es un proyecto que pretende recopilar la mayor cantidad de listas negras públicas de dominios (para bloquear porno, descargas, drogas, malware, spyware, trackers, bots, redes sociales, warez, venta de armas, etc), con el objeto de unificarlas y hacerlas compatibles con Squid-Cache (Tested in v3.5.x). Para lograrlo, realizamos una depuración de urls, para evitar duplicados, dominios inválidos (validación de ccTLD, ccSLD, sTLD, uTLD, gSLD, gTLD, etc), y un filtrado con listas blancas de dominios (falsos positivos, como google, hotmail, yahoo, etc), para obtener una mega ACL, optimizada para Squid-Cache, libre de "overlapping domains" (e.g: "ERROR: '.sub.example.com' is a subdomain of '.example.com'").

Blackweb is a project that aims to collect as many public domain blacklists (to block porn, downloads, drugs, malware, spyware, trackers, Bots, social networks, warez, arms sales, etc.), in order to unify them and make them compatible with Squid-Cache (Tested in v3.5.x ). To do this, we perform a debugging of urls, to avoid duplicates, invalid domains (validation, ccTLD, ccSLD, sTLD, uTLD, gSLD, gTLD, etc), and filter with white lists of domains (false positives such as google , hotmail, yahoo, etc.), to get a mega ACL, optimized for Squid-Cache, free of overlapping domains (eg: "ERROR: '.sub.example.com' is a subdomain of '.example.com'").

FICHA TECNICA / DATA SHEET


ACL Black Domains Size
blackweb.txt 2.106.781 51,0 MB

DEPENDENCIAS / DEPENDENCIES


git squid bash tar zip wget subversion python

DESCARGA / DOWNLOAD


git clone --depth=1 https://github.com/maravento/blackweb.git

MODO DE USO / HOW TO USE


La ACL blackweb.txt ya viene optimizada para Squid-Cache. Descárguela con blackweb.sh. Por defecto, la ruta de blackweb.txt es /etc/acl. Ejemplo:

The ACL blackweb.txt is already optimized for Squid-Cache. Download it with blackweb.sh. By default, blackweb.txt path is /etc/acl. Example:

wget -q -N https://github.com/maravento/blackweb/raw/master/blackweb.sh && sudo chmod +x blackweb.sh && sudo ./blackweb.sh

ACTUALIZACIÓN / UPDATE


El script bwupdate.sh actualiza la ACL blackweb.txt, realizando la captura, depuración y limpieza de dominios, sin embargo puede generar conflíctos por errores en las fuentes, por tanto deberá depurarlos manualmente. Tenga en cuenta que este script consume gran cantidad de recursos de hardware durante el procesamiento y puede tomar mucho tiempo.

The bwupdate.sh script updates blackweb.txt ACL, doing the capture, debugging and cleaning of domains, however it can generate conflicts for errors in the sources, therefore you must manually debug conflicts. Keep in mind that this script consumes a lot of hardware resources during processing and it can take a long time.

wget -q -N https://github.com/maravento/blackweb/raw/master/bwupdate/bwupdate.sh && sudo chmod +x bwupdate.sh && sudo ./bwupdate.sh
Verifique la ejecución / Check execution

path: /var/log/syslog

Blackweb for Squid: Done 06/05/2017 15:47:14

Y verifique el contenido del archivo SquidError.txt en su escritorio para corregirlos / And check the contents of the SquidError.txt file on your desktop to fix them

Importante Antes de Usar / Important Before Use
  • Antes de utilizar bwupdate.sh debe activar la regla en Squid-Cache / You must activate the rule in Squid-Cache before using bwupdate.sh
  • La actualización debe ejecutarse en equipos de pruebas destinados para este propósito. Nunca en servidores en producción. / The update must run on test equipment designed for this purpose. Never on servers in production.
Extra ACLs

cloudsync.txt es una ACL contiene urls de servicios appcloud, diferentes a onedrive y gdrive (dropbox, pcloud, mega, etc.) y remoteurls.txt es una ACL contiene urls de servicios remotos (Teamviewer, Anydesk, Logmein, etc). Por defecto se excluyen de blackweb.txt. Para modificarlo, edite bwupdate.sh, y elimine las líneas referentes a estas ACLs (,cloudsync,remoteurls)

cloudsync.txt is an ACL containing appcloud service urls, different from onedrive and gdrive (dropbox, pcloud, mega, etc.) and remoteurls.txt is an ACL containing remote service urls (Teamviewer, Anydesk, Logmein, etc). By default they are excluded from blackweb.txt. To modify it, edit bwupdate.sh, and delete the lines referring to these ACLs (,cloudsync, remoteurls)

REGLA Squid-Cache / Squid-Cache RULE


Edit /etc/squid/squid.conf:

# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
acl blackweb dstdomain -i "/etc/acl/blackweb.txt"
http_access deny blackweb

EDICIÓN / EDITION


Blackweb contiene millones de dominios bloqueados, por tanto, editarla manualmente puede ser frustrante. Entonces, si detecta un falso positivo, utilice la ACL whitedomains.txt y reporte el incidente, para corregirlo en la próxima actualización. Lo mismo aplica para dominios no incluidos en Blackweb, que quiera bloquear, puede agregarlos en blackdomains.

Blackweb contains million domains blocked therefore manually editing can be frustrating. Then, if it detects a false positive, use the ACL whitedomains.txt and report the incident to correct it in the next update. The same applies for domains not included in Blackweb, you want to block, you can add them in blackdomains.

acl whitedomains dstdomain -i "/etc/acl/whitedomains.txt"
acl blackdomains dstdomain -i "/etc/acl/blackdomains.txt"
acl blackweb dstdomain -i "/etc/acl/blackweb.txt"
http_access allow whitedomains
http_access deny blackdomains
http_access deny blackweb

blackdomains.txt contiene dominios no incluidos en Blackweb (e.g. .youtube.com .googlevideo.com, .ytimg.com) y whitedomains.txt contiene el subdominio accounts.youtube.com desde Feb 2014, Google utiliza el subdominio accounts.youtube.com para autenticar sus servicios.

blackdomains.txt contains domains not included in Blackweb (e.g. .youtube.com .googlevideo.com, .ytimg.com) and whitedomains.txt contains subdomain accounts.youtube.com since February 2014, Google uses the accounts subdomain .youtube.com to authenticate their services.

FUENTES / SOURCES


URLs Blacklists

Shallalist

Capitole - Direction du Système d'Information (DSI)

MESD blacklists

Yoyo Serverlist

Oleksiig Blacklist

HP Hosts-file

Winhelp2002

Cibercrime-Tracker

Joewein Blacklist

Tracking-Addresses

Adaway

Lehigh Malwaredomains

Easylist for adblockplus

Zeus tracker

Malwaredomain Hosts List

Malware-domains

malc0de

BambenekConsulting

openphish

Tracking Disconnect.me

Carl Spam

Ultimate Hosts Blacklist

Hacked Malware Web Sites

Nginx Ultimate Bad Bot Blocker

Taz SpamDomains

The Big List of Hacked Malware Web Sites

StevenBlack Hosts

Matomo-org referrer-spam-blacklist

Quedlin blacklist

Joelotz URL Blacklist

dshield.org

Stopforumspam Toxic Domains

CHEF-KOCH BarbBlock-filter-list

URLs Blacklists (Discontinued or Replaced)

Passwall SpamAssassin (Server Down since Dec 2016). Last Update

UrlBlacklist (Server Down since July 24, 2017)

Badd-Boyz-Hosts, Someonewhocares and KADhosts. Replaced by StevenBlack Hosts

Web Miner

Anti-WebMiner

Ransomware

Ransomware Abuse

Ransomware Database

External Debugging (URLs/TLDs Whitelists, Invalid Domains, etc)

TLDs IANA

TLDs Mozilla Public Suffix

Wikipedia Top Level Domains

Whoisxmlapi GTLDs

ipv6-hosts (Partial)

O365IPAddresses (Partial)

University Domains and Names Data List

Central Repo Dead Domains

Internal Debugging (URLs/TLDs Whitelists, Invalid Domains, etc)

WhiteTLDs

WhiteURLs

RemoteURLs

CloudSync

BlackURLs

BlackTLDs

Debug (Common Errors)

Invalid Domains/TLDs

External Tools

Parse Domains (modified)

httpstatus

CTFR

Internal Tools

httpstatus bash

Debugging list

CONTRIBUCIONES / CONTRIBUTIONS


Agradecemos a todos aquellos que han contribuido a este proyecto. Los interesados pueden contribuir, enviándonos enlaces de nuevas listas, para ser incluidas en este proyecto / We thank all those who have contributed to this project. Those interested can contribute, sending us links of new lists, to be included in this project

DONACION / DONATE


BTC: 3M84UKpz8AwwPADiYGQjT9spPKCvbqm4Bc

LICENCIA / LICENCE


GPL-3.0

License maravento.com, gateproxy.com and dextroyer.com is licensed under a Creative Commons Reconocimiento-CompartirIgual 4.0 Internacional License.

© 2018 Maravento Studio

EXENCION DE RESPONSABILIDAD / DISCLAIMER


THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.