Last Run Stats
- Script Started Mon Jun 17 00:20:17 UTC 2019
- Script Ended Mon Jun 17 00:34:04 UTC 2019
- Script Took 13 Minutes To Filter 128 Lists. See Log Here.
- The Edited AllParsed File is 61 MB And Contains 2746030 Domains.
- Average Parsing Of 14435 lines in 4 BlackLists was 23 Seconds.
- 1742 Valid Top Level Domains. No New TLD's.
- 14 Lists That Do NOT Use https
Other aims of this project:
- Lists update daily if there are any changes.
- Build a user-driven blacklist.
- Build a user-driven whitelist.
- Mirror and Filter, any user-submitted blacklist.
- Handle ANY list, even if it is compressed.
Because people ask often:
I run this alongside several other things on a Raspberry Pi 3.
- This is THE ONLY list I use, with nothing whitelisted.
- Because of this, If there is something blocked that shouldn't be, my wife and I would know.
- Individual lists tend to be safer than all of them Combined.
- You will find them Within the "Subscribable-Lists" directory.
- There are now Country Specific Lists!
Simply copy the RAW format url for the list and add them.
- In the Web Interface on the Settings page.
- Directly to your adlists.list file found in /etc/pihole/.
All of the lists combined.
- Note, I honestly don't recomend adding the big list, it may break websites.
I also have a list that is driven by the userbase.
To request a list to be whitelisted or blacklisted, please submit an issue containing WHY it should be added or removed.
IF YOU ARE NEW TO LINUX AND PI-HOLE, CONSIDER ADDING THE LISTS I HAVE ALREADY PARSED
- I'm already parsing all of the lists daily and uploading them to the parsed directory in this repository.
- If you prefer to use this project yourself locally, Keep Reading.
Caution: The Script Has Evolved to the point that it runs other analytical tasks that add time to the process. You have been warned.
IF YOU HAVE USED THIS SCRIPT BEFORE 8/24/2017, YOU WILL NEED TO RUN/RERUN THIS (updated) INSTALLER
sudo wget https://raw.githubusercontent.com/deathbybandaid/piholeparser/master/piholeparserinstall.sh sudo bash piholeparserinstall.sh
sudo bash /etc/updaterunpiholeparser.sh
A local copy of the big list is available to be used.
Query Lists Tool
There is a querylists.sh within the scripts directory.
This will allow me to query the individual parsed files for a specific domain.
There is a Log Available
This should provide some insights as to what lists are dead, empty, or too large for github.
A pun on Pi-hole's gravity.sh, this tool allows me to see what domains are on my list versus gravity.list
Basic Things about this script
- Script updates first thing on every run, always the most up to date version.
- Script Checks for dependencies.
- .lst files are named on purpose to help name the end results better.
- Script skips steps if the file is empty
- Script skips IP Lists (for now)
- Script appends RecentRunLog to tell me that a list is no longer dead.
- Script Pushes the results to localhost, and Github (if selected).
- Script runs daily with cron, or Manually.
- Allparsed list is based on the userbase.
- Checks to see if host of list is available.
- Checks to see if a list was updated online.
- Download based on host availability, file extension (tar or 7z), or attempt to use a mirrored copy from this repository.
- Creates a mirror if file is not empty, or over the Github 100MB limit.
- Remove Commented lines #'s !'s and Empty Lines.
- Remove Invalid Characters. FQDN's are allowed to use dashes, underscores, and emoji's. all other symbols are not allowed.
- Remove Pipes | and Carots ^
- Removes IP Addresses.
- Remove Empty Space.
- Checks for FQDN Requirements. A Period and a Letter.
- Remove Periods at Beginning and End Of Lines.
- Filter out common file extensions used in assets
- Reverse Searches Top Level Domains
- Remove Duplicates, If any.
- Create Parsed File, if it survives this process.
- This will take all the small lists and merge them.
- I then take that list, add user-submitted blacklists, remove user-submitted whitelists, and produce another Big List.
- I take the Big List and generate small lists based on Country Codes.
All "Original Unaltered Lists" are located within the mirroredlists directory. After going through the parser, many lists contain zero lines and are deleted. The filtered lists are in the parsed directory; with filenames to reflect the Original Creators work/effort.