Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for local blocking lists #1779

Merged
merged 6 commits into from Nov 29, 2017

Conversation

Projects
None yet
4 participants
@DL6ER
Copy link
Member

DL6ER commented Nov 21, 2017

By submitting this pull request, I confirm the following (please check boxes, eg [X]) Failure to fill the template will close your PR:

Please submit all pull requests against the development branch. Failure to do so will delay or deny your request

  • I have read and understood the contributors guide.
  • I have checked that another pull request for this purpose does not exist.
  • I have considered, and confirmed that this submission will be valuable to others.
  • I accept that this submission may not be used, and the pull request closed at the will of the maintainer.
  • I give this submission freely, and claim no ownership to its content.

How familiar are you with the codebase?:

10


This PR adds support for local blocking lists using the file protocol (e.g. file:///home/pi/mylist.txt).

Exemplary adlists.list:

https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist
https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
https://hosts-file.net/ad_servers.txt
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
https://mirror1.malwaredomains.com/files/justdomains

file:///home/pi/test.list
file:///home/pi/test123.list

Note the two file:// lists at the end of the file. The first one exists, the second one doesn't exist. Running pihole -g produces the following output:

  [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range

  [i] Target: zeustracker.abuse.ch (blocklist.php?download=domainblocklist)
  [✓] Status: No changes detected

  [i] Target: s3.amazonaws.com (simple_tracking.txt)
  [✓] Status: No changes detected

  [i] Target: s3.amazonaws.com (simple_ad.txt)
  [✓] Status: No changes detected

  [i] Target: hosts-file.net (ad_servers.txt)
  [✓] Status: No changes detected

  [i] Target: raw.githubusercontent.com (hosts)
  [✓] Status: Retrieval successful

  [i] Target: mirror1.malwaredomains.com (justdomains)
  [✓] Status: No changes detected

  [i] Target: local (test.list)        <------------------- this file exists
  [✓] Status: Retrieval successful

  [i] Target: local (test123.list)     <------------------- this file doesn't exist
  [✗] Status: Not found / empty list
  [✗] List download failed: no cached list available

  [✓] Consolidating blocklists
  [✓] Extracting domains from blocklists
  [i] 109.715 domains being pulled in by gravity
  [✓] Removing duplicate domains
  [i] 100.825 unique domains trapped in the Event Horizon

  [✓] Adding 6 blocklist source domains to the whitelist
  [✓] Whitelisting 12 domains
  [i] Blacklisted 1 domain
  [i] Wildcard blocked 5 domains
  [✓] Parsing domains into hosts format
  [✓] Cleaning up stray matter

  [✓] DNS service is running
  [✓] Pi-hole blocking is Enabled

Querying also works fine:

$ cat /home/pi/test.list
0.0.0.0 testing.com
1.2.3.4 testing2.com

$ pihole -q testing2.com
 Match found in list.6.local.domains:
   testing2.com

This template was created based on the work of udemy-dl.

DL6ER added some commits Nov 21, 2017

Allow to add local lists to gravity using e.g. file:///path/to/my.lis…
…t in adlists.list

Signed-off-by: DL6ER <dl6er@dl6er.de>
Properly detect if local file was successfully downloaded
Signed-off-by: DL6ER <dl6er@dl6er.de>
Clarify error message: File could also have been of zero size
Signed-off-by: DL6ER <dl6er@dl6er.de>

@DL6ER DL6ER added the Enhancement label Nov 21, 2017

@DL6ER DL6ER requested a review from Nov 21, 2017

@DL6ER

This comment has been minimized.

Copy link
Member Author

DL6ER commented Nov 21, 2017

We also have a connected (although independent) Discourse feature request

gravity.sh Outdated
print $1
gsub(/(.*:\/\/|)/, "", $0)
if(length($1)>0){print $1}
else {print "local"}

This comment has been minimized.

@Mcat12

Mcat12 Nov 21, 2017

Member

Does this remove the username:password@ filtering?

gravity.sh Outdated
else
echo -e "${OVER} ${CROSS} ${str} Not found / empty list"
fi;;
* ) echo -e "${OVER} ${CROSS} ${str} ${url} ${httpCode}";;

This comment has been minimized.

@Mcat12

Mcat12 Nov 21, 2017

Member

If this last case is a the same level as http and file, then it should have some indentation removed to represent that.

DL6ER and others added some commits Nov 21, 2017

Repair username:password filtering
Signed-off-by: DL6ER <dl6er@dl6er.de>
Improve indentation
Signed-off-by: DL6ER <dl6er@dl6er.de>
@Mcat12

Mcat12 approved these changes Nov 21, 2017

@Mcat12 Mcat12 merged commit 472f772 into development Nov 29, 2017

5 checks passed

codacy/pr Good work! A positive pull request.
Details
code-review/pullapprove Approved by DL6ER, Mcat12
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
license/cla Contributor License Agreement is signed.
Details

@Mcat12 Mcat12 deleted the new/locallists branch Nov 29, 2017

@DL6ER

This comment has been minimized.

Copy link
Member Author

DL6ER commented Nov 29, 2017

@PromoFaux Can we add file:// entries via the web interface's settings page? If No, should the user be able to do this there?

@PromoFaux

This comment has been minimized.

Copy link
Member

PromoFaux commented Nov 30, 2017

OK, turns out you can add them, but they do not display on the web interface, therefore you cannot remove them. It also means that if you type it in twice, it will appear in adlists.list twice.

Actually, further testing shows that you can type anything you want into the text box on the admin and it will get added to adlists.list

@dschaper dschaper added this to the v3.2.1 milestone Dec 16, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.