Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

User-friendly queryFunc() output #1483

Merged
merged 13 commits into from Jul 15, 2017

Conversation

Projects
None yet
4 participants
@WaLLy3K
Copy link
Collaborator

WaLLy3K commented May 19, 2017

By submitting this pull request, I confirm the following:

  • I have read and understood the contributors guide.
  • I have checked that another pull request for this purpose does not exist.
  • I have considered, and confirmed that this submission will be valuable to others.
  • I accept that this submission may not be used, and the pull request closed at the will of the maintainer.
  • I give this submission freely, and claim no ownership to its content.

How familiar are you with the codebase?:

10


Features:

  • User friendly query output, similar to master branch behaviour
  • Speedy query results (1.35s to perform pihole -q doubleclick.net -exact on 38MB of block lists compared to 1.33s on current dev branch and 7.59s on master)
  • -adlist option, to return the URL's of the block lists that contain the query
  • Positions of -exact and -adlist can be placed before or after the query
  • If a general query has over 10 matches in one file, the rest of the file is skipped
  • -all option, to remove match limit
  • pihole -q --help text
  • Grep errors are silenced

Tests:

  • pihole -q -h
  • pihole -q doubleclick.com
  • pihole -q -exact doubleclick.com
  • pihole -q doubleclick.com -exact
  • pihole -q doubleclick.com -adlist -exact
  • pihole -q <whitelisted domain>
  • pihole -q <blacklisted domain>
  • pihole -q <wildcard domain>

Output:

pihole -q -h

Usage: pihole -q [option] <domain>
Example: 'pihole -q -exact domain.com'
Query the adlists for a specified domain

Options:
  -adlist             Print the name of the block list URL
  -exact              Search the block lists for exact domain matches
  -all                Return all query matches within a block list
  -h, --help          Show this help dialog

pihole -q doubleclick.com

Result from list.5.v.firebog.net.domains
  doubleclick.com
  www.doubleclick.com
Result from list.14.v.firebog.net.domains
  ad-apac.doubleclick.com
  ad-emea.doubleclick.com
  ad.doubleclick.com
  ads.doubleclick.com
  advertisers.doubleclick.com
  advertisersapitest.doubleclick.com
  amazon.doubleclick.com
  betareportcentral.doubleclick.com
  click.comms.doubleclick.com
  doubleclick.com
  Over 10 results found, skipping rest of file
Result from list.52.v.firebog.net.domains
  doubleclick.com

pihole -q -exact doubleclick.com (and pihole -q doubleclick.com -exact)

Exact result in list.5.v.firebog.net.domains
Exact result in list.14.v.firebog.net.domains
Exact result in list.52.v.firebog.net.domains

pihole -q doubleclick.com -adlist -exact

Exact result in https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
Exact result in http://sysctl.org/cameleon/hosts
Exact result in https://hosts-file.net/ad_servers.txt

pihole -q <whitelist>

Result from whitelist.txt
  www.dshield.org
::: No results found for www.dshield.org found within block lists

pihole -q <blacklist>

Result from blacklist.txt
  fullstory.com
Result from list.3.v.firebog.net.domains
  fullstory.com
Result from list.15.v.firebog.net.domains
  fullstory.com

pihole -q <wildcard>

Result from whitelist.txt
  3879194.fls.doubleclick.net
Result from 03-pihole-wildcard.conf
  *.doubleclick.net
::: Over 1000 results found for doubleclick.net found within block lists

pihole -q <wildcard> -exact

Wildcard result in 03-pihole-wildcard.conf *.doubleclick.net
Exact result in list.1.v.firebog.net.domains
Exact result in list.5.v.firebog.net.domains

pihole -q -foo

::: No results found for -foo found within block lists
User-friendly queryFunc() output
* Silence grep errors
* Provide 'pihole -q -h' help output
* Rewrite option handling
* Loop through grep stdout to make query output user friendly
* Add -adlist option to show block list URL instead of internal file name
* Limit general searches to 10 matches per block list
* Add -all option to override 10 match limit
* Fixed 'pihole -h' wording

@WaLLy3K WaLLy3K requested a review from PromoFaux May 19, 2017

Further query optimisations
* Optimised scanList() output by switching folder
* Re-added processWildcards() function
* Added "-bp" exact matching option for use with block page
* Standardised query output
* Separated wildcard search from blacklist/whitelist search
* Optimised sorting by sorting glob output and not scanList() output

@WaLLy3K WaLLy3K changed the title [WIP] User-friendly queryFunc() output User-friendly queryFunc() output May 20, 2017

@WaLLy3K WaLLy3K changed the title User-friendly queryFunc() output [WIP] User-friendly queryFunc() output May 20, 2017

@WaLLy3K WaLLy3K changed the title [WIP] User-friendly queryFunc() output User-friendly queryFunc() output May 20, 2017

WaLLy3K added some commits May 20, 2017

Fix wildcard result output
* Multiple wildcard matches on exact query could cause unexpected output
@DL6ER

This comment has been minimized.

Copy link
Member

DL6ER commented May 20, 2017

Is this PR ready for reviewing?

@WaLLy3K

This comment has been minimized.

Copy link
Collaborator Author

WaLLy3K commented May 20, 2017

Yeah, it's good to go.

@DL6ER

This comment has been minimized.

Copy link
Member

DL6ER commented May 20, 2017

?

pi@raspberrypi:~ $ pihole -q abc
Result from list.0.raw.githubusercontent.com.domains
  0.0.0.0
Result from static.contentabc.com
  static.contentabc.com
Result from list.0.raw.githubusercontent.com.domains
  0.0.0.0
Result from abclauncher.com
  abclauncher.com
Result from list.0.raw.githubusercontent.com.domains
  0.0.0.0
Result from abctoppictures.net
  abctoppictures.net
Result from list.0.raw.githubusercontent.com.domains
  0.0.0.0

[...]

Result from [Disney
  [Disney
Result from Company][ABC
  Company][ABC
Result from News][INFOSEEK]
  News][INFOSEEK]

[...]

Result from vod.mobile-abc.com
  vod.mobile-abc.com
Result from list.6.hosts-file.net.domains
  127.0.0.1
Result from www.abcjmp.com
  www.abcjmp.com
Result from list.6.hosts-file.net.domains
  127.0.0.1
Result from www.mobile-abc.com
  www.mobile-abc.com

[ ... ] = I removed similar stuff for the sake of convenience

Make grep only output matching text
* HOSTS format lists will also output the IP address
* That substitution was necessary

@WaLLy3K WaLLy3K changed the title User-friendly queryFunc() output [WIP] User-friendly queryFunc() output May 20, 2017

@WaLLy3K WaLLy3K changed the title [WIP] User-friendly queryFunc() output User-friendly queryFunc() output May 20, 2017

Filter unwanted content
* Add /dev/null to grep, to always print file name (even when searching only one block list)
* Use three seds to remove unwanted content from block lists
@WaLLy3K

This comment has been minimized.

Copy link
Collaborator Author

WaLLy3K commented May 20, 2017

Latest commit should fix all known issues with content found in public block lists. Best that all the dumb things that I haven't thought to test against breaks during the review, and not after the merge 😄

pihole -q abc
Result from blacklist.txt
  collector.abc-test.net.au
Result from list.0.raw.githubusercontent.com.domains
  static.contentabc.com
  abclauncher.com
  abctoppictures.net
  abc.xyz
  www.abc7.pl
  abcdg.pro
  abcdespanol.com
  abcstats.com
  abc.doublegear.com
  adsatt.abcnews.starwave.com
  Over 10 results found, skipping rest of file
Result from list.1.mirror1.malwaredomains.com.domains
  prod-abc.ro
  abc.yuedea.com
  abcmlm.com
  abcommunication.it
  bb01abc4net.com
  labconnectlc.com
  abc-check.com
  abconstructions.us
  abcosecurity.ca
  cn.abcibankcard.com
  Over 10 results found, skipping rest of file
@DL6ER

This comment has been minimized.

Copy link
Member

DL6ER commented May 24, 2017

Setting this on hold for now (marking as [WIP])

@DL6ER DL6ER changed the title User-friendly queryFunc() output [WIP] User-friendly queryFunc() output May 24, 2017

@WaLLy3K WaLLy3K referenced this pull request May 24, 2017

Merged

Roll back merge #1417 #1494

5 of 5 tasks complete

@WaLLy3K WaLLy3K removed the Merge Conflict label Jun 21, 2017

@WaLLy3K WaLLy3K changed the title [WIP] User-friendly queryFunc() output User-friendly queryFunc() output Jun 21, 2017

@WaLLy3K WaLLy3K requested a review from DL6ER Jun 21, 2017

@PromoFaux

This comment has been minimized.

Copy link
Member

PromoFaux commented Jun 28, 2017

@WaLLy3K I just looked at fixing the conflicts on this, but I think it may be best if you do it, as you know your changes better!

@DL6ER

This comment has been minimized.

Copy link
Member

DL6ER commented Jun 29, 2017

@WaLLy3K What is the status of this PR?

WaLLy3K added some commits Jun 30, 2017

@WaLLy3K WaLLy3K removed the Merge Conflict label Jun 30, 2017

@PromoFaux

This comment has been minimized.

Copy link
Member

PromoFaux commented Jul 15, 2017

Approved

Approved with PullApprove

@PromoFaux
Copy link
Member

PromoFaux left a comment

My name is Adam Warner - and this is favourite merge on the citadel.

@PromoFaux PromoFaux merged commit 3a50b91 into development Jul 15, 2017

5 checks passed

codacy/pr Good work! A positive pull request.
Details
code-review/pullapprove Approved by PromoFaux, WaLLy3K
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
license/cla Contributor License Agreement is signed.
Details

@PromoFaux PromoFaux deleted the tweak/queryFunc2 branch Jul 15, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.