Whitelist Being Ignored completely (2.9.2 issue on Ubuntu?) #775

sam24and48 · 2016-10-10T15:44:25Z

This template was created based on the work of udemy-dl.

In raising this issue, I confirm the following (please check boxes, eg [X]):

I have read and understood the contributors guide.
The issue I am reporting can be replicated
The issue I'm reporting isn't a duplicate (see FAQs, closed issues, and open issues).

How familiar are you with the codebase?:

[FEATURE REQUEST | QUESTION | OTHER]:

{replace this section with your content or delete if not a FEATURE REQUEST/QUESTION/OTHER}

[BUG | ISSUE] Expected Behaviour: I have added multiple addresses including msftncsi.com, www.msftncsi.com, support.microsoft.com, and msn.com. We performed this last week and it fixed the issue of limited connectivity on our machines and also the ability to get to pages on support.microsoft.com and confirmed connectivity to msftncsi.com. (However today after the weekend and the 2.9.2 update it is pi-holing all of those addresses)

[BUG | ISSUE] Actual Behaviour: It appears to be completely ignoring the whitelist entirely. Today as I have done nslookups on the machines for all of those addresses they are pointing directly to the pi-hole ip. I have rebooted the pi-hole device multiple times along with dnsflushes on the machines multiple times. I can see in the web admin interface where it is flat out pi-holing the address when I go to it even though it is specifically added to the whitelist (example: support.microsoft.com).

[BUG | ISSUE] Steps to reproduce:

(Optional) Debug Log generated by pihole -d:

http://termbin.com/s1lw

The text was updated successfully, but these errors were encountered:

dschaper · 2016-10-10T15:48:23Z

Hi, how are you adding the domains to the whitelist? Via commandline or via the web interface?

dschaper · 2016-10-10T15:50:09Z

And I do show that support is blacklisted on one of the lists:

::: /etc/pihole/list.7.raw.githubusercontent.com.domains (2 results)
diagnostics.support.microsoft.com #Microsoft
support.microsoft.com #Microsoft

sam24and48 · 2016-10-10T16:03:20Z

I added it via the web interface

Yeah i saw that too, not sure it should be but that's a discussion for another day. The weird thing is I know whitelisted sites were working Friday so I'm not sure outside of the update possibly what changed it.

dschaper · 2016-10-10T16:04:12Z

Okay, I'd like to move this issue over to the Web Interface repository for further review. Better venue to handle the questions.

dschaper · 2016-10-10T16:04:21Z

Issue moved to pi-hole/AdminLTE #159 via ZenHub

sam24and48 · 2016-10-12T11:44:31Z

I am having this issue again, is there any way to reopen this case as I was instructed to do if it re-appeared via the referenced issue above?

It appears to be completely ignoring the whitelist entries again and I have to run pihole -g about 10 times before it starts working correctly. In the meantime while I do that they domains being blocked statistic on the web interface goes from 105k to 790, 340, 1 and then back up when it is fixed after running the -g command a ton of times. Any insight on what might be going on? My users are getting frustrated they can't see msn.com in the morning (but maybe that's a good thing :) )

PromoFaux · 2016-10-12T11:50:23Z

Could you run off another pihole -d for us to take a look at? Thanks!

sam24and48 · 2016-10-12T12:00:57Z

Ok it has been uploaded to http://termbin.com/d4yd

As another note I have run pihole -g multiple times and did a reboot after it looked fine and only got msn.com back. support.microsoft.com and www.msftncsi.com are still showing dns resolution to the pi-hole server. I did do multiple dns flush commands on the windows machines as well.

PromoFaux · 2016-10-12T12:15:49Z

OK, let's try rolling back to 2.9.1 and seeing if you still get the issue:

cd /etc/.pihole
sudo git checkout V2.9.1
./automated\ install/basic-install.sh (choose the update option to replace the files, or install to go through the options again)

sam24and48 · 2016-10-12T12:39:20Z

Ok, I have two pi-holes running right now both on different ips. I did this to the second one which is really only used in the case of a failure of the first one. I noticed after rolling back and doing a pihole -g that the domains blocked is 44k whereas the 2.9.2 primary which I have not rolled back yet is around 105k does that sound right?

PromoFaux · 2016-10-12T12:41:05Z

what does cat /etc/pihole/gravity.list | wc -l give you on the 2.9.1 install?

sam24and48 · 2016-10-12T12:47:25Z

sam24and48 · 2016-10-12T14:40:39Z

I just updated on the first one as well now and ran the same command and am showing significantly less. the web admin shows only 20k domains being blocked.

PromoFaux · 2016-10-12T14:43:47Z

Hmm, could you pastebin /etc/pihole/install.log for each please?

sam24and48 · 2016-10-12T14:51:49Z

Here's the one with 41052

PromoFaux · 2016-10-12T14:53:37Z

Interesting, that one shoes 0 domains....

sam24and48 · 2016-10-12T14:55:11Z

Here's the one with 95551

sam24and48 · 2016-10-12T14:55:38Z

I also ran pihole -g after each one as well I believe a couple times.

sam24and48 · 2016-10-12T14:57:33Z

Here is the output from pihole -g on the one that originally showed no domains.

sam24and48 · 2016-10-12T14:58:36Z

My word, now that I just ran it for that output it is up to 47k like the other one.... not sure why it was so low before...

PromoFaux · 2016-10-12T14:59:37Z

Ok, 2.9.1 seems to only pull in ~47k domains (Not sure why, I don't remember it happening at the time)

I am unable to reproduce the whitelist being ignored, however, in both 2.9.1 and 2.9.2

sam24and48 · 2016-10-12T15:00:00Z

sam24and48 · 2016-10-12T15:02:15Z

So far I'm not having any issues with the whitelist being in 2.9.1 however it was fine yesterday after running the pihole -g a few times after letting it sit. Then this morning it was broke again. So at this point I'll let it sit the rest of today and will check in tomorrow and see if we get a replication of the issue. At least by Friday we should have an idea if it was something I was having an issue with in 2.9.2 or not

Maybe then I can upgrade the secondary one again and we can test further with the primary on the "stable" version in my case.

PromoFaux · 2016-10-12T15:03:13Z

Hopefully we'll be up to 2.9.3 / 2.10 by the weekend (Don't hold me on that one though!) As we've been hard at work squashing some other bugs.

sam24and48 · 2016-10-12T15:06:47Z

Sounds great, thanks again for your help on this!

Removed-5an · 2016-10-17T18:49:10Z

still an issue with 2.9.3 for me

dschaper · 2016-10-17T18:57:17Z

pihole -d please.

Removed-5an · 2016-10-17T19:59:09Z

Test case:

support.microsoft.com

The important snippets:

I see a lot of errors in the webserver part:

#######################################
######### lighttpd error.log ##########
#######################################

2016-10-17 19:41:46: (mod_fastcgi.c.2695) FastCGI-stderr: PHP Notice: Undefined index: piholeIPv6 in /var/www/html/admin/data.php on line 3

#######################################
########### 01-pihole.conf ############
#######################################
addn-hosts=/etc/pihole/gravity.list
domain-needed
bogus-priv
no-resolv
server=8.8.8.8
server=8.8.4.4
interface=eth0
listen-address=127.0.0.1
cache-size=10000
log-queries
log-facility=/var/log/pihole.log
local-ttl=300
log-async

#######################################
############## Whitelist ##############
#######################################
raw.githubusercontent.com
mirror1.malwaredomains.com
sysctl.org
zeustracker.abuse.ch
s3.amazonaws.com
hosts-file.net
ransomwaretracker.abuse.ch
support.microsoft.com

#######################################
############## Blacklist ##############
#######################################

#######################################
############ adlists.list #############
#######################################
No adlists.list file found... using adlists.default!

#######################################
############# pihole.log ##############
#######################################

Oct 17 19:53:35 dnsmasq[159]: query[A] support.microsoft.com from [MYMACHINEIP]
Oct 17 19:53:35 dnsmasq[159]: /etc/pihole/gravity.list support.microsoft.com is [MYPIHOLEIP]

AzureMarker · 2016-10-17T20:00:58Z

cat /etc/pihole/gravity.list | grep support.microsoft.com

Removed-5an · 2016-10-18T09:08:29Z

[MYPIHOLEIPv4] diagnostics.support.microsoft.com
[MYPIHOLEIPv6] diagnostics.support.microsoft.com
[MYPIHOLEIPv4] support.microsoft.com
[MYPIHOLEIPv6] support.microsoft.com

Edit: updating to 2.9.4 and removing gravity.list and all the list.X.. files and using pihole -g results in a total blocked domains of 99,870 while it used to be above 200k.

Now the result of is:

[MYPIHOLEIPv4] diagnostics.support.microsoft.com
[MYPIHOLEIPv6] diagnostics.support.microsoft.com
[MYPIHOLEIPv6] support.microsoft.com

So,,, It's still being blocked via IPv6?

dschaper · 2016-10-26T19:48:49Z

@5an1ty For tracking, can you open a new issue with your current problem, it may be unrelated to the OP's problem.

PromoFaux · 2016-10-26T19:51:08Z

The development branch contains some changes to the way we apply the whitelists, so hopefully we should have any whitelisting issues fixed... (don't quote me on that 😉 )

sam24and48 · 2016-10-28T19:59:28Z

I believe the issue that 5an1ty is describing is the same issue I am having as well. I'm in the process of updating to 2.9.4 after a hiatus to see if I see any improvement.

dschaper · 2016-10-28T20:15:43Z

The next version has a completely rewritten white and blacklist function. (It been merged to a single function and redundancy removed to help eliminate bugs.) The issue should be fixed in Master, but it definitely is fixed in development and will be released coming up shortly.

sco01 · 2016-10-31T09:24:41Z

Confirmed working in v2.9.5/v1.4.4.2 running on Ubuntu server 16.04.1. I had all sorts of problems both with Whitelists being ignored (trying to get Spotify on iOS to work) and domains being blocked going up and down between restarts. Everything seems fine now. Thanks!

PromoFaux · 2016-10-31T09:51:13Z

@sam24and48 did 2.9.5 fix the issue for you, too?

sam24and48 · 2016-10-31T19:58:03Z

I am updating right now and will begin testing to see if it did in fact work. I'll check back in shortly. Thanks!

sam24and48 · 2016-10-31T20:15:49Z

It is looking good so far. I am going to check it again in the morning to make sure but I think this version might have done the trick.

sam24and48 · 2016-11-01T14:48:38Z

It looks like that did the trick. Thanks again for getting that fixed and following up!

PromoFaux · 2016-11-01T14:50:53Z

You're welcome.

dschaper closed this as completed Oct 10, 2016

dschaper mentioned this issue Oct 10, 2016

Whitelist Being Ignored completely (2.9.2 issue on Ubuntu?) pi-hole/web#159

Closed

13 tasks

PromoFaux reopened this Oct 12, 2016

PromoFaux added the ON HOLD label Oct 12, 2016

PromoFaux closed this as completed Nov 1, 2016

Whitelist Being Ignored completely (2.9.2 issue on Ubuntu?) #775

Whitelist Being Ignored completely (2.9.2 issue on Ubuntu?) #775

Comments

sam24and48 commented Oct 10, 2016

dschaper commented Oct 10, 2016

dschaper commented Oct 10, 2016

sam24and48 commented Oct 10, 2016 • edited Loading

dschaper commented Oct 10, 2016

dschaper commented Oct 10, 2016

sam24and48 commented Oct 12, 2016

PromoFaux commented Oct 12, 2016

sam24and48 commented Oct 12, 2016

PromoFaux commented Oct 12, 2016

sam24and48 commented Oct 12, 2016

PromoFaux commented Oct 12, 2016

sam24and48 commented Oct 12, 2016

sam24and48 commented Oct 12, 2016

PromoFaux commented Oct 12, 2016

sam24and48 commented Oct 12, 2016

PromoFaux commented Oct 12, 2016

sam24and48 commented Oct 12, 2016

sam24and48 commented Oct 12, 2016

sam24and48 commented Oct 12, 2016

sam24and48 commented Oct 12, 2016

PromoFaux commented Oct 12, 2016

sam24and48 commented Oct 12, 2016

sam24and48 commented Oct 12, 2016

PromoFaux commented Oct 12, 2016

sam24and48 commented Oct 12, 2016

Removed-5an commented Oct 17, 2016

dschaper commented Oct 17, 2016

Removed-5an commented Oct 17, 2016 • edited Loading

AzureMarker commented Oct 17, 2016

Removed-5an commented Oct 18, 2016 • edited Loading

dschaper commented Oct 26, 2016

PromoFaux commented Oct 26, 2016

sam24and48 commented Oct 28, 2016 • edited Loading

dschaper commented Oct 28, 2016

sco01 commented Oct 31, 2016

PromoFaux commented Oct 31, 2016

sam24and48 commented Oct 31, 2016

sam24and48 commented Oct 31, 2016

sam24and48 commented Nov 1, 2016

PromoFaux commented Nov 1, 2016

sam24and48 commented Oct 10, 2016 •

edited

Loading

Removed-5an commented Oct 17, 2016 •

edited

Loading

Removed-5an commented Oct 18, 2016 •

edited

Loading

sam24and48 commented Oct 28, 2016 •

edited

Loading