Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[enh] Nameserver rotate #352

Open
wants to merge 8 commits into
base: stretch-unstable
Choose a base branch
from
Open

Conversation

@maniackcrudelis
Copy link
Contributor

@maniackcrudelis maniackcrudelis commented Aug 15, 2017

Add a load balancing in the nameserver selection.
And keep all the comments about which dns servers they are.

@alexAubin
Copy link
Member

@alexAubin alexAubin commented Aug 15, 2017

I find it pretty hard to find documation about option rotate but here's a page with a few hints : https://linux-audit.com/linux-dns-tuning-for-performance-and-resilience/

@maniackcrudelis
Copy link
Contributor Author

@maniackcrudelis maniackcrudelis commented Aug 15, 2017

So, in fact, this option has no effect on what dnsmasq doing...
Dnsmasq still doing what it wants to do...

But, I discovered that if we comment a nameserver in resolv.dnsmasq.conf, dnsmasq will stopped immediately to used it.
In fact dnsmasq polls its resolvconf file and stay aware of each move in it.
So, if we want to be careful about the charge of the dns servers. There another way to do it, more dirty...
Simply use a cron for select some nameservers in the file and comment the others.

# server, this list is *shuffled* during every regen-conf of dnsmasq
# In the possibility where the first nameserver is down, dnsmasq
# will automatically switch to the next as primary server.

This comment has been minimized.

@Psycojoker

Psycojoker Aug 16, 2017
Member

Might still be worth it to keep a comment explaining this file.

@alexAubin alexAubin added this to the 2.7.x (fixes) or 2.8 milestone Nov 27, 2017
@maniackcrudelis
Copy link
Contributor Author

@maniackcrudelis maniackcrudelis commented Nov 29, 2017

Hi guys
So let's go to change the dns list by using a cron.

Here 2 propositions to do that.
The first one can be used directly in a cron file.

grep '^nameserver' /usr/share/yunohost/templates/dnsmasq/plain/resolv.dnsmasq.conf | shuf -n3

But will give us a minimalist file, like that:

nameserver 80.67.169.40
nameserver 141.255.128.100
nameserver 141.255.128.101

The second one need a script put somewhere

#!/bin/bash

# Copy the plain resolv file
cp /usr/share/yunohost/templates/dnsmasq/plain/resolv.dnsmasq.conf /etc/resolv.dnsmasq.conf

# Get the total number of dns
max_dns=$(grep -c '^nameserver' /etc/resolv.dnsmasq.conf)

# Then select ramdomly 3 dns
random_dns () {
	echo $[($RANDOM % ($[$max_dns - 1] + 1)) + 1]
}
dns1=$(random_dns)
dns2=$dns1
while [ $dns2 -eq $dns1 ]
do
	dns2=$(random_dns)
done
dns3=$dns2
while [ $dns3 -eq $dns1 ] || [ $dns3 -eq $dns2 ]
do
	dns3=$(random_dns)
done

# Comment all dns
sed -i 's/^nameserver/#  nameserver/g' /etc/resolv.dnsmasq.conf

# Then uncomment the 3 selected dns
for i in `seq 1 3`
do
	# Find the line for this dns
	dns_line=$(grep -m$(eval echo \$dns${i}) "#  nameserver" /etc/resolv.dnsmasq.conf | tail -n1)
	# Remove the comment at the beginning
	dns_line=${dns_line#\#  }
	# Remove the comment for this dns
	sed -i "s/^#  $dns_line/$dns_line/" /etc/resolv.dnsmasq.conf
done

And will generate a complete file

# [Comment at the beginning of the file...]

# List taken from
# http://diyisp.org/dokuwiki/doku.php?id=technical:dnsresolver

# (FR) FDN
#  nameserver 80.67.169.12
nameserver 80.67.169.40
# (FR) LDN
#  nameserver 80.67.188.188
# (FR) ARN
#  nameserver 89.234.141.66
# (FR) gozmail / grifon
#  nameserver 89.234.186.18
# (DE) FoeBud / Digital Courage
#  nameserver 85.214.20.141
# (FR) Aquilenet [added manually, following comments from @sachaz]
nameserver 141.255.128.100
nameserver 141.255.128.101
# (DE) CCC Berlin
#  nameserver 213.73.91.35
# (DE) Ideal-Hosting
#  nameserver 84.200.69.80
#  nameserver 84.200.70.40
# (DK) censurfridns
#  nameserver 91.239.100.100
#  nameserver 89.233.43.71
@maniackcrudelis
Copy link
Contributor Author

@maniackcrudelis maniackcrudelis commented Feb 28, 2018

To go forward on this matter, I added this script to my own server, with a cron

* */1 * * * root /usr/share/yunohost/dns_rotate.sh

We will see...

@alexAubin
Copy link
Member

@alexAubin alexAubin commented May 11, 2018

We will see...

Any feedback on this ? Does it work as expected ? :P

@maniackcrudelis
Copy link
Contributor Author

@maniackcrudelis maniackcrudelis commented May 11, 2018

I do not have a log to really check if it's works as expected now.
But, when I first set it, it worked fine, stopping asking for all dns, and using only those are not commented.

So far, I can say that my dns resolution have worked perfectly without any issues. And as you now, I'm really using it, since all my network is using this dnsmasq.

Right now, my /etc/resolv.dnsmasq.conf looks like that:

# This file will be used to generate /etc/resolv.dnsmasq.conf
# To avoid that every instance rely on the first server as primary
# server, this list is *shuffled* during every regen-conf of dnsmasq
# In the possibility where the first nameserver is down, dnsmasq
# will automatically switch to the next as primary server.

# List taken from
# http://diyisp.org/dokuwiki/doku.php?id=technical:dnsresolver

# (FR) FDN
#  nameserver 80.67.169.12
nameserver 80.67.169.40
# (FR) LDN
#  nameserver 80.67.188.188
# (FR) ARN
#  nameserver 89.234.141.66
# (FR) gozmail / grifon
#  nameserver 89.234.186.18
# (DE) FoeBud / Digital Courage
#  nameserver 85.214.20.141
# (FR) Aquilenet [added manually, following comments from @sachaz]
#  nameserver 141.255.128.100
#  nameserver 141.255.128.101
# (DE) CCC Berlin
#  nameserver 213.73.91.35
# (DE) Ideal-Hosting
nameserver 84.200.69.80
#  nameserver 84.200.70.40
# (DK) censurfridns
#  nameserver 91.239.100.100
nameserver 89.233.43.71

So, I think that we can go forward with that.

@Psycojoker
Copy link
Member

@Psycojoker Psycojoker commented May 12, 2018

Waiting a bit of time to see if there are other reactions before merging then I guess.

@alexAubin
Copy link
Member

@alexAubin alexAubin commented May 12, 2018

I don't think it can be merged right away because the code doesn't match what was discussed 😅 As far as I understand / remember, we need to add Maniack's cron job somewhere inside yunohost (and make it so that it gets installed at some point)

@Psycojoker
Copy link
Member

@Psycojoker Psycojoker commented May 12, 2018

Ok

@maniackcrudelis
Copy link
Contributor Author

@maniackcrudelis maniackcrudelis commented May 12, 2018

As far as I understand / remember, we need to add Maniack's cron job somewhere inside yunohost (and make it so that it gets installed at some point)

Exactly, we need to have this script somewhere and the cron to run it.
If you explain me how to do that, I can do it myself and update this PR.

@Psycojoker
Copy link
Member

@Psycojoker Psycojoker commented May 12, 2018

Sadly you don't have commit rights here so you can't do it :(

You can do a PR on the brnach of this PR but that's a bit stupid to ask you to do that...

@alexAubin
Copy link
Member

@alexAubin alexAubin commented May 17, 2018

So I integrated Maniack's work into the dnsmasq regen-conf. It also creates an hourly cron job that run service regen-conf dnsmasq with the purpose of enabling a nameserver rotation.

One small issue that remains is that every time a dnsmasq regen conf is called for any other reason, the file /etc/resolv.dnsmasq.conf gets updated (not a regression of this PR, this is already the case currently). This is a bit intriguing for people, though not a big issue. But we could manage to avoid this by setting the RNG seed to some value using the date and/or hour such that if you call the regen-conf in rapid succession, the file would in fact not change.

@maniackcrudelis
Copy link
Contributor Author

@maniackcrudelis maniackcrudelis commented May 18, 2018

Do we really need to run a complete regen-conf of dnsmasq to use that script ?
With pre and post, it could be long maybe. And that means also that all YunoHost instances are going to do a request on ip.yunohost.org and ip6.yunohost.org every hour.

And it could be interesting to know that you can use the script to change the DNS without reloading dnsmasq.

@alexAubin
Copy link
Member

@alexAubin alexAubin commented May 19, 2018

that means also that all YunoHost instances are going to do a request on ip.yunohost.org and ip6.yunohost.org every hour.

Well we already have this every 2 minutes for instances with dyndns domains :P But yup, that should ideally be avoided...

We kinda need to be "inside" the regenconf if we don't want this file to then be flagged as manually modified ... Sure we could also manage it outside the regen-conf, but we should also keep in mind that this file is "taken over" by the internet cube (or people who want to use a specific list of DNS) and therefore in that case we shouldn't overwrite it ... (which is one of the feature the regen-conf has)

In term of perfs, it's basically ~10 sec max on slow hardware I think, so I wouldnt be too worried about this.

I don't know what would be a proper solution ... to me the current issue is that regen-conf can't manage files individually and is tightly-coupled to the concept of services, but fixing that implies some heavy refactoring I think ...

@Psycojoker
Copy link
Member

@Psycojoker Psycojoker commented May 19, 2018

With pre and post, it could be long maybe. And that means also that all YunoHost instances are going to do a request on ip.yunohost.org and ip6.yunohost.org every hour.

In practice this is going to be something like arround 2k requests on a full nginx file, which for nginx is like : nothing. It's really a very hard rock server and this is a super lightweight request.

Nevertheless it would be a good pratice to put a random value somewhere to load balance all of that like we did for dyndns.

@maniackcrudelis
Copy link
Contributor Author

@maniackcrudelis maniackcrudelis commented May 20, 2018

The idea Josue-T isn't to use the 3 fastest DNS, which is already the method used by dnsmasq, the idea here is to change the dns used to avoid using always the same ones.
Just because we know that the DNS are a vulnerability point of our personal data, by this PR we'll force dnsmasq to change randomly its source dns.

@Josue-T
Copy link
Contributor

@Josue-T Josue-T commented May 20, 2018

Ah, ok I understand.

@alexAubin alexAubin mentioned this pull request May 21, 2018
0 of 4 tasks complete
@zamentur
Copy link
Contributor

@zamentur zamentur commented Jun 11, 2018

Running a regen-conf every hour will display the pacman every hour on the yunohost-admin interface. Could be quite annoying. I suggest to put this daily and not hourly.

@alexAubin alexAubin changed the base branch from unstable to stretch-unstable Jun 17, 2018
@alexAubin alexAubin modified the milestones: 2.7.x, 3.1.x Jun 19, 2018
@alexAubin alexAubin modified the milestones: 3.1.x, 3.2.x Aug 15, 2018
@alexAubin alexAubin modified the milestones: 3.2.x, 3.x Oct 24, 2018
@alexAubin alexAubin mentioned this pull request Feb 18, 2019
0 of 4 tasks complete
@alexAubin
Copy link
Member

@alexAubin alexAubin commented Apr 18, 2019

#653 got merged so we could move forward with this

@alexAubin alexAubin force-pushed the stretch-unstable branch from 58ead9e to 40f48ff Oct 29, 2019
kay0u and others added 2 commits Apr 13, 2020
@maniackcrudelis
Copy link
Contributor Author

@maniackcrudelis maniackcrudelis commented Apr 13, 2020

Ok, I have a simple but working code for that rotation of nameserver.

I made the regen conf work on a separate file, and a hourly cron job does the rotation from this file.
An user can change this file to add permanent modification, and so block the regen-conf if he wants to.

My hourly cron job is:

#!/bin/bash

# Copy the full resolv file, from the regen conf
cp /etc/resolv.dnsmasq.full.conf /etc/resolv.dnsmasq.conf

# Get the total number of dns for ipv4 and ipv6
max_dns4=$(grep '^nameserver' /etc/resolv.dnsmasq.conf | grep --count "\.")
max_dns6=$(grep '^nameserver' /etc/resolv.dnsmasq.conf | grep --count ":")

# Then select randomly 4 dns
random_dns () {
        max_dns=max_dns$1
        echo $[($RANDOM % ($[$max_dns - 1] + 1)) + 1]
}

# Start with 2 ipv4 dns
dns1=$(random_dns 4)
dns2=$dns1
while [ $dns2 -eq $dns1 ]
do
        dns2=$(random_dns 4)
done

# And 2 ipv6 dns
dns3=$(random_dns 6)
dns4=$dns3
while [ $dns3 -eq $dns4 ]
do
        dns4=$(random_dns 6)
done

# Comment all dns
sed -i 's/^nameserver/#  nameserver/g' /etc/resolv.dnsmasq.conf

# Then uncomment the 4 selected dns
for i in `seq 1 4`
do
        dns_count=dns$i
        # For dns1 and 2, use an ipv4 address
        if [ $i -le 2 ]
        then
                ip_filter="\."
        # For dns3 and 4, use an ipv6 address
        else
                ip_filter=":"
        fi

        # Find the line for this dns
        # Grep the number of ipv4/6 dns corresponding to the value got with random_dns
        # Then tail to get only the last one, which is the value found with random_dns
        dns_line="$(grep "#  nameserver" /etc/resolv.dnsmasq.conf | grep -m${!dns_count} "$ip_filter" | tail -n1)"
        # Remove the comment at the beginning
        dns_line=${dns_line#\#  }
        # Uncomment this dns into the file
        sed -i "s/^#  $dns_line/$dns_line/" /etc/resolv.dnsmasq.conf
done

With 2 ipv4 and 2 ipv6 nameserver.

The only thing now is that I need some help to know where to put this script so it would end up into /etc/cron.hourly ?

edit: My "hourly" eventually decided to run at 19:17:01, but it worked perfectly, and my regen-conf is not bothered by the rotation.

@zamentur zamentur modified the milestones: 4.x, Horizon Jan 3, 2021
@zamentur zamentur added this to Needs triage in Pending Jan 4, 2021
@zamentur zamentur removed this from the Horizon milestone Jan 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Pending
Needs triage
6 participants