New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[enh] Nameserver rotate #352

Open
wants to merge 5 commits into
base: stretch-unstable
from

Conversation

Projects
None yet
5 participants
@maniackcrudelis
Copy link
Contributor

maniackcrudelis commented Aug 15, 2017

Add a load balancing in the nameserver selection.
And keep all the comments about which dns servers they are.

@alexAubin

This comment has been minimized.

Copy link
Member

alexAubin commented Aug 15, 2017

I find it pretty hard to find documation about option rotate but here's a page with a few hints : https://linux-audit.com/linux-dns-tuning-for-performance-and-resilience/

@maniackcrudelis

This comment has been minimized.

Copy link
Contributor

maniackcrudelis commented Aug 15, 2017

So, in fact, this option has no effect on what dnsmasq doing...
Dnsmasq still doing what it wants to do...

But, I discovered that if we comment a nameserver in resolv.dnsmasq.conf, dnsmasq will stopped immediately to used it.
In fact dnsmasq polls its resolvconf file and stay aware of each move in it.
So, if we want to be careful about the charge of the dns servers. There another way to do it, more dirty...
Simply use a cron for select some nameservers in the file and comment the others.

# server, this list is *shuffled* during every regen-conf of dnsmasq
# In the possibility where the first nameserver is down, dnsmasq
# will automatically switch to the next as primary server.

This comment has been minimized.

@Psycojoker

Psycojoker Aug 16, 2017

Member

Might still be worth it to keep a comment explaining this file.

@alexAubin alexAubin added this to the 2.7.x (fixes) or 2.8 milestone Nov 27, 2017

@maniackcrudelis

This comment has been minimized.

Copy link
Contributor

maniackcrudelis commented Nov 29, 2017

Hi guys
So let's go to change the dns list by using a cron.

Here 2 propositions to do that.
The first one can be used directly in a cron file.

grep '^nameserver' /usr/share/yunohost/templates/dnsmasq/plain/resolv.dnsmasq.conf | shuf -n3

But will give us a minimalist file, like that:

nameserver 80.67.169.40
nameserver 141.255.128.100
nameserver 141.255.128.101

The second one need a script put somewhere

#!/bin/bash

# Copy the plain resolv file
cp /usr/share/yunohost/templates/dnsmasq/plain/resolv.dnsmasq.conf /etc/resolv.dnsmasq.conf

# Get the total number of dns
max_dns=$(grep -c '^nameserver' /etc/resolv.dnsmasq.conf)

# Then select ramdomly 3 dns
random_dns () {
	echo $[($RANDOM % ($[$max_dns - 1] + 1)) + 1]
}
dns1=$(random_dns)
dns2=$dns1
while [ $dns2 -eq $dns1 ]
do
	dns2=$(random_dns)
done
dns3=$dns2
while [ $dns3 -eq $dns1 ] || [ $dns3 -eq $dns2 ]
do
	dns3=$(random_dns)
done

# Comment all dns
sed -i 's/^nameserver/#  nameserver/g' /etc/resolv.dnsmasq.conf

# Then uncomment the 3 selected dns
for i in `seq 1 3`
do
	# Find the line for this dns
	dns_line=$(grep -m$(eval echo \$dns${i}) "#  nameserver" /etc/resolv.dnsmasq.conf | tail -n1)
	# Remove the comment at the beginning
	dns_line=${dns_line#\#  }
	# Remove the comment for this dns
	sed -i "s/^#  $dns_line/$dns_line/" /etc/resolv.dnsmasq.conf
done

And will generate a complete file

# [Comment at the beginning of the file...]

# List taken from
# http://diyisp.org/dokuwiki/doku.php?id=technical:dnsresolver

# (FR) FDN
#  nameserver 80.67.169.12
nameserver 80.67.169.40
# (FR) LDN
#  nameserver 80.67.188.188
# (FR) ARN
#  nameserver 89.234.141.66
# (FR) gozmail / grifon
#  nameserver 89.234.186.18
# (DE) FoeBud / Digital Courage
#  nameserver 85.214.20.141
# (FR) Aquilenet [added manually, following comments from @sachaz]
nameserver 141.255.128.100
nameserver 141.255.128.101
# (DE) CCC Berlin
#  nameserver 213.73.91.35
# (DE) Ideal-Hosting
#  nameserver 84.200.69.80
#  nameserver 84.200.70.40
# (DK) censurfridns
#  nameserver 91.239.100.100
#  nameserver 89.233.43.71
@maniackcrudelis

This comment has been minimized.

Copy link
Contributor

maniackcrudelis commented Feb 28, 2018

To go forward on this matter, I added this script to my own server, with a cron

* */1 * * * root /usr/share/yunohost/dns_rotate.sh

We will see...

@alexAubin

This comment has been minimized.

Copy link
Member

alexAubin commented May 11, 2018

We will see...

Any feedback on this ? Does it work as expected ? :P

@maniackcrudelis

This comment has been minimized.

Copy link
Contributor

maniackcrudelis commented May 11, 2018

I do not have a log to really check if it's works as expected now.
But, when I first set it, it worked fine, stopping asking for all dns, and using only those are not commented.

So far, I can say that my dns resolution have worked perfectly without any issues. And as you now, I'm really using it, since all my network is using this dnsmasq.

Right now, my /etc/resolv.dnsmasq.conf looks like that:

# This file will be used to generate /etc/resolv.dnsmasq.conf
# To avoid that every instance rely on the first server as primary
# server, this list is *shuffled* during every regen-conf of dnsmasq
# In the possibility where the first nameserver is down, dnsmasq
# will automatically switch to the next as primary server.

# List taken from
# http://diyisp.org/dokuwiki/doku.php?id=technical:dnsresolver

# (FR) FDN
#  nameserver 80.67.169.12
nameserver 80.67.169.40
# (FR) LDN
#  nameserver 80.67.188.188
# (FR) ARN
#  nameserver 89.234.141.66
# (FR) gozmail / grifon
#  nameserver 89.234.186.18
# (DE) FoeBud / Digital Courage
#  nameserver 85.214.20.141
# (FR) Aquilenet [added manually, following comments from @sachaz]
#  nameserver 141.255.128.100
#  nameserver 141.255.128.101
# (DE) CCC Berlin
#  nameserver 213.73.91.35
# (DE) Ideal-Hosting
nameserver 84.200.69.80
#  nameserver 84.200.70.40
# (DK) censurfridns
#  nameserver 91.239.100.100
nameserver 89.233.43.71

So, I think that we can go forward with that.

@Psycojoker

This comment has been minimized.

Copy link
Member

Psycojoker commented May 12, 2018

Waiting a bit of time to see if there are other reactions before merging then I guess.

@alexAubin

This comment has been minimized.

Copy link
Member

alexAubin commented May 12, 2018

I don't think it can be merged right away because the code doesn't match what was discussed 😅 As far as I understand / remember, we need to add Maniack's cron job somewhere inside yunohost (and make it so that it gets installed at some point)

@Psycojoker

This comment has been minimized.

Copy link
Member

Psycojoker commented May 12, 2018

Ok

@maniackcrudelis

This comment has been minimized.

Copy link
Contributor

maniackcrudelis commented May 12, 2018

As far as I understand / remember, we need to add Maniack's cron job somewhere inside yunohost (and make it so that it gets installed at some point)

Exactly, we need to have this script somewhere and the cron to run it.
If you explain me how to do that, I can do it myself and update this PR.

@Psycojoker

This comment has been minimized.

Copy link
Member

Psycojoker commented May 12, 2018

Sadly you don't have commit rights here so you can't do it :(

You can do a PR on the brnach of this PR but that's a bit stupid to ask you to do that...

@alexAubin

This comment has been minimized.

Copy link
Member

alexAubin commented May 17, 2018

So I integrated Maniack's work into the dnsmasq regen-conf. It also creates an hourly cron job that run service regen-conf dnsmasq with the purpose of enabling a nameserver rotation.

One small issue that remains is that every time a dnsmasq regen conf is called for any other reason, the file /etc/resolv.dnsmasq.conf gets updated (not a regression of this PR, this is already the case currently). This is a bit intriguing for people, though not a big issue. But we could manage to avoid this by setting the RNG seed to some value using the date and/or hour such that if you call the regen-conf in rapid succession, the file would in fact not change.

@maniackcrudelis

This comment has been minimized.

Copy link
Contributor

maniackcrudelis commented May 18, 2018

Do we really need to run a complete regen-conf of dnsmasq to use that script ?
With pre and post, it could be long maybe. And that means also that all YunoHost instances are going to do a request on ip.yunohost.org and ip6.yunohost.org every hour.

And it could be interesting to know that you can use the script to change the DNS without reloading dnsmasq.

@alexAubin

This comment has been minimized.

Copy link
Member

alexAubin commented May 19, 2018

that means also that all YunoHost instances are going to do a request on ip.yunohost.org and ip6.yunohost.org every hour.

Well we already have this every 2 minutes for instances with dyndns domains :P But yup, that should ideally be avoided...

We kinda need to be "inside" the regenconf if we don't want this file to then be flagged as manually modified ... Sure we could also manage it outside the regen-conf, but we should also keep in mind that this file is "taken over" by the internet cube (or people who want to use a specific list of DNS) and therefore in that case we shouldn't overwrite it ... (which is one of the feature the regen-conf has)

In term of perfs, it's basically ~10 sec max on slow hardware I think, so I wouldnt be too worried about this.

I don't know what would be a proper solution ... to me the current issue is that regen-conf can't manage files individually and is tightly-coupled to the concept of services, but fixing that implies some heavy refactoring I think ...

@Psycojoker

This comment has been minimized.

Copy link
Member

Psycojoker commented May 19, 2018

With pre and post, it could be long maybe. And that means also that all YunoHost instances are going to do a request on ip.yunohost.org and ip6.yunohost.org every hour.

In practice this is going to be something like arround 2k requests on a full nginx file, which for nginx is like : nothing. It's really a very hard rock server and this is a super lightweight request.

Nevertheless it would be a good pratice to put a random value somewhere to load balance all of that like we did for dyndns.

@Josue-T

This comment has been minimized.

Copy link
Contributor

Josue-T commented May 19, 2018

And what happen if the dnsmasq config is customised by the user if we try to run the regen-conf every hours ?

@Psycojoker

This comment has been minimized.

Copy link
Member

Psycojoker commented May 19, 2018

Conf won't be modified because it has been modified by the user (except if we do a --force)

@Josue-T

This comment has been minimized.

Copy link
Contributor

Josue-T commented May 19, 2018

So in this case the rotation of the nameserver won't be done ?

@alexAubin

This comment has been minimized.

Copy link
Member

alexAubin commented May 19, 2018

Yes

@maniackcrudelis

This comment has been minimized.

Copy link
Contributor

maniackcrudelis commented May 20, 2018

So in this case the rotation of the nameserver won't be done ?
Yes

That's kind of too bad...
I have maybe an idea, but tell me if is too dirty.

My idea is that:

  • The script stay as an independent script, external to the regen-conf process.
  • Before each execution, the script check is there's a copy of /etc/resolv.dnsmasq.conf, and make one if needed.
    The idea is to keep a clear copy of this file, without modifications from the script.
    Could be even better if this copy is made by YunoHost during the setup of this script.
  • The script will run with the cron, without the regen-conf. So just the script itself. And doing its job on /etc/resolv.dnsmasq.conf
  • In case of regen-conf, a pre script will move the backup file of /etc/resolv.dnsmasq.conf to its original place. So the regen-conf won't be bother by the work done by the script.
    The file could even be modified by the regen-conf, if needed.
  • Then after the regen-conf, a post script make a new backup copy of /etc/resolv.dnsmasq.conf from the new version of the file.
    And then force the script to select 3 dns in the new /etc/resolv.dnsmasq.conf.

Like that

  • We don't have to run a regen-conf every hour.
  • An user can still modify his /etc/resolv.dnsmasq.conf (but he should also change the backup file as well) without losing the usage of the script to select 3 dns.
  • But, that means also creating 2 other simple scripts, for pre and post regen-conf.
  • And put aside this script itself.

If we don't want to use a backup file, we could also simply uncomment all dns in the pre script to restore the file to its original shape.

@Josue-T

This comment has been minimized.

Copy link
Contributor

Josue-T commented May 20, 2018

I have maybe an idea, but tell me if is too dirty.
My idea is that:
...

It's an idea

I though know about an other solution to select in the list DNS server. My idea is select the 3 best quickest server. For this the idea is to check for each server the query time (we can do that by a ping and/or dig). And after we select the 3 shortest value. We could do this in the regen-conf and after we keep these values.

@maniackcrudelis

This comment has been minimized.

Copy link
Contributor

maniackcrudelis commented May 20, 2018

The idea Josue-T isn't to use the 3 fastest DNS, which is already the method used by dnsmasq, the idea here is to change the dns used to avoid using always the same ones.
Just because we know that the DNS are a vulnerability point of our personal data, by this PR we'll force dnsmasq to change randomly its source dns.

@Josue-T

This comment has been minimized.

Copy link
Contributor

Josue-T commented May 20, 2018

Ah, ok I understand.

@alexAubin alexAubin referenced this pull request May 21, 2018

Open

Decouple the regen-conf mechanism from services #480

0 of 4 tasks complete
@zamentur

This comment has been minimized.

Copy link
Contributor

zamentur commented Jun 11, 2018

Running a regen-conf every hour will display the pacman every hour on the yunohost-admin interface. Could be quite annoying. I suggest to put this daily and not hourly.

@alexAubin alexAubin changed the base branch from unstable to stretch-unstable Jun 17, 2018

@alexAubin alexAubin modified the milestones: 2.7.x, 3.1.x Jun 19, 2018

@alexAubin alexAubin modified the milestones: 3.1.x, 3.2.x Aug 15, 2018

@alexAubin alexAubin modified the milestones: 3.2.x, 3.x Oct 24, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment