Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Router mode supports loading from disk #261

Open
liang-hiwin opened this issue Sep 3, 2022 · 15 comments
Open

Router mode supports loading from disk #261

liang-hiwin opened this issue Sep 3, 2022 · 15 comments

Comments

@liang-hiwin
Copy link

Router

Configuration:

[routers.router1]
routes = [
{ name = '(^|.)google.com.$', types = ["A", "AAAA", "MX"], resolver="google-udp" },
{ name = '(^|.)youtube.com.$', types = ["A", "AAAA", "MX"], resolver="quad9-udp" },
............. #many domain names
{ resolver="cloudflare-dot" }, # default route
]

When more than 500 lines are added here, routedns cannot be started or when the program can start the program, it cannot be used normally, and no parsing can be performed at this moment.

@liang-hiwin
Copy link
Author

It is recommended to change to something like this to read the list

[routers.router1]
routes = [
{ file = "/path/to/google_domain.list", types = ["A", "AAAA", "MX"], resolver="google_udp" },
{ file = "/path/to/youtube_domain.list", types = ["A", "AAAA", "MX"], resolver="quad9_udp" },
{ file = "/path/to/github_domain.list", types = ["A", "AAAA", "MX"], resolver="opendns_udp" },
............. #many domain names
{ resolver="cloudflare_dot" }, # default route
]

The content of google_domain.list is one domain name per line,

  1. When filling in a single string, such as google.com, it will match "google.com", "google.com.hk" and "www.google.com", but not "google.cn".

  2. When filling in a string starting with "domain:", such as doamin:google.com, it matches "www.google.com", "google.com", but does not match "adsgoogle.com"

  3. When filling in a string starting with "full:", such as full:google.com, it matches "google.com", but does not match "www.google.com"

  4. When filling in a string starting with "regexp:", such as regexp:google.com$, the matching method fully supports go regexp syntax

@folbricht
Copy link
Owner

Routers use regexp which are less efficient, though I would still expect it work work (not tested with 500 rules yet). From your example it doesn't look like you need a regex list. To achieve the same thing without regex, you can use a blocklist. It'll let you define a large list of domain names and give you a resolver for everything that matches the list, and one for everything else.

@liang-hiwin
Copy link
Author

Routers use regexp which are less efficient, though I would still expect it work work (not tested with 500 rules yet). From your example it doesn't look like you need a regex list. To achieve the same thing without regex, you can use a blocklist. It'll let you define a large list of domain names and give you a resolver for everything that matches the list, and one for everything else.

I test your method , thanks

@liang-hiwin
Copy link
Author

@folbricht

Can resolvers add sock5 proxy mode

[resolvers.cloudflare-dot-1]
address = "1.1.1.1:853"
protocol = "dot"
sock5 = "127.0.0.1:1080"   

 ######1.1.1.1:853 All traffic is forwarded by the sock5 proxy, which can avoid use when 1.1.1.1 is blocked by the firewall

@liang-hiwin
Copy link
Author

Routers use regexp which are less efficient, though I would still expect it work work (not tested with 500 rules yet). From your example it doesn't look like you need a regex list. To achieve the same thing without regex, you can use a blocklist. It'll let you define a large list of domain names and give you a resolver for everything that matches the list, and one for everything else.

Response Blocklist -> resolvers - Array of upstream resolvers, only one is supported. Only one can be filled in the upstream of the list. How to define a domain name resolver that is not in the list?

@folbricht
Copy link
Owner

There is a blocklist-resolver option you can use. All matches will be sent to that. See https://github.com/folbricht/routedns/blob/master/doc/configuration.md#Query-Blocklist

@liang-hiwin
Copy link
Author

liang-hiwin commented Sep 4, 2022

There is a blocklist-resolver option you can use. All matches will be sent to that. See https://github.com/folbricht/routedns/blob/master/doc/configuration.md#Query-Blocklist

Can an input format like this be added?

[groups.alldns]
type = "blocklist-v2"
resolvers = ["random"]
blocklist-refresh = 30
blocklist-source = [
   {format = "hosts", source = "/opt/routedns/hosts.txt"},
]

allowlist-resolver = "google_domain"
allowlist-refresh = 30
allowlist-source = [
   {format = "domain", source = "/opt/routedns/google_list.txt"},
]


##############Adding the following code on the basis of the above will report an error

allowlist-resolver = "youtube_domain"
allowlist-refresh = 30
allowlist-source = [
   {format = "domain", source = "/opt/routedns/youtube_list.txt"},
]


@folbricht
Copy link
Owner

You can't repeat the same properties in one group. So things like allowlist-source can only exist once, but you can define multiple lists, so something like this should work:

allowlist-resolver = "google_domain"
allowlist-refresh = 30
allowlist-source = [
   {format = "domain", source = "/opt/routedns/google_list.txt"},
   {format = "domain", source = "/opt/routedns/youtube_list.txt"},
]

If you want to break this up and have different allowlist-resolvers per list, you'll need to chain a couple of blocklistv2, each with a different allowlist

@liang-hiwin
Copy link
Author

"random"]
blocklist-refresh = 30
blocklist-source = [
{format = "hosts", source = "/opt/routedns/hosts.txt"},
]

How to connect two blocklist-v2, I'm a little confused。

[groups.alldns]
type = "blocklist-v2"
resolvers = ["youtube_dns"]
blocklist-refresh = 30
blocklist-source = [
   {format = "hosts", source = "/opt/routedns/hosts.txt"},
]

allowlist-resolver = "google_domain"
allowlist-refresh = 30
allowlist-source = [
   {format = "domain", source = "/opt/routedns/google_list.txt"},
]

[groups.youtube_dns]
type = "blocklist-v2"
resolvers = ["random"]
allowlist-resolver = "youtube_domain"
allowlist-refresh = 30
allowlist-source = [
   {format = "domain", source = "/opt/routedns/youtube_list.txt"},
]

@folbricht
Copy link
Owner

It's actually much simpler. You don't even need to use the allowlists if all you want to do is categorize queries and route them separately

[groups.youtube]
type = "blocklist-v2"
resolvers = ["microsoft"] # Forward all non-matches to the next splitter
blocklist-resolver = "youtube-resolver" # This is where all matches (youtube traffic) are routed
blocklist-refresh = 30
blocklist-source = [
   {format = "hosts", source = "/opt/routedns/youtube.txt"}, # This list contains all YT domains
]

[groups.microsoft]
type = "blocklist-v2"
resolvers = ["upstream"] # Forward all non-matches to the next resolver
blocklist-resolver = "microsoft-resolver" # This is where all matches (microsoft traffic) are routed
blocklist-refresh = 30
blocklist-source = [
   {format = "hosts", source = "/opt/routedns/microsoft.txt"}, # This list contains all microsoft domains
]

@folbricht
Copy link
Owner

It's actually much simpler. You don't even need to use the allowlists if all you want to do is categorize queries and route them separately. Something like this.

[groups.youtube]
type = "blocklist-v2"
resolvers = ["microsoft"] # Forward all non-matches to the next splitter
blocklist-resolver = "youtube-resolver" # This is where all matches (youtube traffic) are routed
blocklist-refresh = 30
blocklist-source = [
   {format = "hosts", source = "/opt/routedns/youtube.txt"}, # This list contains all YT domains
]

[groups.microsoft]
type = "blocklist-v2"
resolvers = ["upstream"] # Forward all non-matches to the next resolver
blocklist-resolver = "microsoft-resolver" # This is where all matches (microsoft traffic) are routed
blocklist-refresh = 30
blocklist-source = [
   {format = "hosts", source = "/opt/routedns/microsoft.txt"}, # This list contains all microsoft domains
]

There are other ways but this is quite straightforward

@cbuijs
Copy link
Contributor

cbuijs commented Sep 7, 2022

Should formats not be domains?

@liang-hiwin
Copy link
Author

Should formats not be domains?

domain is true

@pmeiyu
Copy link

pmeiyu commented Jan 17, 2024

Routers use regexp which are less efficient, though I would still expect it work work (not tested with 500 rules yet). From your example it doesn't look like you need a regex list. To achieve the same thing without regex, you can use a blocklist. It'll let you define a large list of domain names and give you a resolver for everything that matches the list, and one for everything else.

@folbricht I want to route 1000 domains to resolver A, another 1000 domains to resolver B, another 1000 domains to resolver C, another 1000 domains to resolver D. But blocklist-v2 can have only three resolvers (default, allow, block). Is there an easy solution apart from chaining several blocklist-v2 together? It would be great if router mode supports easily specifying thousands of domains in a route, either in a file or in a list, like blocklist-v2 format.

@folbricht
Copy link
Owner

There isn't currently anything like that available. Chained blocklists is likely the fastest solution. But you can also use a router with a ton of routes, though if this gets very long, let's says 10s or 100s of thousands it'll slow things down during querying since it treats them as regex and evaluates all in order. To keep the config readable, you can break out the router config into a separate file as per https://github.com/folbricht/routedns/blob/master/doc/configuration.md#Split-Configuration

If you see an issue with routedns having issues with a that let me know. The original comment in this thread suggested a limit of around 500 but I can't reproduce that. Even with 2k routes the startup time isn't impacted at all for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants