Problem cross-compiling for MIPS router support

[john9527]

I currently maintain a fork of Asuswrt-Merlin firmware for ASUS routers.
https://github.com/john9527/asuswrt-merlin

I am trying to integrate getdns/stubby support. I have successfully built for arm7l based routers, which use a customized kernel 2.6.36.4....it's working very well.

However, trying to build for MIPS routers using a kernel based on 2.6.22.19, I can successfully build/compile, but there are problems running the executable. Here's some basic info on what I am seeing. The same yml config runs fine on the ARM routers (I dynamically generate it in the router firmware). The return code is rather interesting. Any ideas?

admin@RT-N66R-2628:/tmp/home/root# stubby -g -l -C /etc/stubby.yml
Could not configure context with config dict: A required parameter had an invalid value.
Could not parse config file "/etc/stubby.yml": A required parameter had an invalid value.

admin@RT-N66R-2628:/tmp/home/root# echo $?
55

admin@RT-N66R-2628:/tmp/home/root# cat /etc/stubby.yml
tls_ca_file: "/rom/ca-bundle.crt"
resolution_type: GETDNS_RESOLUTION_STUB
dns_transport_list:
  - GETDNS_TRANSPORT_TLS
tls_authentication: GETDNS_AUTHENTICATION_REQUIRED
tls_query_padding_blocksize: 256
edns_client_subnet_private : 1
round_robin_upstreams: 1
idle_timeout: 10000
tls_backoff_time: 900
appdata_dir: "/var/tmp/stubby"
listen_addresses:
  - 127.0.0.1@5453
upstream_recursive_servers:
# Quad 9
  - address_data: 9.9.9.9
    tls_auth_name: "dns.quad9.net"

[abelbeck]

Possibly it is an issue with your libyaml lib on the MIPS.

As a quick test, on the working arm7l box generate a subby.conf

stubby -i > stubby.conf

and then use that JSON-like config on the MIPS box (.conf is important)

stubby -g -l -C /etc/stubby.conf

Disclaimer: never tried this, but should work.

If that works your libyaml lib on the MIPS looks suspect.

[john9527]

Tried it exporting both the default config and a running config.

Trying to start on the MIPS box fails with both, printing out the JSON config and ending with
Could not parse config file "/etc/stubby.conf": Generic error

EDIT: Sorry I forgot to say THANKS for the quick response!

[john9527]

Also wanted to mention I tried downleveling both getdns and libyaml to the next to last releases with the same result.

[abelbeck]

getdns-1.4.2 has an import parsing bugfix.

Note the stubby -i output includes a bunch of inherited values, you might try (carefully) trimming that to only your added configs in stubby.conf.

Possibly a stubby developer @wtoorop has an idea.

[john9527]

Currently running getdns-1.4.2 and libyaml-0.2.1
Tried backleveling to getdns-1.4.1 and libyaml-0.1.7 with the same results.

Since the built in default Opportunistic config fails the same way, I have the feeling it's something basic if we could narrow it down. Possibly something in the ancient MIPS kernel is causing a problem (can't update the kernel since it's tied to the SDK for hardware). If stubby isn't setting the return code, the return code 55 (NO ANODE) would seem to point that way. If we could figure out what was wrong, I might be able to backport the required fixes to the kernel.

[saradickinson]

If the default config is failing then it is probably something fundamental in the internal JSON parsing (the default config uses JSON not YAML). The default config looks like this:

{ resolution_type: GETDNS_RESOLUTION_STUB
, dns_transport_list: [ GETDNS_TRANSPORT_TLS
                      , GETDNS_TRANSPORT_UDP
                      , GETDNS_TRANSPORT_TCP ]
, idle_timeout: 10000
, listen_addresses: [ 127.0.0.1@53, 0::1@53 ]
, tls_query_padding_blocksize: 256
, edns_client_subnet_private : 1
, round_robin_upstreams: 1}

You can save this to a file with a .conf extension and use this on the command line. In theory you should be able to use a single line file with just the listen address in, so you could strip out each extra line and see if you see any difference.... pretty empirical at this point but maybe worth a try. (Apologies our error reporting is so poor here...)

I could try to provide a patch to improve the error reporting?

I would be interested to know if getdns_query works - the initialisation is slightly different to stubby?

[john9527]

Thanks for the help. Your test suggestions I think verified what I was thinking that the error message was misleading. I went down to your single line conf with the following results.

admin@RT-N66R-2628:/tmp/home/root# cat /etc/stubbydef.conf
{   listen_addresses: [ 127.0.0.1@53, 0::1@53 ]
}

admin@RT-N66R-2628:/tmp/home/root# stubby -i -C /etc/stubbydef.conf
[02:12:07.289634] STUBBY: Read config from file /etc/stubbydef.conf
{
  "all_context":
  {
    "add_warning_for_bad_dns": GETDNS_EXTENSION_FALSE,
    "appdata_dir": <bindata of "/root/.getdns/">,
    "append_name": GETDNS_APPEND_NAME_TO_SINGLE_LABEL_FIRST,
    "dns_transport_list":
    [
      GETDNS_TRANSPORT_TLS,
      GETDNS_TRANSPORT_UDP,
      GETDNS_TRANSPORT_TCP
    ],
    "dnssec_allowed_skew": 0,
    "dnssec_return_all_statuses": GETDNS_EXTENSION_FALSE,
    "dnssec_return_full_validation_chain": GETDNS_EXTENSION_FALSE,
    "dnssec_return_only_secure": GETDNS_EXTENSION_FALSE,
    "dnssec_return_status": GETDNS_EXTENSION_FALSE,
    "dnssec_return_validation_chain": GETDNS_EXTENSION_FALSE,
    "edns_client_subnet_private": 1,
    "edns_cookies": GETDNS_EXTENSION_FALSE,
    "edns_do_bit": 0,
    "edns_extended_rcode": 0,
    "edns_version": 0,
    "follow_redirects": GETDNS_REDIRECTS_FOLLOW,
    "hosts": <bindata of "/etc/hosts">,
    "idle_timeout": 10000,
    "limit_outstanding_queries": 0,
    "max_backoff_value": 1000,
    "namespaces":
    [
      GETDNS_NAMESPACE_LOCALNAMES,
      GETDNS_NAMESPACE_DNS
    ],
    "resolution_type": GETDNS_RESOLUTION_STUB,
    "resolvconf": <bindata of "/etc/resolv.conf">,
    "return_both_v4_and_v6": GETDNS_EXTENSION_FALSE,
    "return_call_reporting": GETDNS_EXTENSION_FALSE,
    "round_robin_upstreams": 1,
    "specify_class": 1,
    "suffix": [],
    "timeout": 5000,
    "tls_authentication": GETDNS_AUTHENTICATION_NONE,
    "tls_backoff_time": 3600,
    "tls_cipher_list": <bindata of "TLS13-AES-256-GCM-SHA384:TLS13-A"...>,
    "tls_connection_retries": 2,
    "tls_query_padding_blocksize": 256,
    "trust_anchors_url": <bindata of "http://data.iana.org/root-anchor"...>,
    "trust_anchors_verify_CA": <bindata of 0x2d2d2d2d2d424547494e204345525449...>,
    "trust_anchors_verify_email": <bindata of "dnssec@iana.org">,
    "upstream_recursive_servers": []
  },
  "api_version_number": 132058112,
  "api_version_string": <bindata of "December 2015">,
  "compilation_comment": <bindata of "getdns 1.4.2 configured on 2018-"...>,
  "default_hosts_location": <bindata of "/etc/hosts">,
  "default_resolvconf_location": <bindata of "/etc/resolv.conf">,
  "default_trust_anchor_location": <bindata of "/etc/unbound/getdns-root.key">,
  "implementation_string": <bindata of "https://getdnsapi.net">,
  "listen_addresses":
  [
    {
      "address_data": <bindata for 127.0.0.1>,
      "address_type": <bindata of "IPv4">,
      "port": 53
    },
    {
      "address_data": <bindata for ::1>,
      "address_type": <bindata of "IPv6">,
      "port": 53
    }
  ],
  "openssl_build_version_number": 268443903,
  "resolution_type": GETDNS_RESOLUTION_STUB,
  "version_number": 17039872,
  "version_string": <bindata of "1.4.2">
}
Result: Config file syntax is valid.

admin@RT-N66R-2628:/tmp/home/root# stubby -g -l -C /etc/stubbydef.conf
[02:13:55.966973] STUBBY: Read config from file /etc/stubbydef.conf
error: Could not bind on given addresses: File exists

admin@RT-N66R-2628:/tmp/home/root# netstat -nap
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:5473            0.0.0.0:*               LISTEN      386/u2ec
tcp        0      0 0.0.0.0:18017           0.0.0.0:*               LISTEN      321/wanduck
tcp        0      0 0.0.0.0:3394            0.0.0.0:*               LISTEN      386/u2ec
tcp        0      0 0.0.0.0:515             0.0.0.0:*               LISTEN      388/lpd
tcp        0      0 0.0.0.0:9100            0.0.0.0:*               LISTEN      388/lpd
tcp        0      0 0.0.0.0:9998            0.0.0.0:*               LISTEN      358/ots
tcp        0      0 192.168.1.1:80          0.0.0.0:*               LISTEN      351/httpd
tcp        0      0 0.0.0.0:3838            0.0.0.0:*               LISTEN      388/lpd
tcp        0      0 :::23                   :::*                    LISTEN      338/telnetd
tcp        0      0 ::ffff:192.168.1.1:23   ::ffff:192.168.1.128:48678 ESTABLISHED 338/telnetd
udp        0      0 0.0.0.0:9999            0.0.0.0:*                           354/infosvr
udp        0      0 127.0.0.1:38032         0.0.0.0:*                           347/nas
udp        0      0 0.0.0.0:42000           0.0.0.0:*                           341/eapd
udp        0      0 127.0.0.1:42032         0.0.0.0:*                           373/acsd
udp        0      0 0.0.0.0:5474            0.0.0.0:*                           386/u2ec
udp        0      0 0.0.0.0:18018           0.0.0.0:*                           321/wanduck
udp        0      0 0.0.0.0:43000           0.0.0.0:*                           341/eapd
udp        0      0 0.0.0.0:59000           0.0.0.0:*                           341/eapd
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path
unix  5      [ ]         DGRAM                       739 170/syslogd         /dev/log
unix  2      [ ]         DGRAM                      1693 353/crond           
unix  2      [ ]         DGRAM                      1540 326/haveged         
unix  2      [ ]         DGRAM                       866 172/klogd           

Nothing should exist on the listen addresses.
Where is the code that does the actual socket operations located? That may give us more of a clue.

[john9527]

Well, now I'm confused again. I added some diagnostic prints in the add_listeners routine in server.c, and none of the error checks tripped. Is there another place that I missed?

[saradickinson]

I suspect the error is hit in the getdns_context_set_listen_addresses() routine before it calls add_listeners.

[john9527]

After installing strace and adding some more diagnostic code, I agree. In fact, it seems as if most(all?) of the getdns_context functions are failing. This is why the results are so varied depending on the contents of the yml/conf file.

I'm just starting to dig further, but it seems as if there are two things unique to these functions

• they make use of memory functions. I see you seem to add a wrapper around things like malloc. Can you give me some insight as to what you are doing here?
• they also depend heavily on a request handler

Did you possibly run into anything here with respect to other platform support that might give me a further clue?

Thanks again for your help.

[john9527]

Sorry about the inadvertent close....definitely still working the issue.

[saradickinson]

@john9527 What you are seeing does seem odd indeed. I'm not aware of any other platform we have run into these problems with but @wtoorop might know more....

As for the memory functions, the original API specification included the use of custom memory management functions so the implementation is such that it uses malloc, etc. as the default functions.

[john9527]

Found it!

getaddrinfo is broken in some linux releases. Usually, this is addressed via an m4 macro, but the version in your distribution doesn't check for linux release dependancies. I would like to suggest you look at the m4 macros packaged with wget which I know handle it properly.

getaddrinfo is called in two places in stubby, context.c and server.c
As a workaround, explicitly specify
hints.ai_socktype = SOCK_STREAM

Thanks again for taking the time to respond.

[john9527]

Just thought I'd pass on that I have formally released the DoT/Stubby support on my Asuswrt-Merlin LTS fork. You can see the user interface I provide here...
https://www.snbforums.com/threads/fork-asuswrt-merlin-374-43-lts-releases-v34e3.18914/page-398#post-426091

[saradickinson]

Closing - thanks again for this work!