mDNS: Do not add ".local" behind the scenes (IDFGH-4787)

[markg85]

Hi,

This whole comment comes from me from #2507.
I was asked by @igrr to open a new bug report for this.

I'm in a linux setup where Avahi is handling mDNS for my pc's.
The pc's themselves can ping each other just fine using <their name>.local.
The ESP8266 could reach those same machines just fine too with mDNS.

I took a little excursion into the code along with wireshark and found one interesting clue that confuses everyone.
When i, in code, do a mdns lookup for foo.local (so the .local included!) then wireshark tells me that the lookup was actually for foo.local.local! Note the double ".local"!

The "fix" might seem to not include the ".local" in my host.
If i then do a mdns lookup using some espressif sample code for a dns lookup, it magically works.

However, if i now pass in these hostnames without ".local", other parts of the code apparently think i want to do a DNS lookup as opposed to an mDNS lookup. And as i don't have those hostnames in my network without the .local suffix those lookups fail in WiFiGeneric.cpp line 654:
[E][WiFiGeneric.cpp:654] hostByName(): DNS Failed for foo

That ".local" is added internally is confirmed in ESPmDNS.h itself in it's comment block:

Usage:
- Include the ESP32 Multicast DNS library in the sketch.
- Call the begin method in the sketch's setup and provide a domain name (without
  the '.local' suffix, i.e. just provide 'foo' to resolve 'foo.local'), and the
  Adafruit CC3000 class instance.  Optionally provide a time to live (in seconds)
  for the DNS record--the default is 1 hour.

Other libraries (i'm in particular having troubles with the PubSubClient library) connect to the hostname via a passed in Client object. That is often a WiFiClient where eventually this code is called:

int WiFiClient::connect(const char *host, uint16_t port, int32_t timeout)
{
    IPAddress srv((uint32_t)0);
    if(!WiFiGenericClass::hostByName(host, srv)){
        return 0;
    }
    return connect(srv, port, timeout);
}

As you can see in the snippet, the name is resolved by WiFiGenericClass::hostByName which looks like this:

int WiFiGenericClass::hostByName(const char* aHostname, IPAddress& aResult)
{
    ip_addr_t addr;
    aResult = static_cast<uint32_t>(0);
    waitStatusBits(WIFI_DNS_IDLE_BIT, 5000);
    clearStatusBits(WIFI_DNS_IDLE_BIT);
    err_t err = dns_gethostbyname(aHostname, &addr, &wifi_dns_found_callback, &aResult);
    if(err == ERR_OK && addr.u_addr.ip4.addr) {
        aResult = addr.u_addr.ip4.addr;
    } else if(err == ERR_INPROGRESS) {
        waitStatusBits(WIFI_DNS_DONE_BIT, 4000);
        clearStatusBits(WIFI_DNS_DONE_BIT);
    }
    setStatusBits(WIFI_DNS_IDLE_BIT);
    if((uint32_t)aResult == 0){
        log_e("DNS Failed for %s", aHostname);
    }
    return (uint32_t)aResult != 0;
}

At this point mDNS is nowhere to be found. The Lookup is done in regular DNS and therefore fails in my case. Remember, if i omit the .local i get a name that isn't known in my network. It must be ending with .local here.

I think there's at least 1 bug here, possible 2.

1. that '.local' should not be added behind the scenes. It should be visible. Thus you should just provide "foo.local" instead of "foo".
2. Perhaps just having this already fixes everything. If not, then somewhere in the host name handling mDNS should be checked for. I don't see it happening in the ESP32 not the ESP8266 code yet it's working in the ESP8266 and not in ESP32.

[negativekelvin]

@markg85 if you want general dns lookup to support mdns then lwip has to be compiled with LWIP_DNS_SUPPORT_MDNS_QUERIES

As for the esp-idf mdns component limitations you could open a new feature request to allow domain name to be specified at runtime as well as multiple instances of mdns

In the other issue it was mentioned that it probably works on esp8266 because of LWIP_DNS_SUPPORT_MDNS_QUERIES espressif/arduino-esp32#3822 (comment). Using lwip as a general dns resolver including for mdns (lwip also only supports .local) is the most practical and fastest solution.

The esp-idf mdns component is based around a single instance tied to a single domain and hardcoded to .local. This is definitely a limitation but I don't think it was ever designed to be part of a general resolver. There should be changes to this component and the easiest one would be don't append .local if the hostname already ends in .local which probably won't do any harm. We'll have to see what espressif thinks about other breaking changes to this component.

[david-cermak]

Hi @markg85

Thanks for the report. I agree that the internal mDNS has some limitations, hardcoded .local suffix one of them. We could possibly add a new API for querying the full hostname foo.local, or as @negativekelvin suggested accepting both. We cannot break the current API, though.

Also agree the the easiest option is to use the lwip resolver enabling LWIP_DNS_SUPPORT_MDNS_QUERIES. This is actually what that latest IDF does when resolving local link names (from the v4.1, see the below for your reference)

• Release notes on v4.1
• lwip resolver
• mdns service

So I would suggest using lwIP's mdns module for resolving hostnames, and internal IDF's mdns for advertising/resolving services and correct conflict resolutions per mDNS/bonjour spec (where the lwip's implementation might fail).

[markg85]

I am curious to know why you folks changed this compared to ESP8266.
Now you're in a mess that you can't easily get out of again.. Not without breaking backwards compatibility. Though i'd argue that anyone relying on the current logic is relying on a bug...

Anyhow, what's the proper way to get LWIP_DNS_SUPPORT_MDNS_QUERIES enabled? I'm using vscode with platformio. Am i correct in assuming that i should just add -DLWIP_DNS_SUPPORT_MDNS_QUERIES=1 to my projects build flags and rebuild?

[markg85]

Any help please?

I'm still stuck in this very same problem.
This is how my platformio.ini looks like:

[env:esp32dev]
platform = espressif32
board = esp32dev
framework = arduino
monitor_speed = 115200
;upload_port = /dev/ttyUSB0
lib_deps = 
	https://github.com/tzapu/WiFiManager.git#development
	adafruit/Adafruit INA260 Library@^1.3.5

build_flags = 
    -DWM_MDNS=1
    -DLWIP_DNS_SUPPORT_MDNS_QUERIES=1
    -DESP32=1
    -DARDUINO_ARCH_ESP32=1
    -DBOARD_HAS_PSRAM
    -std=c++17
    -std=gnu++17
build_unflags =
    -std=gnu++11
monitor_flags = 
	--eol
	LF

Right now i can only make a connection via IP to a host that is reachable through mDNS but does not have a "regular dns".
So i want my ESP32 to behave exactly like the ESP8266 behaves in name resolution. How do i do that?

During compilation i do see -DLWIP_DNS_SUPPORT_MDNS_QUERIES=1 and -DWM_MDNS=1 being passed so i'm really not sure what i'm doing wrong. It should work, right?

Edit:
It might be of importance to know that i'm using https://github.com/espressif/arduino-esp32 with the default settings. That currently means i'm using version 3.2.0

[igrr]

@markg85 I suppose this won't work, since this macro needs to be defined at the time when LwIP is compiled. In Arduino, all ESP-IDF libraries are already pre-compiled using https://github.com/espressif/esp32-arduino-lib-builder/. It looks like in the latest version, CONFIG_LWIP_DNS_SUPPORT_MDNS_QUERIES is already set: https://github.com/espressif/esp32-arduino-lib-builder/blob/d491d3a565f016d86aa474098a5abf2e88dd0df8/sdkconfig.esp32#L954. It is also enabled in the master branch of arduino-esp32: https://github.com/espressif/arduino-esp32/blob/7856de7a57420e494176c16c5138174fe2c1dad0/tools/sdk/esp32/sdkconfig#L954. I suppose that if you use the latest version of arduino-esp32 repository, you don't need to enable this separately. I'm sorry I can't help with platformio usage, though.

[markg85]

@markg85 I suppose this won't work, since this macro needs to be defined at the time when LwIP is compiled. In Arduino, all ESP-IDF libraries are already pre-compiled using https://github.com/espressif/esp32-arduino-lib-builder/. It looks like in the latest version, CONFIG_LWIP_DNS_SUPPORT_MDNS_QUERIES is already set: https://github.com/espressif/esp32-arduino-lib-builder/blob/d491d3a565f016d86aa474098a5abf2e88dd0df8/sdkconfig.esp32#L954. It is also enabled in the master branch of arduino-esp32: https://github.com/espressif/arduino-esp32/blob/7856de7a57420e494176c16c5138174fe2c1dad0/tools/sdk/esp32/sdkconfig#L954. I suppose that if you use the latest version of arduino-esp32 repository, you don't need to enable this separately. I'm sorry I can't help with platformio usage, though.

Thank you for that!
That does help me one step further :)

platform_packages =
    platformio/framework-arduinoespressif32@https://github.com/espressif/arduino-esp32.git

That's how to get the master branch in platformio.ini.

Now i'm using WiFiManager which thankfully has a branch to work with the master of arduino-esp32 but even so, the compile fails with:

-I/home/mark/.platformio/packages/framework-arduinoespressif32@src-537c58760dafe7fcc8a1d9bbcf00b6f6/tools/sdk/esp32/include/e/home/mark/.platformio/packages/toolchain-xtensa32/bin/../lib/gcc/xtensa-esp32-elf/5.2.0/../../../../xtensa-esp32-elf/bin/ld: cannot find crt1-sim.o: No such file or directory
s/home/mark/.platformio/packages/toolchain-xtensa32/bin/../lib/gcc/xtensa-esp32-elf/5.2.0/../../../../xtensa-esp32-elf/bin/ld: cannot find _vectors.o: No such file or directory

That doesn't help :(

[igrr]

This doesn't look right indeed, as the toolchain being used is based on GCC 5.2. For the recent versions of IDF it should be based on GCC 8.4. I do see the build script at https://github.com/espressif/arduino-esp32/blob/master/tools/platformio-build-esp32.py, but I don't know where the dependency on the toolchain is declared, or how the newer toolchain can be installed. Do you mind opening an issue about the failing build with platformio at https://github.com/espressif/arduino-esp32/issues?

[markg85]

I was about to before i realized that i need to specify that toolchain too :)
So i have this now:

platform_packages =
    toolchain-xtensa32@3.80200.200512
    platformio/framework-arduinoespressif32@https://github.com/espressif/arduino-esp32.git

It does compile! Just doesn't link now :)

/home/mark/.platformio/packages/toolchain-xtensa32/bin/../lib/gcc/xtensa-esp32-elf/8.2.0/../../../../xtensa-esp32-elf/bin/ld: /home/mark/.platformio/packages/framework-arduinoespressif32@src-537c58760dafe7fcc8a1d9bbcf00b6f6/tools/sdk/esp32/lib/libesp_system.a(startup.c.obj):(.iram1.29.literal+0x10): undefined reference to `_Unwind_SetEnableExceptionFdeSorting'
/home/mark/.platformio/packages/toolchain-xtensa32/bin/../lib/gcc/xtensa-esp32-elf/8.2.0/../../../../xtensa-esp32-elf/bin/ld: /home/mark/.platformio/packages/framework-arduinoespressif32@src-537c58760dafe7fcc8a1d9bbcf00b6f6/tools/sdk/esp32/lib/libesp_system.a(startup.c.obj): in function `__esp_system_init_fn_init_components0':
/home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/build/../esp-idf/components/esp_system/startup.c:450: undefined reference to `_Unwind_SetEnableExceptionFdeSorting'

Is that an issue for the arduino-esp32 repo or for this current one?

[igrr]

This looks like toolchain version mismatch... Unfortunately I don't know which toolchain version does platformio package toolchain-xtensa32@3.80200.200512 contain. Given the "80200" part, I'd hazard a guess that it is a GCC 8.2 based toolchain package, so probably not the latest release, which is at the moment esp-2020r3 (https://github.com/espressif/crosstool-NG/releases).

[markg85]

Fixed, finally :)
I needed:

platform_packages = 
	toolchain-xtensa32@~2.80400.0
	platformio/framework-arduinoespressif32@https://github.com/espressif/arduino-esp32.git

I was close to that in one attempt. I just had toolchain-xtensa32@2.80400.0 note the missing tilde sign..

Now i have the latest framework + toolchain. And to confirm my initial issue, it's fixed in this. I can just set a .local and it resolves!

Thanx to Max from https://community.platformio.org/t/how-to-recompile-framework-or-update-it