net: LookupTXT bug

[BangtianZhou]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.11 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

linux, ubantu

What did you do?

Previously(go version go1.10.4) worked net.LookupTXT function gives wrong and different result but the difference cannot be found in any release announcements.

The runnable code is below

package main

import (
	"fmt"
	"net"
)

func main() {
	result, err := net.LookupTXT("stanford.edu")
	if err != nil {
		panic(err)
	}
	for i, txt := range result {
		fmt.Printf("result[%2d]: %s\n", i, txt)
	}
}

or you can find it on https://play.golang.org/p/-HtW1Zn3gFN.
It is not runnable on playground because it tries DNS lookups. It is just a place to share the code.

What did you expect to see?

It expect to see (go version 1.10.4 result, correct result)

result[ 0]: blitz=mu-0e24b17b-23dce65a-c0b14b75-e742cd66
result[ 1]: bwZknJMLNV1XuaJAyUmKlAFrdZo+p5yDlTNACmDUWhgtyihfJc8oWMnK7hWLreN+ozU3mX91yHHzZx0adJPPPg==
result[ 2]: firebase=cartalab
result[ 3]: v=spf1 ip4:171.67.219.64/27 ip4:171.67.224.0/28 ip4:171.67.214.160/27 ip4:171.67.216.240/28 ip4:171.67.43.137 ip4:171.67.43.138 ip4:171.67.43.139 ip4:171.67.43.140 ip4:171.67.43.141 ip4:204.63.229.192/28 include:_spf.google.com include:_spf.qualtrics.com include:icpbounce.com ip4:148.163.149.245 ip4:148.163.153.235 ?all
result[ 4]: globalsign-domain-verification=IKP42b_14E6c1gUHTYKgXqj3o5aasL2hd9ECM1tSJP
result[ 5]: htBOpqv5il135Xfa09GRiVHY09Xb9qXNj6o5lAfj8gNoBvyyGu5rSb4gvSj9lCUwY1NF0+wL2BO1ZDIcGIpADQ==
result[ 6]: google-site-verification=UE66cfsA35PT37pENcs3Sho_gWa6UFC015Iudr6JU7I

When digging the DNS,

$ dig txt stanford.edu

Record(correct answer from DNS lookups) shows:

; <<>> DiG 9.11.2-P1-1ubuntu5-Ubuntu <<>> txt stanford.edu
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30787
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 7, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;stanford.edu. IN TXT

;; ANSWER SECTION:
stanford.edu. 1800 IN TXT "globalsign-domain-verification=IKP42b_14E6c1gUHTYKgXqj3o5aasL2hd9ECM1tSJP"
stanford.edu. 1800 IN TXT "v=spf1 ip4:171.67.219.64/27 ip4:171.67.224.0/28 ip4:171.67.214.160/27 ip4:171.67.216.240/28 ip4:171.67.43.137 ip4:171.67.43.138 ip4:171.67.43.139 ip4:171.67.43.140 ip4:171.67.43.141 ip4:204.63.229.192/28 " "include:_spf.google.com include:_spf.qualtrics.com include:icpbounce.com ip4:148.163.149.245 ip4:148.163.153.235 ?all"
stanford.edu. 1800 IN TXT "firebase=cartalab"
stanford.edu. 1800 IN TXT "bwZknJMLNV1XuaJAyUmKlAFrdZo+p5yDlTNACmDUWhgtyihfJc8oWMnK7hWLreN+ozU3mX91yHHzZx0adJPPPg=="
stanford.edu. 1800 IN TXT "htBOpqv5il135Xfa09GRiVHY09Xb9qXNj6o5lAfj8gNoBvyyGu5rSb4gvSj9lCUwY1NF0+wL2BO1ZDIcGIpADQ=="
stanford.edu. 1800 IN TXT "google-site-verification=UE66cfsA35PT37pENcs3Sho_gWa6UFC015Iudr6JU7I"
stanford.edu. 1800 IN TXT "blitz=mu-0e24b17b-23dce65a-c0b14b75-e742cd66"

;; Query time: 659 msec
;; SERVER: 9.9.9.9#53(9.9.9.9)
;; WHEN: Wed Sep 19 17:19:35 EDT 2018
;; MSG SIZE rcvd: 832

What did you see instead?

Instead I see the following as the output (go version 1.11 result)

result[ 0]: bwZknJMLNV1XuaJAyUmKlAFrdZo+p5yDlTNACmDUWhgtyihfJc8oWMnK7hWLreN+ozU3mX91yHHzZx0adJPPPg==
result[ 1]: firebase=cartalab
result[ 2]: blitz=mu-0e24b17b-23dce65a-c0b14b75-e742cd66
result[ 3]: htBOpqv5il135Xfa09GRiVHY09Xb9qXNj6o5lAfj8gNoBvyyGu5rSb4gvSj9lCUwY1NF0+wL2BO1ZDIcGIpADQ==
result[ 4]: v=spf1 ip4:171.67.219.64/27 ip4:171.67.224.0/28 ip4:171.67.214.160/27 ip4:171.67.216.240/28 ip4:171.67.43.137 ip4:171.67.43.138 ip4:171.67.43.139 ip4:171.67.43.140 ip4:171.67.43.141 ip4:204.63.229.192/28
result[ 5]: include:_spf.google.com include:_spf.qualtrics.com include:icpbounce.com ip4:148.163.149.245 ip4:148.163.153.235 ?all
result[ 6]: globalsign-domain-verification=IKP42b_14E6c1gUHTYKgXqj3o5aasL2hd9ECM1tSJP
result[ 7]: google-site-verification=UE66cfsA35PT37pENcs3Sho_gWa6UFC015Iudr6JU7I

The result 4 and 5 should be conbined to one record.

This bug is the same as the test failure in the release build log:

--- FAIL: TestLookupLongTXT (0.01s)
lookup_test.go:342: LookupTXT golang.rsc.io incorrect
have ["gophers rule"]
want ["abcdefghijklmnopqrstuvwxyABCDEFGHJIKLMNOPQRSTUVWXYabcdefghijklmnopqrstuvwxyABCDEFGHJIKLMNOPQRSTUVWXYabcdefghijklmnopqrstuvwxyABCDEFGHJIKLMNOPQRSTUVWXYabcdefghijklmnopqrstuvwxyABCDEFGHJIKLMNOPQRSTUVWXYabcdefghijklmnopqrstuvwxyABCDEFGHJIKLMNOPQRSTUVWXYabcdefghijklmnopqrstuvwxyABCDEFGHJIKLMNOPQRSTUVWXYabcdefghijklmnopqrstuvwxyABCDEFGHJIKLMNOPQRSTUVWXYabcdefghijklmnopqrstuvwxyABCDEFGHJIKLMNOPQRSTUVWXYabcdefghijklmnopqrstuvwxyABCDEFGHJIKLMNOPQRSTUVWXYabcdefghijklmnopqrstuvwxyABCDEFGHJIKLMNOPQRSTUVWXY" "gophers rule"]
FAIL
FAIL net 10.402s

[BangtianZhou]

This bug is not only for spf records, DKIM selectors using multiple strings in one record also shows the same problem. Take Facebook selector as example:
code shows

result[ 0]: k=rsa; t=s; h=sha256; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDAUemE56fSDRo+H9Cu8u0uEIOXKON0YbB5A10wuWvNc7bUFIjL0tiUIzhktZjhAXc
result[ 1]: 5CWw2/TZnTZaLZtmtJ2MRfd2e+ty7LylkRAiZUWaT3dcDGVVibWn27DIz3+oCnbL7CFiLzxCZnxHx8B7BC/UM7UCCJMrAgaJWJR6tYwz0MwIDAQAB

Same here the two result combined is the correct record. They are not separate.

Reference: https://tools.ietf.org/html/rfc1464

Related issues: https://github.com/golang/go/issues/21472, https://github.com/golang/go/issues/22857

[BangtianZhou]

The code change I found related to this issue is
https://github.com/golang/go/commit/672729ebbd15e0b0dfac1ba22e35e92557215e1a

if the the code change in lookupTXT is reverted, this issue should be solved.

After that, the code to pass the fail can be removed. https://github.com/golang/go/commit/dcedcaa5fb15748cc2e5cb9461fc6b4f4fc942cb

[agnivade]

/cc @mikioh

[mwwaters]

The issue is multiple strings in one TXT record. The TXT record is one or more character strings, which are a one-byte length followed by that number of bytes as the string. There is a long discussion in #21472 about multiple strings in a TXT record. Windows native resolver gave two strings in one TXT record as two TXT records.

I looked back in go1.7.6 and 2+ strings in one TXT records were simply concatenated. The dnsmessage parser, for a single TXTResource, returns a slice of strings. lookupTXT appends the slice. So 7 TXT records with 8 total TXT strings is returned as 8 strings.

The fix to replicate previous behavior is trivial. lookupTXT needs to concatenate the txt.TXT strings slice and then add that string to the returned slice.

[gopherbot]

Change https://golang.org/cl/136955 mentions this issue: net: concatenate multiple TXT strings in single TXT record

[ianlancetaylor]

See also #21472.

[BangtianZhou]

@mwwaters @agnivade @ianlancetaylor Thank you for your guys. I tested the patch in Change https://golang.org/cl/136955. It fixes the bug. LookupTXT now gives the expected result.
Hope it will go into 1.11 point versions as soon as possible.

[iangudger]

@gopherbot please consider this for backport to 1.11, it's a regression.

[gopherbot]

Backport issue(s) opened: #27886 (for 1.11).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

[gopherbot]

Change https://golang.org/cl/138177 mentions this issue: [release-branch.go1.11] net: concatenate multiple TXT strings in single TXT record