net/url: strict escaping cause issues while dealing with MDN Compliance APIs.

[buddhika-ranasinghe]

What version of Go are you using (go version)?

$ go version
go version go1.15.7 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\User\AppData\Local\go-build
set GOENV=C:\Users\User\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\User\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\User\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=c:\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\User\AppData\Local\Temp\go-build809586734=/tmp/go-build -gno-record-gcc-switches

What did you do?

I try to utilize Microsoft's Graph API.
What I experienced was the password which I need to get a JWT in below post request fails

https://login.microsoftonline.com/{{TenantID}}/oauth2/v2.0/token

package main

import (
	"fmt"
	"net/url"
)


func main() {
	password := "my!pass*w'-ith(and)%[]"
	
	
	fmt.Println("Password : " + password)
	fmt.Println("Password QueryEscape : " + url.QueryEscape(password))
	fmt.Println("Password PathEscape : " + url.PathEscape(password))

	fmt.Println("Expected Output : my!pass*w'-ith(and)%25%5B%5D")
	

}

Output :

$ go run main.go 

Password : my!pass*w'-ith(and)%[]
Password QueryEscape : my%21pass%2Aw%27-ith%28and%29%25%5B%5D
Password PathEscape : my%21pass%2Aw%27-ith%28and%29%25%5B%5D
Expected Output : my!pass*w'-ith(and)%25%5B%5D

My experience with postman can get the token without any issue.
But when I do the same in Go Native I face the issue since we need to encode the password to generate x-www-form-urlencode. I believe this happens because Go is strict rfc-3986 compliance. But this could lead to many of the API Services built with JS will fail to communicate with Go applications since they use encodeURIComponent()

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent

Posman Screenshot :

What did you expect to see?

Expected Output : my!pass*w'-ith(and)%25%5B%5D

What did you see instead?

Password QueryEscape : my%21pass%2Aw%27-ith%28and%29%25%5B%5D
Password PathEscape : my%21pass%2Aw%27-ith%28and%29%25%5B%5D

I used below method to fix the issue.
Expect feedback and if this is accurate, also a fix in code.

package main

import (
	"fmt"
	"net/url"
	"strings"
)

func main() {
	password := "my!pass*w'-ith(and)%[]"
	fmt.Println("Password : " + password)

	fmt.Println("Password QueryEscape: " + url.QueryEscape(password))
	fmt.Println("Password PathEscape: " + url.PathEscape(password))

	encoded_pass := url.QueryEscape(password)

	encoded_pass = strings.ReplaceAll(encoded_pass, "%21", "!")
	encoded_pass = strings.ReplaceAll(encoded_pass, "%2A", "*")
	encoded_pass = strings.ReplaceAll(encoded_pass, "%27", "'")
	encoded_pass = strings.ReplaceAll(encoded_pass, "%28", "(")
	encoded_pass = strings.ReplaceAll(encoded_pass, "%29", ")")

	fmt.Println("Password Correct RFC Escape : " + encoded_pass)

}

Output :

Password : my!pass*w'-ith(and)%[]
Password QueryEscape: my%21pass%2Aw%27-ith%28and%29%25%5B%5D
Password PathEscape: my%21pass%2Aw%27-ith%28and%29%25%5B%5D
Password Correct RFC Escape : my!pass*w'-ith(and)%25%5B%5D

[ZekeLu]

From the perspective of the server, my!pass*w'-ith(and)%25%5B%5D and my%21pass%2Aw%27-ith%28and%29%25%5B%5D should be the same. Both of them will be decoded to my!pass*w'-ith(and)%[]. (I have tested with the javascript function decodeURIComponent, and the golang function url.ParseQuery).

So I don't think that url.QueryEscape or url.PathEscape has any issue here. I guess that maybe you escaped the password twice (one by you, and one by the HTTP client). Can you provide the source code which composes the HTTP request?

[buddhika-ranasinghe]

Hi @ZekeLu

First of all, thanks for your attention on this.
Below is the code which I use to send the HTTP POST Script.

payload := strings.NewReader("grant_type=password&client_id=" + os.Getenv("CLIENTID") + "&client_secret=" + os.Getenv("CLIENTSECRET") + "&scope=" + os.Getenv("SCOPE") + "&userName=" + os.Getenv("USERNAME") + "&password=" + utils.JSEncodeURL(os.Getenv("PASSWORD")))

request, err := http.NewRequest("POST", os.Getenv("LOGIN_URL")+os.Getenv("TENANTID")+"/oauth2/v2.0/token", payload)

if err != nil {
	return "", errors.New("Faile while creating the HTTP Request. | " + err.Error())
}

request.Header.Set("Content-Type", "application/x-www-form-urlencoded")

client := http.Client{}
resp, err := client.Do(request)

if err != nil {
	return "", errors.New("Failed while performing the HTTP Request. | " + err.Error())
}

Thanks

[ZekeLu]

You do not escape all the parameters. I guess it fails because some of the parameters contain special characters. Please have a test to see whether http.PostForm works for you:

f := url.Values{
	"grant_type":    {"password"},
	"client_id":     {os.Getenv("CLIENTID")},
	"client_secret": {os.Getenv("CLIENTSECRET")},
	"scope":         {os.Getenv("SCOPE")},
	"userName":      {os.Getenv("USERNAME")},
	"password":      {os.Getenv("PASSWORD")},
}

// compose the URL like this is dangerous too
resp, err := http.PostForm(os.Getenv("LOGIN_URL")+os.Getenv("TENANTID")+"/oauth2/v2.0/token", f)

[buddhika-ranasinghe]

Hi @ZekeLu

After replace characters it worked fine. But this http.PostForm method gave same result back from API.

I'm not sure this is something related to GRAPH API.

But once i replaced the escaped charaters back with unscaped as mentioned above, this http request works without any issu.

Thanks
Buddhika

[ZekeLu]

@buddhika-ranasinghe Thanks for the clarification. It's very likely that this is an issue of the GRAPH API. Can you raise the issue to the GRAPH API owner and report back once you got an answer?

[buddhika-ranasinghe]

@ZekeLu

Thanks a lot for your support on this.
I'm not sure I can raise an issue with Microsoft Graph API. I'll try to search for Microsoft's Graph APIs support link to raise a ticket.
Let's keep this ticket open to see if someone has any other opinion on the same.

Thanks
Buddhika

[dr2chase]

Does anyone here believe that this is still a Go bug?

[seankhliao]

cc @neild @rsc for net/url
though I believe this is purely a microsoft issue for not adhering to spec

[ZekeLu]

@buddhika-ranasinghe I have set up an Azure account to test this issue, and both password formats work (please note that my!pass*w'-ith(and)%[] is not allowed by Azure because it's not complicated enough. I have prefixed it with the character 2 to make it accepted). And http.PostForm works too. So I'm pretty sure there must be something wrong in your code.

Please examine your posted form body carefully to find out what's wrong. You can send the request with curl and modify the form body directly. It's recommended to put the encoded form body in a file so that you don't have to escape special shell characters.

# post the data in the "form.txt" file in the current directory
curl -i --raw -X POST -d "@form.txt" -H "Content-Type: application/x-www-form-urlencoded" https://login.microsoftonline.com/[TENANT_ID]/oauth2/v2.0/token

And here is an example form.txt file:

client_id=YOUR_CLIENT_ID&client_secret=YOUR_CLIENT_SECRET&grant_type=password&password=2my%21pass%2Aw%27-ith%28and%29%25%5B%5D&scope=user.read&userName=YOUR_USER_NAME

P.S. I noticed that you're using Windows. curl is available in Git for Windows.

[buddhika-ranasinghe]

@ZekeLu Thanks for the suggestion. I can't paste the actual password here. But I can confirm that this is a valid password since once after I replace the characters of encoded string using below code, it works without any issue.

encoded_pass = strings.ReplaceAll(encoded_pass, "%21", "!")
encoded_pass = strings.ReplaceAll(encoded_pass, "%2A", "*")
encoded_pass = strings.ReplaceAll(encoded_pass, "%27", "'")
encoded_pass = strings.ReplaceAll(encoded_pass, "%28", "(")
encoded_pass = strings.ReplaceAll(encoded_pass, "%29", ")")

I have tried the same password with curl using "--data-urlencode" option, it also working.
I still can't confirm this is a Bug in Go as this can be an effect of I'm doing encoding twice.
But I don't see any encoding method I have called. It could be something happens inside Go.
Please refer below code again and let me know if anything I'm doing related to multiple encoding.

payload := strings.NewReader("grant_type=password&client_id=" + os.Getenv("CLIENTID") + "&client_secret=" + os.Getenv("CLIENTSECRET") + "&scope=" + os.Getenv("SCOPE") + "&userName=" + os.Getenv("USERNAME") + "&password=" + utils.JSEncodeURL(os.Getenv("PASSWORD")))

request, err := http.NewRequest("POST", os.Getenv("LOGIN_URL")+os.Getenv("TENANTID")+"/oauth2/v2.0/token", payload)

if err != nil {
	return "", errors.New("Faile while creating the HTTP Request. | " + err.Error())
}

request.Header.Set("Content-Type", "application/x-www-form-urlencoded")

client := http.Client{}
resp, err := client.Do(request)

if err != nil {
	return "", errors.New("Failed while performing the HTTP Request. | " + err.Error())
}

[ZekeLu]

@buddhika-ranasinghe Sorry that I didn't make it clear in the last comment that I want you to test with the raw form body without any encoding/decoding involved.

For example, let's assume that my!pass*w'-ith(and)%25%5B%5D works, then let's replace ! with %21 to test whether it works (using curl --raw and loading the form body from a file, as suggested in my last comment). In fact, I'm very sure that it works. I just want to prove that the Microsoft Graph API can handle it correctly.

Another approach is to use a tool (such as Fiddler) to capture the HTTP traffic, and to see what's the difference between the one that works and the one that doesn't.

[buddhika-ranasinghe]

Hi @ZekeLu

Thanks for all the support you have provided on this. I have identified the issue.
Issue is which I have missed the Content-Length header. After adding the Content-Length header to the request, the issue got sorted. I can confirm there is no issue with Go and Graph API.

Thanks for all the support provided by @ZekeLu and @seankhliao @dr2chase

Kept ticket open to receive any further feedback. I'll close with a comment by another 7 days since this is sorted.

[ZekeLu]

Hey @buddhika-ranasinghe, well done!

I have one question though, http.PostForm should have set the Content-Length header automatically. Why it doesn't work?