[BUG] user-ca-bundle.json: json: offset 1425: invalid character '\r' in string literal

[maxmoser1]

General information

• OS: Windows
• Hypervisor: Hyper-V
• Did you run crc setup before starting it? Yes
• Running CRC on: Laptop

CRC version

CRC version: 2.24.1+81aeeb
OpenShift version: 4.13.6
Podman version: 4.4.4

CRC status

crc status --log-level debug
time="2023-08-05T18:58:06+02:00" level=warning msg="Failed to add proxy CA to crc http transport"
DEBU CRC version: 2.24.1+81aeeb
DEBU OpenShift version: 4.13.6
DEBU Podman version: 4.4.4
DEBU Running 'crc status'
CRC VM:          Running
OpenShift:       Unreachable (v4.13.6)
RAM Usage:       6.61GB of 15.34GB
Disk Usage:      16.39GB of 32.68GB (Inside the CRC VM)
Cache Usage:     22.7GB
Cache Directory: C:\Users\max\.crc\cache

CRC config

- consent-telemetry                     : no
- cpus                                  : 8
- http-proxy                            : http://user@domain:PASS...@1.2.3.4:8080
- https-proxy                           : http://user@domain:PASS...@1.2.3.4:8080
- ingress-http-port                     : 8080
- memory                                : 15000
- nameserver                            : 10.0.0.1
- proxy-ca-file                         : c:\temp\proxy.crt

Host Operating System

Microsoft Windows 10 Enterprise
10.0.19044 Build 19044

Steps to reproduce

crc start  --log-level debug

Expected

crc start completes without errors.

Actual

Failed to update cluster proxy configuration: Failed to add proxy cert details ssh command error:
command : timeout 30s oc apply -f /tmp/user-ca-bundle.json --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
: error: error parsing /tmp/user-ca-bundle.json: json: offset 1425: invalid character '\r' in string literal

Similar bugreport from the past: #1957

Logs

https://gist.github.com/maxmoser1/903373202dda41558a6b9a9eb6be11cc

Before gather the logs try following if that fix your issue

$ crc delete -f
$ crc cleanup
$ crc setup
$ crc start --log-level debug

Tried, but did not help.

[gbraad]

Would it be possible to log in to the VM and share the content of /tmp/user-ca-bundle.json ?

[maxmoser1]

Is it possible to submit the file in a private Red Hat Support Ticket?
If not I'll generate a new cert that I will be able to share publicly (even if it is just about the public part).

[cfergeau]

This code

crc/pkg/crc/cluster/cluster.go

Lines 388 to 391 in 97705a0

	`
	// Replace the carriage return ("\n" or "\r\n") with literal `\n` string
	re := regexp.MustCompile(`\r?\n`)
	p := fmt.Sprintf(proxyCABundleTemplate, re.ReplaceAllString(proxy.ProxyCACert, `\n`), trustedCAName)

replaces \r\n with \n, but according to the error message there are some \r left after doing this.

Could you check your cert file for occurrences of \r, in particular occurrences which are not followed by \n? https://stackoverflow.com/questions/1446370/text-editor-which-shows-r-n has some suggestions to see these special chars on Windows.

[gbraad]

replaces \r\n with \n, b

@cfergeau according to the comment it

Replace the carriage return ("\n" or "\r\n") with the string appended with \n

To what I understand, this matches 'once' all the options:

• optionally match '\r'
• Exactly once match '\n'

So what if the file contained multiple matches of a newline?

[cfergeau]

The code uses re.ReplaceAllString which should replace all occurrences.

[maxmoser1]

This is how the certificate file looks like in notepad++ after enabling "show all characters" - so I would say nothing unusual:

(content is censored as the black block)

[maxmoser1]

I also copied the user-ca-bundle.json via scp to the Windows host and opened in notepad++.
The highlighted part (yellow) looks strange, no?

CR"LF

[maxmoser1]

This is how the certificate file looks like in notepad++ after enabling "show all characters" - so I would say nothing unusual:

(content is censored as the black block)

I managed to work around the issue by removing line 23 (the last empty line).
This should give you a good hint as to where the root cause of this issue is so it can be fixed for other users.

[cfergeau]

I managed to work around the issue by removing line 23 (the last empty line).

What does the file looks like after removing this 'empty' line? and/or on line 22 disappears?
The on line 4 of user-ca-bundle.json is what is causing the problem. Are there more occurences in line 4 or is this the only one?

I tried to reproduce with data similar to yours ( cacert := "----BEGIN CERTIFICATE----\r\nXXXXV\r\nXXXX3\r\nXXXXF\r\nXXXXM=\r\n----END CERTIFICATE----\r\n") but this behaves as expected - no extra /r.

[maxmoser1]

This is how the certificate file looks like with the "workaround" applied:

• no empty line 23
• no CRLF in line 22

[maxmoser1]

The on line 4 of user-ca-bundle.json is what is causing the problem. Are there more occurences in line 4 or is this the only one?

Did you mean

The "one" line 4 of ...

there?

[praveenkumar]

I found out the issue and able to reproduce it, looks like our trimTrailingEOL need to be updated for \r\n instead just for \n . I am going to update this function and also try to add test case around it even replacing \r\n => \n should be part of httpproxy package.

package main

import (
	"fmt"
	"strings"
)

func parseProxyCAData() (string, error) {
	rawData := "-----BEGIN CERTIFICATE-----\r\nMIID6zCCAtOgAwIB\r\nKMGinNDOEwKTFXf\r\n-----END CERTIFICATE-----\r\n\r\n"
	return trimTrailingEOL(rawData), nil
}

func trimTrailingEOL(s string) string {
	fmt.Printf("s: %#v", s)
	s1 := strings.TrimRight(s, "\n")
	fmt.Println()
	fmt.Printf("s1: %#v", s1)
	return s1
}

func main() {
	_, err := parseProxyCAData()
	if err != nil {
		fmt.Printf("error: %v", err)
	}
}

Output

$ go run main.go 
s: "-----BEGIN CERTIFICATE-----\r\nMIID6zCCAtOgAwIB\r\nKMGinNDOEwKTFXf\r\n-----END CERTIFICATE-----\r\n\r\n"
s1: "-----BEGIN CERTIFICATE-----\r\nMIID6zCCAtOgAwIB\r\nKMGinNDOEwKTFXf\r\n-----END CERTIFICATE-----\r\n\r" <= here is the extra \r