Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] user-ca-bundle.json: json: offset 1425: invalid character '\r' in string literal #3785

Closed
maxmoser1 opened this issue Aug 5, 2023 · 12 comments · Fixed by #3799
Closed
Assignees
Labels
kind/bug Something isn't working status/need more information Issue needs more information before it will be looked at

Comments

@maxmoser1
Copy link

General information

  • OS: Windows
  • Hypervisor: Hyper-V
  • Did you run crc setup before starting it? Yes
  • Running CRC on: Laptop

CRC version

CRC version: 2.24.1+81aeeb
OpenShift version: 4.13.6
Podman version: 4.4.4

CRC status

crc status --log-level debug
time="2023-08-05T18:58:06+02:00" level=warning msg="Failed to add proxy CA to crc http transport"
DEBU CRC version: 2.24.1+81aeeb
DEBU OpenShift version: 4.13.6
DEBU Podman version: 4.4.4
DEBU Running 'crc status'
CRC VM:          Running
OpenShift:       Unreachable (v4.13.6)
RAM Usage:       6.61GB of 15.34GB
Disk Usage:      16.39GB of 32.68GB (Inside the CRC VM)
Cache Usage:     22.7GB
Cache Directory: C:\Users\max\.crc\cache

CRC config

- consent-telemetry                     : no
- cpus                                  : 8
- http-proxy                            : http://user@domain:PASS...@1.2.3.4:8080
- https-proxy                           : http://user@domain:PASS...@1.2.3.4:8080
- ingress-http-port                     : 8080
- memory                                : 15000
- nameserver                            : 10.0.0.1
- proxy-ca-file                         : c:\temp\proxy.crt

Host Operating System

Microsoft Windows 10 Enterprise
10.0.19044 Build 19044

Steps to reproduce

crc start  --log-level debug

Expected

crc start completes without errors.

Actual

Failed to update cluster proxy configuration: Failed to add proxy cert details ssh command error:
command : timeout 30s oc apply -f /tmp/user-ca-bundle.json --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
: error: error parsing /tmp/user-ca-bundle.json: json: offset 1425: invalid character '\r' in string literal

Similar bugreport from the past: #1957

Logs

https://gist.github.com/maxmoser1/903373202dda41558a6b9a9eb6be11cc

Before gather the logs try following if that fix your issue

$ crc delete -f
$ crc cleanup
$ crc setup
$ crc start --log-level debug

Tried, but did not help.

@maxmoser1 maxmoser1 added kind/bug Something isn't working status/need triage labels Aug 5, 2023
@gbraad
Copy link
Contributor

gbraad commented Aug 7, 2023

Would it be possible to log in to the VM and share the content of /tmp/user-ca-bundle.json ?

@praveenkumar praveenkumar added the status/need more information Issue needs more information before it will be looked at label Aug 8, 2023
@maxmoser1
Copy link
Author

Is it possible to submit the file in a private Red Hat Support Ticket?
If not I'll generate a new cert that I will be able to share publicly (even if it is just about the public part).

@cfergeau
Copy link
Contributor

This code

`
// Replace the carriage return ("\n" or "\r\n") with literal `\n` string
re := regexp.MustCompile(`\r?\n`)
p := fmt.Sprintf(proxyCABundleTemplate, re.ReplaceAllString(proxy.ProxyCACert, `\n`), trustedCAName)
replaces \r\n with \n, but according to the error message there are some \r left after doing this.

Could you check your cert file for occurrences of \r, in particular occurrences which are not followed by \n? https://stackoverflow.com/questions/1446370/text-editor-which-shows-r-n has some suggestions to see these special chars on Windows.

@gbraad
Copy link
Contributor

gbraad commented Aug 21, 2023

replaces \r\n with \n, b

@cfergeau according to the comment it

Replace the carriage return ("\n" or "\r\n") with the string appended with \n

To what I understand, this matches 'once' all the options:

  • optionally match '\r'
  • Exactly once match '\n'

So what if the file contained multiple matches of a newline?

@cfergeau
Copy link
Contributor

The code uses re.ReplaceAllString which should replace all occurrences.

@maxmoser1
Copy link
Author

maxmoser1 commented Aug 22, 2023

This is how the certificate file looks like in notepad++ after enabling "show all characters" - so I would say nothing unusual:

cert-crlf (content is censored as the black block)

@maxmoser1
Copy link
Author

I also copied the user-ca-bundle.json via scp to the Windows host and opened in notepad++.
The highlighted part (yellow) looks strange, no?

CR"LF
user-ca-bundle json 2

@maxmoser1
Copy link
Author

This is how the certificate file looks like in notepad++ after enabling "show all characters" - so I would say nothing unusual:

cert-crlf (content is censored as the black block)

I managed to work around the issue by removing line 23 (the last empty line).
This should give you a good hint as to where the root cause of this issue is so it can be fixed for other users.

@cfergeau
Copy link
Contributor

I managed to work around the issue by removing line 23 (the last empty line).

What does the file looks like after removing this 'empty' line? and/or on line 22 disappears?
The on line 4 of user-ca-bundle.json is what is causing the problem. Are there more occurences in line 4 or is this the only one?

I tried to reproduce with data similar to yours ( cacert := "----BEGIN CERTIFICATE----\r\nXXXXV\r\nXXXX3\r\nXXXXF\r\nXXXXM=\r\n----END CERTIFICATE----\r\n") but this behaves as expected - no extra /r.

@maxmoser1
Copy link
Author

This is how the certificate file looks like with the "workaround" applied:

  • no empty line 23
  • no CRLF in line 22
fixed

@maxmoser1
Copy link
Author

The on line 4 of user-ca-bundle.json is what is causing the problem. Are there more occurences in line 4 or is this the only one?

Did you mean

The "one" line 4 of ...

there?

@praveenkumar
Copy link
Member

I found out the issue and able to reproduce it, looks like our trimTrailingEOL need to be updated for \r\n instead just for \n . I am going to update this function and also try to add test case around it even replacing \r\n => \n should be part of httpproxy package.

package main

import (
	"fmt"
	"strings"
)

func parseProxyCAData() (string, error) {
	rawData := "-----BEGIN CERTIFICATE-----\r\nMIID6zCCAtOgAwIB\r\nKMGinNDOEwKTFXf\r\n-----END CERTIFICATE-----\r\n\r\n"
	return trimTrailingEOL(rawData), nil
}

func trimTrailingEOL(s string) string {
	fmt.Printf("s: %#v", s)
	s1 := strings.TrimRight(s, "\n")
	fmt.Println()
	fmt.Printf("s1: %#v", s1)
	return s1
}

func main() {
	_, err := parseProxyCAData()
	if err != nil {
		fmt.Printf("error: %v", err)
	}
}

Output

$ go run main.go 
s: "-----BEGIN CERTIFICATE-----\r\nMIID6zCCAtOgAwIB\r\nKMGinNDOEwKTFXf\r\n-----END CERTIFICATE-----\r\n\r\n"
s1: "-----BEGIN CERTIFICATE-----\r\nMIID6zCCAtOgAwIB\r\nKMGinNDOEwKTFXf\r\n-----END CERTIFICATE-----\r\n\r" <= here is the extra \r

praveenkumar added a commit to praveenkumar/crc that referenced this issue Aug 23, 2023
It will fix the proxy parse issue in case user update the cert on
windows and add blank line at the end of certificate. Unit test is also
added for same.

fixes: crc-org#3785
@praveenkumar praveenkumar self-assigned this Aug 23, 2023
praveenkumar added a commit to praveenkumar/crc that referenced this issue Aug 23, 2023
It will fix the proxy parse issue in case user update the cert on
windows and add blank line at the end of certificate. Unit test is also
added for same.

When the cert file ends with \r\n, trimTrailingEOL would only remove \n
and the file will end with \r. Then this causes issues in regexp which
only replace `\r\n` => `\n` but doesn't perform any action for `\r` and
eventualy turn up this bug.

fixes: crc-org#3785
praveenkumar added a commit that referenced this issue Aug 24, 2023
It will fix the proxy parse issue in case user update the cert on
windows and add blank line at the end of certificate. Unit test is also
added for same.

When the cert file ends with \r\n, trimTrailingEOL would only remove \n
and the file will end with \r. Then this causes issues in regexp which
only replace `\r\n` => `\n` but doesn't perform any action for `\r` and
eventualy turn up this bug.

fixes: #3785
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working status/need more information Issue needs more information before it will be looked at
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

5 participants