Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Authentication failure: Out of memory [$25 awarded] #915

Closed
radonish opened this issue Dec 6, 2019 · 25 comments
Closed

Authentication failure: Out of memory [$25 awarded] #915

radonish opened this issue Dec 6, 2019 · 25 comments
Labels
bounty There is a bounty for this issue

Comments

@radonish
Copy link

radonish commented Dec 6, 2019 •
edited by CendioOssman
Loading

When I attempt to connect to the TigerVNC server using TigerVNC viewer 1.10.0 I get an error "Authentication failure: Out of memory".

I was previously using TigerVNC 1.9.0 viewer successfully and upgraded to 1.10.0 to get past the macOS security warning associated with 1.9.0.

To Reproduce
Steps to reproduce the behavior:

  1. Launch TigerVNC Viewer
  2. Enter VNC Server address
  3. Click Connect
  4. Click "Yes" to continue when warned about the certificate being signed by an unknown authority.
  5. Error window displayed with "Authentication failure: Out of memory"

Expected behavior
A successful connection is established, as it was with TigerVNC Viewer 1.9.0.

Client (please complete the following information):

  • OS: macOS 10.15.2
  • VNC client: TigerVNC
  • VNC client version: 1.10.0
  • Client downloaded from: Bintray

Server (please complete the following information):

  • OS: Red Hat Enterprise Linux 7.4
  • VNC server: TigerVNC
  • VNC server version: 1.9.0, 1.10.0 (both tried)
  • Server downloaded from: Bintray
  • Server was started using: vncserver :${DISPLAY} -rfbport 3389 -httpd /dev/null -SecurityTypes X509plain -X509Cert ~/.vnc/certs/${SHOST}_vnc.crt -X509Key ~/.vnc/certs/${SHOST}_vnc.key -PAMService sshd -PlainUsers ${VNCUSER}

Thank you

The $25 bounty on this issue has been claimed at Bountysource.
@CendioOssman CendioOssman changed the title Authentication failure: Out of memory Authentication failure: Out of memory [$25] Dec 7, 2019
@CendioOssman CendioOssman added the bounty There is a bounty for this issue label Dec 7, 2019
@radonish
Copy link
Author

radonish commented Dec 8, 2019 •
edited
Loading

To add, I get the reported error regardless of whether the server is running 1.9.0 or 1.10.0.

Server log:
Xvnc TigerVNC 1.10.0 - built Nov 18 2019 09:12:45
Copyright (C) 1999-2019 TigerVNC Team and many others (see README.rst)
See https://www.tigervnc.org for information on TigerVNC.
Underlying X server release 12001000, The X.Org Foundation

Sun Dec 8 07:50:20 2019
vncext: VNC extension running!
vncext: Listening for VNC connections on all interface(s), port 3389
vncext: created VNC server for screen 0

Sun Dec 8 07:50:50 2019
Connections: accepted: x.x.x.x::62788
SConnection: Client needs protocol version 3.8
SConnection: Client requests security type VeNCrypt(19)
SVeNCrypt: Client requests security type X509Plain (262)

----Certificate accepted on client side at this point.----

Sun Dec 8 07:50:55 2019
VNCSConnST: closing x.x.x.x::62788: Clean disconnection
EncodeManager: Framebuffer updates: 0
EncodeManager: Total: 0 rects, 0 pixels
EncodeManager: 0 B (1:-nan ratio)
TLS: TLS session wasn't terminated gracefully
Connections: closed: x.x.x.x::62788
ComparingUpdateTracker: 0 pixels in / 0 pixels out
ComparingUpdateTracker: (1:-nan ratio)

@samhed
Copy link
Member

samhed commented Dec 8, 2019

Well, are you out of memory?

@radonish
Copy link
Author

radonish commented Dec 8, 2019 •
edited
Loading

@samhed, nope. I haven't had a chance to look through the source - is the MacOS TigerVNC client ultimately a Java application?

It's awkward that an OOM error exception is propagating up the system as an authentication failure.

@bphinz
Copy link
Member

bphinz commented Dec 8, 2019 via email

@radonish
Copy link
Author

More information; I was able to try with macOS Sierra tonight:
macOS 10.12.6 (Sierra) with TigerVNC Viewer 1.10.0 works fine, as 1.9.0 did.
macOS 10.15.1 (Catalina) with TigerVNC Viewer 1.10.0 gets the "Authentication failure: Out of memory"

Any help is appreciated.

@radonish
Copy link
Author

radonish commented Dec 12, 2019 •
edited
Loading

macOS 10.15.2 results in the same failure.

I do get a warning about the certificate being signed by an unknown authority prior to the failure (I say yes to the "Do you want to save it and continue?" question). Does that shed any light?

(Also updated the original post according to the bug report template.)

@bphinz
Copy link
Member

bphinz commented Dec 12, 2019 via email

@radonish
Copy link
Author

@bphinz, yes, it is the remote server's cert. I am using x509 secType.

Thanks

@radonish radonish mentioned this issue Dec 13, 2019
Closed
@tarun-dholariya
Copy link

Hi, @radonish Asked me to post my answer which I posted on other thread because it will help here as well.
Here is my version:
I have catalina 10.15 macOS installed on my mac,
I was getting error "Authentication failure: Out of memory" with Tiger VNC viewer 1.9
Then I found solution on thread #881
I downloaded TigerVNC-1.10.80.dmg from http://tigervnc.bphinz.com/nightly/ "Mac OS-X 10.9+ packages" section, and it worked fine.
Thanks to @gsmurray who posted the solution.

Below are the steps that were posted by @gsmurray

Further Testing Reveals that the following Method will allow the Application to open:

  1. Download 1.9.80 from http://tigervnc.bphinz.com/nightly/
  2. Click on .dmg ->copy the app in your Applications folder;
  3. Double click shows you "Apple cannot check it for malicious software"
  4. Go to System Preferences > Security & Privacy > General (Tab) > clicked on Open Anyway nearby TigerVPN app;
  5. This will Show the shows you "Apple cannot check it for malicious software" dialog again->Select OK
  6. Attempt to open the Application again by DoubleClicking

@radonish
Copy link
Author

radonish commented Dec 13, 2019 •
edited
Loading

Thank you for posting your details, @tarun-dholariya. To be clear, you had success with 1.10.80? His steps on #881 from October referenced 1.9.80.

Also, what TigerVNC server version are you connecting to from your Mac?

Thanks

@tarun-dholariya
Copy link

tarun-dholariya commented Dec 13, 2019 •
edited
Loading

Oh right.
His steps mentions 1.9.80, but when I opened link http://tigervnc.bphinz.com/nightly/
it has 1.10.80 which worked with my Mac.

To clarify
1.10.80 worked with my Mac

Do not download 1.9.80. (which is not available on the link anyway)

Thanks @radonish for pointing that out.

about VNC server version, let me find out.
PS: vnc server version: tigervnc-server.x86_64 1.8.0-17.el7
Server OS: CentOS 7
Note: i use my MAC TigerVNC viewer with many other servers which might have different vncserver version, I have no problem connecting with any of them.

@tarun-dholariya
Copy link

@radonish
post if my answer works for you.

Thanks.

@radonish
Copy link
Author

Thank you, @tarun-dholariya! I will post a follow up tonight.

@bphinz
Copy link
Member

bphinz commented Dec 13, 2019 via email

@radonish
Copy link
Author

@tarun-dholariya @bphinz, unfortunately I still get an error with 1.10.80.

To be clear, I do not have to do steps 3 and 4 as you did, @tarun-dholariya - I simply hit "open anyway" and retry to successfully launch TigerVNC Viewer. I still end up with the cryptic "Authentication failure: Out of memory" failure, however.

Client log:
TigerVNC Viewer 64-bit v1.10.80
Built on: 2019-12-01 01:57
Copyright (C) 1999-2019 TigerVNC Team and many others (see README.rst)
See https://www.tigervnc.org for information on TigerVNC.

Fri Dec 13 18:26:17 2019
DecodeManager: Detected 6 CPU core(s)
DecodeManager: Creating 4 decoder thread(s)
CConn: Connected to host XXX port 3389
CConnection: Server supports RFB protocol version 3.8
CConnection: Using RFB protocol version 3.8
CConnection: Choosing security type VeNCrypt(19)
CVeNCrypt: Choosing security type X509Plain (262)

Fri Dec 13 18:26:18 2019
TLS: Could not load system certificate trust store

Fri Dec 13 18:26:19 2019
CConn: Authentication failure: Out of memory

@radonish
Copy link
Author

@bphinz, do you have a 1.9.80 nightly build macOS version you could send my way? It would be another interesting data point. My latest test is on a brand new iMac with macOS Catalina on it - I'm at a loss as to what is special about my configuration.

Thanks

@bphinz
Copy link
Member

bphinz commented Dec 15, 2019 via email

@radonish
Copy link
Author

Another data point: I installed TigerVNC 1.10.0 via Homebrew and see the same problem. Given that I'm running into this problem with a very vanilla, non-customized macOS system, I'm left thinking that this cryptic "Authentication failure: Out of memory" failure is due to a server-side configuration that is exposing a client-side bug.

If folks could post more detail about how they've configured/started the TigerVNC server I would appreciate it.

@bphinz
Copy link
Member

bphinz commented Dec 24, 2019

I'm skeptical that it will fix this problem, but could you please try the latest nightly build? I updated the underlying gnutls library and fixed a minor linker issue in the macOS build.

@radonish
Copy link
Author

@bphinz, I appreciate the thought - I gave it a try but unfortunately it does fail in the same way. I'm working on getting a development environment set up on the Mac so I can debug.

Thank you

@radonish
Copy link
Author

I'm not familiar with the TigerVNC code or the gnutls library, but the issue to me appears to be with the interface to gnutls_x509_crt_export() within common/rfb/CSecurityTLS.cxx.

Based on the TigerVNC code, my best guess is that the developer was first calling gnutls_x509_crt_export() with a NULL output buffer in an attempt to get the minimum buffer size required to store the certificate, knowing GNUTLS_E_SHORT_MEMORY_BUFFER would be returned. The second call, I assume, would then actually store the certificate in the non-NULL buffer.

Please see the proof-of-concept patch attached; the other noise in the patch are changes I made to get it to build on macOS using the compiler noted below. This fixed the issue for me.

$ /Library/Developer/CommandLineTools/usr/bin/c++ --version
Apple clang version 11.0.0 (clang-1100.0.33.16)
Target: x86_64-apple-darwin19.2.0
Thread model: posix

Thanks

Updates-for-macOS-and-gnutls-usage.patch.txt

@radonish radonish mentioned this issue Dec 28, 2019
Closed
@radonish
Copy link
Author

radonish commented Dec 28, 2019 •
edited
Loading

I've created 2 pull requests:

  1. macOS and clang compilation-related updates #928 - changes made to get vncviewer to build with clang 11.0.0
  2. Fix gnutls X.509 certificate exporting #927 - fix for this issue

Thanks

@IntiQuan
Copy link

Dear all, had the same issue and was not understanding most o the above. But solved the problem as follows:

  • When using the X.509 certificate with TigerVNC viewer 1.8.0 all works fine. In the folder "C:\Users<>\AppData\Roaming\vnc" the file "x509_savedcerts.pem" is created.
  • If using vncviewer 1.10.0 this "x509_savedcerts.pem" file is not created and the out of memory message is shown.
  • When first running 1.8.0 then 1.10.0 runs as well just fine.
  • If deleting the "x509_savedcerts.pem" file then 1.10.0 again shows "out of memory"

So somehow 1.10.0 (and 1.10.1) fail in the generation of the "x509_savedcerts.pem" file .. once its there - no problem. Also, when providing the path to the cert in the vncviewer security options: no problem as well!

Maybe that helps is debugging it?

@CendioOssman
Copy link
Member

I think that is just because you already have the exception save in that case so you're avoiding the broken code.

Not sure how this ever worked, but it should be fixed as of dbad687.

Thanks for helping out.

@CendioOssman
Copy link
Member

@radonish, feel free to claim the bounty if you want as you did the leg work on this issue.

@CendioOssman CendioOssman changed the title Authentication failure: Out of memory [$25] Authentication failure: Out of memory [$25 awarded] May 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bounty There is a bounty for this issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@samhed @CendioOssman @bphinz @IntiQuan @tarun-dholariya @radonish