Clients unable to connect over HTTPS as of 5/30/2020. HTTP working. v.20.05

[bruzona]

Describe the bug
A clear and concise description of what the bug is.
Clients are unable to connect over HTTPS, but once you change configuration to http / port 80 problem resolves. This began happening 5/30/2020.

Today (Saturday, May 30th) we found that none of our Libki clients were able to connect to Libki over https. Prior to this, NO changes to settings or updates were given to:
a. Libki
b. Apache2
c. Ubuntu
d. MySQL
e. Configuration files.
*We have a cardinal rule about not making system changes on Fridays unless you want to ruin your weekend! :)
The WebGUI was running fine, responsive, and all of the users, history, data were intact.
Libki's log files showed no client attempts to connect.
Our firewalls and routers showed no blocking events.

Diagnosis steps:
Our environment is fully virtualized (not just Docker) - so we tried
a. Rolling back to a known good configuration from a week ago. (Did not work)
b. Restoring from a full backup backup on May 13th (fresh build day). (Did not work)
c. Restoring our prior Libki (v.1901). (Did not work.)
d. Changing clients to use scheme http and port 80 instead of https/443. (Worked).
--Note that in this setting apache is still reverse-proxying Libki, just not over https.

Possible causes (unproven speculation)
A mandatory uncheduled Windows 10 update broke openssh on the clients?

To Reproduce
Steps to reproduce the behavior:

1. Configure clients to use scheme https, port 443
2. Restart clients.
3. Clients vanish from libki and do not re-register.

Expected behavior
A clear and concise description of what you expected to happen.
Clients should be registering against Libki server in a timely manner over either http or https.

Screenshots
If applicable, add screenshots to help explain your problem.

Deployment architecture:

• OS: [e.g. iOS] Ubuntu 20.04 server
• Deployment style ( LAMP, Docker, etc ) LAMP
• Versions of software used Libki server 20.05 as of May 13th build status.

Desktop (please complete the following information):

• OS: [e.g. iOS] Windows 10 Professional 64
• Browser [e.g. chrome, safari] N/A - but clients can connect to libki GUI using any browser.
• Version [e.g. 22]

Additional context
Add any other context about the problem here.
Our most recent Windows updates (clients) were run Tuesday, May 26. We control when we deploy updates, but sometimes Microsoft knows best and forces an unannounced patch. Clients were connecting over https/443 through late Friday (may 29) with no issues.

[loidor]

Huh, that is weird behaviour. It's a Libki server error since you've already taken that out of the equation by restoring to a previously known working instance, but it's still weird.

A couple short questions:

• I guess you didn't updated your clients this Friday?
• Does the webgui respond as usual over https? Also, test this at one of the client machines too, not just your staff machine. I don't know your network setup, but them being on different networks can give these kinds of errors.

If you have the possibility to boot one of the clients on a linux usb stick, I'd do that and compile and test the client from linux. That will clean out all doubts about it being Microsoft or not.

[bruzona]

1. No, we did not update the clients on Friday.
2. Yes, the WebGUI is fine over https, INCLUDING opening a browser from one of the clients.

NEW OBSERVATION:
Client communicating over HTTP/80 refuses to display https content/images for the Client Banners area, but does register with the server. (Client is refusing to display https content.)

A year or so ago we encountered similar symptoms, and it was finally determined that openssh DLL's were missing in our windows environment, but not on Kyle's testing machine. Subsequently the installer was modified to include the opensssh DLLs if needed.

Since W10 v1803, and above, OpenSSH server and Client are both built-in choices under Optional "Apps & Features."

I enabled both of these on a test client.
Unfortunately it did not resolve the issue.

[loidor]

Right.

Since you can access the server webgui from the client machine, the port is open and the connection is clear. And since both the server and the client are the same, the only thing left is that Microsoft did something.

Could you access the windows update logs and check if anything's been installed lately?

[bruzona]

Client was running 2.0.3.0. (Clients were updated about a week ago, and everything was working fine over https through Friday 5/29/20.)

Today: I uninstalled client, reinstalled using 20.05 official/stable release.
Note that Windows squacks about this EXE installer being unsigned. (DANGER DANGER!)

Fresh install matches release of 2.0.3.0, but again, client won't communicate over https.

Will test Linux client when time permits.

[bruzona]

I have mostly built Linux client VM (Ubuntu 20.04 Desktop) - but it seems to load without actually reading its config file, per: http://manual.libki.org/v20.05/libki-manual.html#_linux.

The client banner area displays a "Host not found" error failed to load URL http:/.

The backdoor password is also ignored.

nano ~/.config/Libki.ini

(I placed copies of Libki in /home/username/.config
and /root/.config

Where should this actually be located so that libkiclient binary will read it when starting?

[loidor]

It should be /home/username/.config/Libki.ini, based off of the example.ini that's in the source code folder. I'll update the manual about that so it's more clear.

[bruzona]

Okay, I was running the client as root rather than user is why it couldn't find the path.

Linux client running on exact same subnet - https working.
Windows 10 client running on exact same subnet - not working. (version 20.05).
(Ran on same subnet to completely eliminate our firewall/router as possible culprit.)

Windows complained about unsigned installer, so perhaps Windows 10 no longer trusts Libki and is sandboxing it. Perhaps signing the installer might resolve the issue (?).

Otherwise, some change in Windows appears to have occurred.

EDIT:

1. Downgraded client to v. 19.08. -Https still not working.
2. Downgraded client to v.19.01 -https still not working.

[loidor]

No, the unsigned installer is just that - an unsigned installer. Problem with that is a cert is quite expensive and timed, so it's a yearly fee for it.

This is some win change.

What win10 version are you running? It struck me that 2004 began rolling out May 27, so that might be it.

[bruzona]

Was grasping at straws regarding certs.

We are running W10Pro64 v1909, build 18368.836

But . . . the May 27 date the problem began, doesn't seem like mere coincidence.

Also we have disabled Windows "feature" updates, only allowing security updates, because feature updates more often tend to break things (especially printers) and have the annoying tendency to turn things back on that we had previously disabled.

[JSahlberg]

This actually happened us just recently since 30/09/21. I highly suspect that it's cert-related in some way... Still haven't found a solution... Any ideas?

[JSahlberg]

Never mind... I installed the newest client version and now it's up and running again! =)

[bruzona]

We had this problem a couple of years back. The linux client connected over https with no issues, but not windows. We tried various combinations of "old client with new server" (worked), "new client with old server" (did not), "New client with direct connection port 3000 excluding Apache reverse proxy" (didn't work). Based on this, it was clearly windows client not the server. I don't remember exactly when or how the client got fixed. Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Johan Sahlberg ***@***.***> Sent: Wednesday, October 20, 2021 12:29:43 AM To: Libki/libki-server ***@***.***> Cc: Bruce Carter ***@***.***>; Author ***@***.***> Subject: Re: [Libki/libki-server] Clients unable to connect over HTTPS as of 5/30/2020. HTTP working. v.20.05 (#172) CAUTION: This email originated from outside your organization. Exercise caution when opening attachments or clicking links, especially from unknown senders. Never mind... I installed the newest client version and now it's up and running again! =) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#172 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AEVULGWXQ7ZOO4LDTUKNRDTUHZVWPANCNFSM4NOZ37LA>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.