[Bug]: Proxy GRPc cant not be accessed via https

[zoltan65]

Summary

It is not possible to connect Defguard proxy GRPc using https protocol from core. Http protocol works without problems.

Https is performed with Custom SSL CA and certificates. Certificates contains CN with exact name of proxy host.
The same is with certificates from LetsEncrypt.

Curl shows: curl: (35) TLS connect error: error:0A00010B:SSL routines::wrong version number

Steps to reproduce

1. Core config contains:
   DEFGUARD_PROXY_URL=https://proxy.abc.local:50051
   DEFGUARD_PROXY_GRPC_CA=/etc/defguard/ca.crt

Core cannot connect to proxy. Error message from core:
Mar 20 14:32:50 DefGuard-Core defguard[20087]: 2026-03-20T13:32:50.773197Z ERROR run_grpc_bidi_stream: defguard_core::grpc: message=Failed to connect to proxy @ https://proxy.abc.local:50051/, retrying in 10s: The service is currently unavailable [1.6.5+e2ec171 Debian 13.0.0 x86_64]

2. When core config contains:
   DEFGUARD_PROXY_URL=http://proxy.abc.local:50051

Core can connect to proxy:
Mar 20 15:03:36 DefGuard-Core defguard[23806]: 2026-03-20T14:03:36.602599Z INFO run_grpc_bidi_stream:proxy_bidi: defguard_core::grpc: message=Connected to proxy at http://proxy.abc.local:50051/ [1.6.5+e2ec171][PX:1.6.3+1c88f69]

Expected behavior

Core should be able to connect to proxy using https protocol and custom ssl certificates, see:
https://docs.defguard.net/deployment-strategies/grpc-ssl-communication

Actual behavior

Logs from core:
Mar 20 14:32:40 DefGuard-Core defguard[20087]: 2026-03-20T13:32:40.761256Z DEBUG run_grpc_bidi_stream: defguard_core::grpc: message=Connecting to proxy at https://proxy.abc.local:50051/
Mar 20 14:32:40 DefGuard-Core defguard[20087]: 2026-03-20T13:32:40.766571Z DEBUG run_grpc_bidi_stream: hyper_util::client::legacy::connect::http: message=connecting to 123.123.123.123:50051
Mar 20 14:32:40 DefGuard-Core defguard[20087]: 2026-03-20T13:32:40.766859Z DEBUG run_grpc_bidi_stream: hyper_util::client::legacy::connect::http: message=connected to 123.123.123.123:50051
Mar 20 14:32:40 DefGuard-Core defguard[20087]: 2026-03-20T13:32:40.766878Z DEBUG run_grpc_bidi_stream: rustls::client::hs: message=No cached session for DnsName("proxy.abc.local") log.target=rustls::client::hs log.module_path=rustls::client::hs log.file=/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rustls-0.23.37/src/client/hs.rs log.line=1124
Mar 20 14:32:40 DefGuard-Core defguard[20087]: 2026-03-20T13:32:40.766884Z DEBUG run_grpc_bidi_stream: rustls::client::hs: message=Not resuming any session log.target=rustls::client::hs log.module_path=rustls::client::hs log.file=/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rustls-0.23.37/src/client/hs.rs log.line=105
Mar 20 14:32:40 DefGuard-Core defguard[20087]: 2026-03-20T13:32:40.767124Z DEBUG run_grpc_bidi_stream: tonic::transport::channel::service::reconnect: message=reconnect::poll_ready: ConnectError(Custom { kind: InvalidData, error: InvalidMessage(InvalidContentType) })
Mar 20 14:32:40 DefGuard-Core defguard[20087]: 2026-03-20T13:32:40.767132Z DEBUG run_grpc_bidi_stream: tower::buffer::worker: service.ready=true message=processing request
Mar 20 14:32:40 DefGuard-Core defguard[20087]: 2026-03-20T13:32:40.767139Z DEBUG run_grpc_bidi_stream: tonic::transport::channel::service::reconnect: message=error: received corrupt message of type InvalidContentType
Mar 20 14:32:40 DefGuard-Core defguard[20087]: 2026-03-20T13:32:40.767164Z ERROR run_grpc_bidi_stream: defguard_core::grpc: message=Failed to connect to proxy @ https://proxy.abc.local:50051/, retrying in 10s: The service is currently unavailable [1.6.5+e2ec171 Debian 13.0.0 x86_64]
Mar 20 14:32:40 DefGuard-Core defguard[20087]: 2026-03-20T13:32:40.867810Z DEBUG run_utility_thread: sqlx::query: summary=SELECT id, name, base_url, … db.statement=\n\nSELECT id, name, base_url, client_id, client_secret, display_name, google_service_account_key, google_service_account_email, admin_email, directory_sync_enabled, directory_sync_interval, directory_sync_user_behavior "directory_sync_user_behavior: DirectorySyncUserBehavior", directory_sync_admin_behavior "directory_sync_admin_behavior: DirectorySyncUserBehavior", directory_sync_target "directory_sync_target: DirectorySyncTarget", okta_private_jwk, okta_dirsync_client_id, directory_sync_group_match, jumpcloud_api_key, prefetch_users FROM openidprovider LIMIT 1
Mar 20 14:32:40 DefGuard-Core defguard[20087]: rows_affected=0 rows_returned=0 elapsed=131.463µs elapsed_secs=0.000131463
Mar 20 14:32:45 DefGuard-Core defguard[20087]: 2026-03-20T13:32:45.869478Z DEBUG run_utility_thread: sqlx::query: summary=SELECT id, name, base_url, … db.statement=\n\nSELECT id, name, base_url, client_id, client_secret, display_name, google_service_account_key, google_service_account_email, admin_email, directory_sync_enabled, directory_sync_interval, directory_sync_user_behavior "directory_sync_user_behavior: DirectorySyncUserBehavior", directory_sync_admin_behavior "directory_sync_admin_behavior: DirectorySyncUserBehavior", directory_sync_target "directory_sync_target: DirectorySyncTarget", okta_private_jwk, okta_dirsync_client_id, directory_sync_group_match, jumpcloud_api_key, prefetch_users FROM openidprovider LIMIT 1
Mar 20 14:32:45 DefGuard-Core defguard[20087]: rows_affected=0 rows_returned=0 elapsed=159.652µs elapsed_secs=0.000159652
Mar 20 14:32:50 DefGuard-Core defguard[20087]: 2026-03-20T13:32:50.768078Z DEBUG run_grpc_bidi_stream: defguard_core::grpc: message=Connecting to proxy at https://proxy.abc.local:50051/
Mar 20 14:32:50 DefGuard-Core defguard[20087]: 2026-03-20T13:32:50.772565Z DEBUG run_grpc_bidi_stream: hyper_util::client::legacy::connect::http: message=connecting to 123.123.123.123:50051
Mar 20 14:32:50 DefGuard-Core defguard[20087]: 2026-03-20T13:32:50.772902Z DEBUG run_grpc_bidi_stream: hyper_util::client::legacy::connect::http: message=connected to 123.123.123.123:50051
Mar 20 14:32:50 DefGuard-Core defguard[20087]: 2026-03-20T13:32:50.772916Z DEBUG run_grpc_bidi_stream: rustls::client::hs: message=No cached session for DnsName("proxy.abc.local") log.target=rustls::client::hs log.module_path=rustls::client::hs log.file=/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rustls-0.23.37/src/client/hs.rs log.line=1124
Mar 20 14:32:50 DefGuard-Core defguard[20087]: 2026-03-20T13:32:50.772921Z DEBUG run_grpc_bidi_stream: rustls::client::hs: message=Not resuming any session log.target=rustls::client::hs log.module_path=rustls::client::hs log.file=/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rustls-0.23.37/src/client/hs.rs log.line=105
Mar 20 14:32:50 DefGuard-Core defguard[20087]: 2026-03-20T13:32:50.773156Z DEBUG run_grpc_bidi_stream: tonic::transport::channel::service::reconnect: message=reconnect::poll_ready: ConnectError(Custom { kind: InvalidData, error: InvalidMessage(InvalidContentType) })
Mar 20 14:32:50 DefGuard-Core defguard[20087]: 2026-03-20T13:32:50.773166Z DEBUG run_grpc_bidi_stream: tower::buffer::worker: service.ready=true message=processing request
Mar 20 14:32:50 DefGuard-Core defguard[20087]: 2026-03-20T13:32:50.773171Z DEBUG run_grpc_bidi_stream: tonic::transport::channel::service::reconnect: message=error: received corrupt message of type InvalidContentType
Mar 20 14:32:50 DefGuard-Core defguard[20087]: 2026-03-20T13:32:50.773197Z ERROR run_grpc_bidi_stream: defguard_core::grpc: message=Failed to connect to proxy @ https://proxy.abc.local:50051/, retrying in 10s: The service is currently unavailable [1.6.5+e2ec171 Debian 13.0.0 x86_64]
Mar 20 14:32:50 DefGuard-Core defguard[20087]: 2026-03-20T13:32:50.871086Z DEBUG run_utility_thread: sqlx::query: summary=SELECT id, name, base_url, … db.statement=\n\nSELECT id, name, base_url, client_id, client_secret, display_name, google_service_account_key, google_service_account_email, admin_email, directory_sync_enabled, directory_sync_interval, directory_sync_user_behavior "directory_sync_user_behavior: DirectorySyncUserBehavior", directory_sync_admin_behavior "directory_sync_admin_behavior: DirectorySyncUserBehavior", directory_sync_target "directory_sync_target: DirectorySyncTarget", okta_private_jwk, okta_dirsync_client_id, directory_sync_group_match, jumpcloud_api_key, prefetch_users FROM openidprovider LIMIT 1
Mar 20 14:32:50 DefGuard-Core defguard[20087]: rows_affected=0 rows_returned=0 elapsed=236.208µs elapsed_secs=0.000236208
Mar 20 14:32:55 DefGuard-Core defguard[20087]: 2026-03-20T13:32:55.873139Z DEBUG run_utility_thread: sqlx::query: summary=SELECT id, name, base_url, … db.statement=\n\nSELECT id, name, base_url, client_id, client_secret, display_name, google_service_account_key, google_service_account_email, admin_email, directory_sync_enabled, directory_sync_interval, directory_sync_user_behavior "directory_sync_user_behavior: DirectorySyncUserBehavior", directory_sync_admin_behavior "directory_sync_admin_behavior: DirectorySyncUserBehavior", directory_sync_target "directory_sync_target: DirectorySyncTarget", okta_private_jwk, okta_dirsync_client_id, directory_sync_group_match, jumpcloud_api_key, prefetch_users FROM openidprovider LIMIT 1
Mar 20 14:32:55 DefGuard-Core defguard[20087]: rows_affected=0 rows_returned=0 elapsed=158.623µs elapsed_secs=0.000158623
Mar 20 14:33:00 DefGuard-Core defguard[20087]: 2026-03-20T13:33:00.774430Z DEBUG run_grpc_bidi_stream: defguard_core::grpc: message=Connecting to proxy at https://proxy.abc.local:50051/
Mar 20 14:33:00 DefGuard-Core defguard[20087]: 2026-03-20T13:33:00.779601Z DEBUG run_grpc_bidi_stream: hyper_util::client::legacy::connect::http: message=connecting to 123.123.123.123:50051
Mar 20 14:33:00 DefGuard-Core defguard[20087]: 2026-03-20T13:33:00.779906Z DEBUG run_grpc_bidi_stream: hyper_util::client::legacy::connect::http: message=connected to 123.123.123.123:50051
Mar 20 14:33:00 DefGuard-Core defguard[20087]: 2026-03-20T13:33:00.779922Z DEBUG run_grpc_bidi_stream: rustls::client::hs: message=No cached session for DnsName("proxy.abc.local") log.target=rustls::client::hs log.module_path=rustls::client::hs log.file=/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rustls-0.23.37/src/client/hs.rs log.line=1124
Mar 20 14:33:00 DefGuard-Core defguard[20087]: 2026-03-20T13:33:00.779931Z DEBUG run_grpc_bidi_stream: rustls::client::hs: message=Not resuming any session log.target=rustls::client::hs log.module_path=rustls::client::hs log.file=/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rustls-0.23.37/src/client/hs.rs log.line=105
Mar 20 14:33:00 DefGuard-Core defguard[20087]: 2026-03-20T13:33:00.780071Z DEBUG run_grpc_bidi_stream: tonic::transport::channel::service::reconnect: message=reconnect::poll_ready: ConnectError(Custom { kind: InvalidData, error: InvalidMessage(InvalidContentType) })
Mar 20 14:33:00 DefGuard-Core defguard[20087]: 2026-03-20T13:33:00.780080Z DEBUG run_grpc_bidi_stream: tower::buffer::worker: service.ready=true message=processing request
Mar 20 14:33:00 DefGuard-Core defguard[20087]: 2026-03-20T13:33:00.780087Z DEBUG run_grpc_bidi_stream: tonic::transport::channel::service::reconnect: message=error: received corrupt message of type InvalidContentType
Mar 20 14:33:00 DefGuard-Core defguard[20087]: 2026-03-20T13:33:00.780114Z ERROR run_grpc_bidi_stream: defguard_core::grpc: message=Failed to connect to proxy @ https://proxy.abc.local:50051/, retrying in 10s: The service is currently unavailable [1.6.5+e2ec171 Debian 13.0.0 x86_64]

Testing connection from core to proxy:
curl -vvv --cacert /etc/defguard/ca.crt https://proxy.abc.local:50051
14:36:48.225612 [0-x] == Info: [READ] client_reset, clear readers
14:36:48.226322 [0-0] == Info: Host proxy.abc.local:50051 was resolved.
14:36:48.226356 [0-0] == Info: IPv6: (none)
14:36:48.226368 [0-0] == Info: IPv4: 123.123.123.123
14:36:48.226382 [0-0] == Info: [HTTPS-CONNECT] adding wanted h2
14:36:48.226419 [0-0] == Info: [HTTPS-CONNECT] added
14:36:48.226468 [0-0] == Info: [HTTPS-CONNECT] connect, init
14:36:48.226518 [0-0] == Info: Trying 123.123.123.123:50051...
14:36:48.226571 [0-0] == Info: [HTTPS-CONNECT] connect -> 0, done=0
14:36:48.226621 [0-0] == Info: [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
14:36:48.226694 [0-0] == Info: [HTTPS-CONNECT] adjust_pollset -> 1 socks
14:36:48.226788 [0-0] == Info: [SSL] cf_connect()
14:36:48.226845 [0-0] == Info: [SSL] ossl_connect, step1
14:36:48.227778 [0-0] == Info: ALPN: curl offers h2,http/1.1
14:36:48.227828 [0-0] == Info: [SSL] ossl_connect, step2
14:36:48.228015 [0-0] => Send SSL data, 5 bytes (0x5)
0000: ....&
14:36:48.228087 [0-0] == Info: TLSv1.3 (OUT), TLS handshake, Client hello (1):
14:36:48.228157 [0-0] => Send SSL data, 1574 bytes (0x626)
0000: .
0040: ..B.....=.
0080: <.5./..................proxy.abc.local......................
00c0: ...............h2.http/1.1.........1.....6.4....................
0100: .................................+........-.....3.........T.Z..
0140: .n#
0180: GZ
01c0: ...)...
0200: .g-h
0240: .f..Tq,W!.
0280: @...'@..Z...N
02c0: ...)t..9f.0]n...
0300: .iE.2@A.!...".*..`.<
0340: ..m?Q,....Z....<.6(.t@........
0380: ....=..!b.Y...RG.s5J.
03c0: ..\H.J....#..9...g..
0400: ....Si>."XB..|pt.b.E......
0440: .g.o.S...h..I...3....t.h...G.(..L.+.H
0480: .8.......Tl.Ni..
04c0: :'.K.}.C..iNL
0500: e..J.....H.....3..!.g
0540: .s.q..F...j.q..1.
0580: .Q.8....7....)..5"z
05c0: ..d....WR.k..... xT.
0600: .S..^.d....y_....H[.u......!.........
14:36:48.230071 [0-0] == Info: [SSL] ossl_bio_cf_out_write(len=1579) -> 1579, err=0
14:36:48.230140 [0-0] == Info: [SSL] ossl_bio_cf_in_read(len=5) -> 5, err=0
14:36:48.230192 [0-0] == Info: [SSL] ossl_populate_x509_store, path=/etc/defguard/ca.crt, blob=0
14:36:48.230536 [0-0] == Info: CAfile: /etc/defguard/ca.crt
14:36:48.230575 [0-0] == Info: CApath: /etc/ssl/certs
14:36:48.230626 [0-0] <= Recv SSL data, 5 bytes (0x5)
0000: .....
14:36:48.230688 [0-0] == Info: [SSL] SSL_connect() -> err=-1, detail=1
14:36:48.230754 [0-0] == Info: TLS connect error: error:0A00010B:SSL routines::wrong version number
14:36:48.230834 [0-0] == Info: [SSL] cf_connect() -> 35, done=0
14:36:48.230890 [0-0] == Info: [HTTPS-CONNECT] connect, all attempts failed
14:36:48.230969 [0-0] == Info: [HTTPS-CONNECT] connect -> 35, done=0
14:36:48.231019 [0-0] == Info: [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 35, done=0
14:36:48.231083 [0-0] == Info: [HTTPS-CONNECT] Curl_conn_connect(), filter returned 35
14:36:48.231147 [0-0] == Info: [WRITE] [OUT] done
14:36:48.231185 [0-0] == Info: closing connection #0
curl: (35) TLS connect error: error:0A00010B:SSL routines::wrong version number

Defguard version

Core 1.6.5, Proxy 1.6.5, Gateway 1.6.5

Environment details

Debian 13 on Proxmox VM in Cluster with CEPH fs. Separate VM for Core, Proxy and Gateway

Deployment / install method

Standalone packages

Relevant logs / output

Relevant configuration (redacted)

[jakub-tldr]

Hello @zoltan65,

Can you confirm if the proxy is configured to listen HTTPS on port 50051?

[omegasupreme3]

I assume this issue is not going to be fixed for 1.6.x as it was added to the 2.0.0 milestone. Is that correct?

I wanted to checkout Defguard as a viable alternative for OpenVPN. Spend a lot of hours trying to get Defguard to work (because the documentation contains errors and contradictions), but ultimately failed. After a day of struggling I came across this bug and realized my failure was not due to configuration errors on my side. I still think Defguard looks great and promising, at least on paper, but I am not going to sell my OpenVPN installation yet.

[jakub-tldr]

Hi @zoltan65 and @omegasupreme3,

Are you running core/proxy as described in documentation?

I tried to reproduce your issue. I ran core with --grpc-cert --grpc-key --proxy-grpc-ca, proxy with --grpc-cert --grpc-key (as described in documentation). There was no issue with connecting via HTTPS.

Core log:

Proxy log:

Can you try this way?

[omegasupreme3]

Hi @zoltan65 and @omegasupreme3,

Are you running core/proxy as described in documentation?

I tried to reproduce your issue. I ran core with --grpc-cert --grpc-key --proxy-grpc-ca, proxy with --grpc-cert --grpc-key (as described in documentation). There was no issue with connecting via HTTPS.

Core log:

Proxy log: Can you try this way?

Thank you for your reply. I followed the documentation to the letter (which was on times frustrating as well because of inconsistencies and errors). If I connect through HTTP, then I am able to connect.

I will try your proposal, but I'm not sure if this is the desired path. I installed Defguard as a standalone package based installation, as in: rpms on RHEL. If I make changes to the way Defguard is started, it might be reverted after an update.

[jakub-tldr]

@omegasupreme3,

I compiled core/proxy from source code (that is why I ran it with cargo r --) , but running it via packages works the same (defguard --grpc-cert..... ).

Could you please describe (or send) your config? (you can anonymize it if needed)