Duplicati with Backblaze B2 - certificate error from mono

[redakula]

App Name

Duplicati

SCALE Version

22.02.0

App Version

5.0.6

Application Events

The error is within duplicati

Application Logs

System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /build/mono-6.12.0.107/external/boringssl/ssl/handshake_client.c:1132
  at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00048] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000da] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)
  at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00006] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000fc] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
   --- End of inner exception stack trace ---
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) [0x00262] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Mono.Net.Security.MonoTlsStream.CreateStream (System.Net.WebConnectionTunnel tunnel, System.Threading.CancellationToken cancellationToken) [0x0016a] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x001ba] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
   --- End of inner exception stack trace ---
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x0021a] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x00141] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebOperation.Run () [0x0009a] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest+AsyncWrapper.GetResponseOrStream () [0x0004d] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse () [0x00044] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Backend.Backblaze.B2AuthHelper.get_Config () [0x0013d] in <f30a9ba7585445e094ae4320fb244dfc>:0 
  at Duplicati.Library.Backend.Backblaze.B2AuthHelper.get_APIUrl () [0x00000] in <f30a9ba7585445e094ae4320fb244dfc>:0 
  at Duplicati.Library.Backend.Backblaze.B2.List () [0x00011] in <f30a9ba7585445e094ae4320fb244dfc>:0 
  at Duplicati.Library.Interface.BackendExtensions.TestList (Duplicati.Library.Interface.IBackend backend) [0x00000] in <fd3642a459884bd9a2412b4eda050109>:0 
  at Duplicati.Library.Backend.Backblaze.B2.Test () [0x00000] in <f30a9ba7585445e094ae4320fb244dfc>:0 
  at Duplicati.Server.WebServer.RESTMethods.RemoteOperation.TestConnection (System.String url, Duplicati.Server.WebServer.RESTMethods.RequestInfo info) [0x000b7] in <156011ea63b34859b4073abdbf0b1573>:0 
  at Duplicati.Server.WebServer.RESTMethods.RemoteOperation.POST (System.String key, Duplicati.Server.WebServer.RESTMethods.RequestInfo info) [0x00094] in <156011ea63b34859b4073abdbf0b1573>:0 
  at Duplicati.Server.WebServer.RESTHandler.DoProcess (Duplicati.Server.WebServer.RESTMethods.RequestInfo info, System.String method, System.String module, System.String key) [0x00289] in <156011ea63b34859b4073abdbf0b1573>:0

Application Configuration

All settings are at their defaults

Describe the bug

During configuration of the backups to backblaze B2 when you test the connection the test failes with the above error.

To Reproduce

Try to configure a backup destination on backblaze B2

Expected Behavior

Expect the test and backblaze to work

Screenshots

N/A

Additional Context

The issue appears to be the one referenced here:
https://forum.duplicati.com/t/http-send-report-errors-duplicati-monitoring/13157/38

Basically it appears a certificate in the chain (DST Root CA X3) is expired so either mono or the certificates need to be updated.

I've read and agree with the following

• I've checked all open and closed issues and my issue is not there.

[stavros-k]

You answered your own problem there..
The problem is within duplicati.

We don't build the container. We just wrap it in a helm chart.

What I can do, is update the digest pin of the image we use.
If they have included a fix in there, you are lucky.
Otherwise you have to ask them to fix it.

Expect the app update in couple of hours. version 5.0.7.

Closing this as we can't do anything else here.

[stavros-k]

Also next time, actually provide ALL configurations / Application Events and Logs in FULL.
Even if you don't see errors or if everything is default.

[redakula]

Hi
Nice with quick updates :)
Deleted the app and installed the new version. No difference in the result - the certificate error above still occurs...
As i read the referenced thread the issue is in the mono version in the pod having an expired certificate in the chain.

A fix for debian is provided here but i have never used pods before so i am unsure of how to apply it to this case.
https://forum.duplicati.com/t/http-send-report-errors-duplicati-monitoring/13157/16

There is actually additional errors about certificates for an update process as well:

28 Mar 2022 16:39: Reporting error gave error
System.ObjectDisposedException: Can not write to a closed TextWriter.
  at System.IO.StreamWriter.Flush (System.Boolean flushStream, System.Boolean flushEncoder) [0x00008] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at System.IO.StreamWriter.Flush () [0x00006] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at Duplicati.Server.WebServer.RESTHandler.DoProcess (Duplicati.Server.WebServer.RESTMethods.RequestInfo info, System.String method, System.String module, System.String key) [0x003bc] in <156011ea63b34859b4073abdbf0b1573>:0 
28 Mar 2022 16:37: Error in updater
System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /build/mono-6.12.0.107/external/boringssl/ssl/handshake_client.c:1132
  at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00048] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000da] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)
  at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00006] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000fc] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
   --- End of inner exception stack trace ---
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) [0x00262] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Mono.Net.Security.MonoTlsStream.CreateStream (System.Net.WebConnectionTunnel tunnel, System.Threading.CancellationToken cancellationToken) [0x0016a] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x001ba] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
   --- End of inner exception stack trace ---
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x0021a] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x00141] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebOperation.Run () [0x0009a] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.HttpWebRequest.GetResponse () [0x00016] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebClient.GetWebResponse (System.Net.WebRequest request) [0x00000] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebClient.DownloadBits (System.Net.WebRequest request, System.IO.Stream writeStream) [0x000e6] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebClient.DownloadFile (System.Uri address, System.String fileName) [0x00088] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.WebClient.DownloadFile (System.String address, System.String fileName) [0x00008] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at (wrapper remoting-invoke-with-check) System.Net.WebClient.DownloadFile(string,string)
  at Duplicati.Library.AutoUpdater.UpdaterManager.CheckForUpdate (Duplicati.Library.AutoUpdater.ReleaseType channel) [0x000ee] in <8d4cb1693e00483189d3952c3f0ed20f>:0 

Sorry about the missing config - here is the configuration:

Application Name: duplicati
 Version: 5.0.9
 :
     Show Advanced Controller Settings: false
     Show Expert Configuration Options: false
 Timezone: 'Europe/Copenhagen' timezone
 Show Expert Config: false
 Configure Service(s):
     Main Service:
         Service Type: Simple
         Service's Port(s) Configuration:
             Main Service Port Configuration:
                 Port: 8200
                 Show Advanced settings: false
 Show Expert Config: false
 Integrated Persistent Storage:
     App Config Storage:
         Type of Storage: PVC (simple)
         readOnly: false
         Show Advanced Options: false
 Additional app storage: 1
 :
     Main Ingress:
         Enable Ingress: false
         Show Expert Configuration Options: false
 Container Security Settings:
     Change PUID / UMASK values: false
 Show Advanced Security Settings: false
 Pod Security Context:
     runAsUser: 568
     runAsGroup: 568
     fsGroup: 568
     When should we take ownership?: OnRootMismatch
 Set Custom Resource Limits/Requests (Advanced): false
 :
     VPN:
         Type: disabled
     Codeserver:
         enabled: false
     Promtail:
         enabled: false
     Netshoot:
         enabled: false
 (Advanced) Horizontal Pod Autoscaler:
     enabled: false
 (Advanced) Network Policy:
     enabled: false

[stavros-k]

As I already said, the mono version or whatever the problem is, is contained in the image THEY provide.
It's not something we can fix.

[TopicsLP]

Sorry to reply on a closed issue, but i have a similar problem with Let's Encrypt certificates,
and i think i got some helpful information.
In my TrueNAS Scale the docker images shows a "CREATED" time of "12 months ago"

root@TrueNasScale[~]# docker images tccr.io/truecharts/duplicati -a
REPOSITORY                     TAG       IMAGE ID       CREATED         SIZE
tccr.io/truecharts/duplicati   <none>    e4ab3b762518   12 months ago   709MB

Version Information from TrueNAS>Apps

duplicati / latest_6.0.5
tccr.io/truecharts/duplicati:latest@sha256:9435ca54cf320b8f6b285e4bb6b304e285e828a2b97f29f3037ac604924d99a0Up to date

So i tested with a normal Docker Container from the Linuxserver.io Team.
I did encounter the same issue in version:
lscr.io/linuxserver/duplicati:v2.0.6.3-2.0.6.3_beta_2021-06-17-ls102 (created 2021-06-17)

I did then test a newer random version, and the issue did not occur:
lscr.io/linuxserver/duplicati:v2.0.6.3-2.0.6.3_beta_2021-06-17-ls131 (created 2022-04-22)

I assume even if the TrueCharts repo got updated 3 months ago (what i found) somewhere inside of TrueCharts there is somewhere an old image for the docker container.

Error Messages from the Updater inside of Duplicati (just for testing):

System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /build/mono-5.20.1.34/external/boringssl/ssl/handshake_client.c:1132
  at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00038] in <91935ad653254a93b9d73a9f8f2f7a2d>:0 
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000a1] in <91935ad653254a93b9d73a9f8f2f7a2d>:0 
  at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)
  at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00006] in <91935ad653254a93b9d73a9f8f2f7a2d>:0 
  at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000ff] in <91935ad653254a93b9d73a9f8f2f7a2d>:0 
  at Mono.Net.Security.AsyncProtocolRequest.StartOperation (System.Threading.CancellationToken cancellationToken) [0x0008b] in <91935ad653254a93b9d73a9f8f2f7a2d>:0 
   --- End of inner exception stack trace ---
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) [0x00252] in <91935ad653254a93b9d73a9f8f2f7a2d>:0 
  at Mono.Net.Security.MonoTlsStream.CreateStream (System.Net.WebConnectionTunnel tunnel, System.Threading.CancellationToken cancellationToken) [0x00126] in <91935ad653254a93b9d73a9f8f2f7a2d>:0 
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x001ba] in <91935ad653254a93b9d73a9f8f2f7a2d>:0 
   --- End of inner exception stack trace ---
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x0021a] in <91935ad653254a93b9d73a9f8f2f7a2d>:0 
  at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x00141] in <91935ad653254a93b9d73a9f8f2f7a2d>:0 
  at System.Net.WebOperation.Run () [0x0009a] in <91935ad653254a93b9d73a9f8f2f7a2d>:0 
  at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <91935ad653254a93b9d73a9f8f2f7a2d>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <91935ad653254a93b9d73a9f8f2f7a2d>:0 
  at System.Net.HttpWebRequest.GetResponse () [0x00016] in <91935ad653254a93b9d73a9f8f2f7a2d>:0 
  at System.Net.WebClient.GetWebResponse (System.Net.WebRequest request) [0x00000] in <91935ad653254a93b9d73a9f8f2f7a2d>:0 
  at System.Net.WebClient.DownloadBits (System.Net.WebRequest request, System.IO.Stream writeStream) [0x000e6] in <91935ad653254a93b9d73a9f8f2f7a2d>:0 
  at System.Net.WebClient.DownloadFile (System.Uri address, System.String fileName) [0x00088] in <91935ad653254a93b9d73a9f8f2f7a2d>:0 
  at System.Net.WebClient.DownloadFile (System.String address, System.String fileName) [0x00008] in <91935ad653254a93b9d73a9f8f2f7a2d>:0 
  at (wrapper remoting-invoke-with-check) System.Net.WebClient.DownloadFile(string,string)
  at Duplicati.Library.AutoUpdater.UpdaterManager.CheckForUpdate (Duplicati.Library.AutoUpdater.ReleaseType channel) [0x000ee] in <8d4cb1693e00483189d3952c3f0ed20f>:0

[stavros-k]

We are not using the LSIO image, we use the official duplicati image.
And looks like they didn't release any image with tag latest (or any not canary tags) within the last year . canary had some releases, which is the dev/test channel and I'm not going to use that. Even their "beta/latest" is unstable.
You have to raise your issue to duplicati devs, to release a newer release.

[Ornias1993]

Also on this:
Don't necro issues when we already concluded it was not something we handle, including a clear reference why that's the case.

You're basically wasting everyones time doing so.