Insufficient memory to continue the execution of the program

[fred-cardoso]

• [ X ] I have searched open and closed issues for duplicates.
• [ X ] I have searched the forum for related topics.

---

Environment info

• Duplicati version: 2.0.5.114
• Operating system: Linux
• Backend: OneDrive

Running under docker on Raspberry Pi 4 (4GB RAM).

Description

Recently, has become impossible to run a backup because sooner or later I get a Insufficient memory to continue the execution of the program error. I tried on multiple backup configurations, but if it takes longer, it crashes.

Steps to reproduce

There aren't really any steps to reproduce. I just need to create a backup of around 500GB, run it, and after a few hours it crashes.

Debug log

  at (wrapper alloc) System.Object.AllocVector(intptr,intptr)
  at Duplicati.Library.Backend.MicrosoftGraphBackend.PutAsync (System.String remotename, System.IO.Stream stream, System.Threading.CancellationToken cancelToken) [0x007dc] in <22708b578dd647fbafbabbcba5b0fd68>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoPut (Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Interface.IBackend backend, System.Threading.CancellationToken cancelToken) [0x00426] in <e4ba2224a87c42cbadcd524985619ee3>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader+<>c__DisplayClass20_0.<UploadFileAsync>b__0 () [0x0010a] in <e4ba2224a87c42cbadcd524985619ee3>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoWithRetry (System.Func`1[TResult] method, Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x0017c] in <e4ba2224a87c42cbadcd524985619ee3>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoWithRetry (System.Func`1[TResult] method, Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x003a5] in <e4ba2224a87c42cbadcd524985619ee3>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.UploadFileAsync (Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x000da] in <e4ba2224a87c42cbadcd524985619ee3>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.UploadBlockAndIndexAsync (Duplicati.Library.Main.Operation.Backup.VolumeUploadRequest upload, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x00098] in <e4ba2224a87c42cbadcd524985619ee3>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.<Run>b__13_0 (<>f__AnonymousType12`1[<Input>j__TPar] self) [0x00780] in <e4ba2224a87c42cbadcd524985619ee3>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.<Run>b__13_0 (<>f__AnonymousType12`1[<Input>j__TPar] self) [0x007d7] in <e4ba2224a87c42cbadcd524985619ee3>:0 
  at CoCoL.AutomationExtensions.RunTask[T] (T channels, System.Func`2[T,TResult] method, System.Boolean catchRetiredExceptions) [0x000d5] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.FlushBackend (Duplicati.Library.Main.BackupResults result, CoCoL.IWriteChannel`1[T] uploadtarget, System.Threading.Tasks.Task uploader) [0x000f2] in <e4ba2224a87c42cbadcd524985619ee3>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00cb2] in <e4ba2224a87c42cbadcd524985619ee3>:0 
   --- End of inner exception stack trace ---
  at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x01048] in <e4ba2224a87c42cbadcd524985619ee3>:0 
   --- End of inner exception stack trace ---
  at CoCoL.ChannelExtensions.WaitForTaskOrThrow (System.Threading.Tasks.Task task) [0x0005d] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.Run (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00009] in <e4ba2224a87c42cbadcd524985619ee3>:0 
  at Duplicati.Library.Main.Controller+<>c__DisplayClass14_0.<Backup>b__0 (Duplicati.Library.Main.BackupResults result) [0x0004b] in <e4ba2224a87c42cbadcd524985619ee3>:0 
  at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0026f] in <e4ba2224a87c42cbadcd524985619ee3>:0 
  at Duplicati.Library.Main.Controller.Backup (System.String[] inputsources, Duplicati.Library.Utility.IFilter filter) [0x00074] in <e4ba2224a87c42cbadcd524985619ee3>:0 
  at Duplicati.Server.Runner.Run (Duplicati.Server.Runner+IRunnerData data, System.Boolean fromQueue) [0x00349] in <d38f460035ab4f3b9bab2fc91e2cec32>:0 
---> (Inner Exception #0) System.AggregateException: Insufficient memory to continue the execution of the program. (Insufficient memory to continue the execution of the program.) (One or more errors occurred. (Insufficient memory to continue the execution of the program.)) ---> System.OutOfMemoryException: Insufficient memory to continue the execution of the program.```

[warwickmm]

Issue #4108 sounds related.

[fred-cardoso]

Hi @warwickmm

I mean, that previous issue #4108 eventually got fixed and everything was working fine but lately over the last 2 or 3 updates, not sure, Duplicati is eating so much memory it gets to a point where I can't even access the web interface and usually give up and restart the docker instance.
Is there any method or steps that you might need me to replicate and trace this down?

[warwickmm]

Hi @fred-cardoso, you'll have better luck getting help by posting in the forums. There are a lot more users there that can help you debug this issue.

https://forum.duplicati.com/

[samuel-w]

Maybe related to #1849?

[fred-cardoso]

Actually, the description of the problem matches. I can only release the memory by restarting Duplicati.
Also, I have 4G of Memory and it consumes it all.

[warwickmm]

You might want to add to this conversation:

https://forum.duplicati.com/t/onedrive-v2-memory-leak/6112

[fred-cardoso]

Sorry for the delay on this. Just to let you know that as soon as I have some time I'll try to better describe my issue on the forum.
Just FYI, I am still having this issue on the latest beta.

[davidtgbe]

I think I have the same problem here. Duplicati restarts when a backup job is initiated. After processing some files, RAM usage increases and duplicati restarts.

• Using docker container from: https://hub.docker.com/r/linuxserver/duplicati
• RPI 4 with 4GB
• Duplicati version: 2.0.6.3_beta_2021-06-17
• Mono version: 5.2.0

Target

od4b://*****-my.sharepoint.com:443//personal/*****/Documents/Backups/RPI-NAS?auth-username=********&auth-password=*******&binary-direct-mode=true

Options

--backup-name=RPI NAS
--dbpath=/config/ZZGVSBKEHJ.sqlite
--encryption-module=aes
--compression-module=zip
--dblock-size=50mb
--passphrase=********
--retention-policy=1W:1D,4W:1W,12M:1M
--snapshot-policy=Auto
--http-enable-buffering=false
--exclude-files-attributes=system,temporary
--disable-module=console-password-input
--console-log-level=verbose 
--log-file-log-level=verbose
--log-file=/config/duplicati.log

Updated container to latest Mono version (6.12.0.122 )and backup keeps on crashing. However this time backup process processed some more files.

Tried --http-enable-buffering =false as showed with no success.

Docker compose file

version: "2.1"
services:
  duplicati:
    image: ghcr.io/linuxserver/duplicati
    container_name: duplicati
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/Madrid
      - CLI_ARGS=
    volumes:
      - /home/pi/docker/duplicati/config:/config
      - /home/pi/docker/duplicati/backups:/backups
      - /home/pi/docker/duplicati/sources:/sources
    ports:
      - 8200:8200
    restart: unless-stopped

Executed cmd: mono Duplicati.Server.exe --webservice-interface=any --server-datafolder=/config --webservice-allowed-hostnames=*

duplicati.log

....
duplicati-ba987cb7270d244c985567ebba7a4207c.dblock.zip.aes attempt 1 of 6 failed with message: Insufficient memory to continue the execution of the program.
System.OutOfMemoryException: Insufficient memory to continue the execution of the program.
  at (wrapper alloc) System.Object.AllocVector(intptr,intptr)
  at System.IO.MemoryStream.set_Capacity (System.Int32 value) [0x00051] in <533173d24dae460899d2b10975534bb0>:0 
  at System.IO.MemoryStream.EnsureCapacity (System.Int32 value) [0x0005c] in <533173d24dae460899d2b10975534bb0>:0 
  at System.IO.MemoryStream.Write (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x00093] in <533173d24dae460899d2b10975534bb0>:0 
  at System.Net.WebRequestStream.ProcessWrite (System.Byte[] buffer, System.Int32 offset, System.Int32 size, System.Threading.CancellationToken cancellationToken) [0x0015b] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0 
  at System.Net.WebConnectionStream.Write (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x00070] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0 
  at Microsoft.SharePoint.Client.File.SaveBinary (Microsoft.SharePoint.Client.ClientContext context, System.String serverRelativeUrl, System.IO.Stream stream, System.String etag, System.Boolean overwriteIfExists, Microsoft.SharePoint.Client.File+SaveBinaryCheckMode checkMode) [0x0009c] in <8e3eb3065b524a909640c2bd1341366d>:0 
  at Microsoft.SharePoint.Client.File.SaveBinaryDirect (Microsoft.SharePoint.Client.ClientContext context, System.String serverRelativeUrl, System.IO.Stream stream, System.Boolean overwriteIfExists) [0x00000] in <8e3eb3065b524a909640c2bd1341366d>:0 
  at Duplicati.Library.Backend.SharePointBackend.doPut (System.String remotename, System.IO.Stream stream, System.Boolean useNewContext, System.Threading.CancellationToken cancelToken) [0x002ab] in <a5c9e803bccb46f885a80a9923ad8150>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoPut (Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Interface.IBackend backend, System.Threading.CancellationToken cancelToken) [0x00426] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader+<>c__DisplayClass20_0.<UploadFileAsync>b__0 () [0x0010a] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoWithRetry (System.Func`1[TResult] method, Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x0017c] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
....
....
....
duplicati-bb47d6bef031c4323959ebd44d99d3e6c.dblock.zip.aes attempt 1 of 6 failed with message: Insufficient memory to continue the execution of the program.
System.OutOfMemoryException: Insufficient memory to continue the execution of the program.
  at (wrapper alloc) System.Object.AllocVector(intptr,intptr)
  at System.IO.MemoryStream.set_Capacity (System.Int32 value) [0x00051] in <533173d24dae460899d2b10975534bb0>:0 
  at System.IO.MemoryStream.EnsureCapacity (System.Int32 value) [0x0005c] in <533173d24dae460899d2b10975534bb0>:0 
  at System.IO.MemoryStream.Write (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x00093] in <533173d24dae460899d2b10975534bb0>:0 
  at System.Net.WebRequestStream.ProcessWrite (System.Byte[] buffer, System.Int32 offset, System.Int32 size, System.Threading.CancellationToken cancellationToken) [0x0015b] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0 
  at System.Net.WebConnectionStream.Write (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x00070] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0 
  at Microsoft.SharePoint.Client.File.SaveBinary (Microsoft.SharePoint.Client.ClientContext context, System.String serverRelativeUrl, System.IO.Stream stream, System.String etag, System.Boolean overwriteIfExists, Microsoft.SharePoint.Client.File+SaveBinaryCheckMode checkMode) [0x0009c] in <8e3eb3065b524a909640c2bd1341366d>:0 
  at Microsoft.SharePoint.Client.File.SaveBinaryDirect (Microsoft.SharePoint.Client.ClientContext context, System.String serverRelativeUrl, System.IO.Stream stream, System.Boolean overwriteIfExists) [0x00000] in <8e3eb3065b524a909640c2bd1341366d>:0 
  at Duplicati.Library.Backend.SharePointBackend.doPut (System.String remotename, System.IO.Stream stream, System.Boolean useNewContext, System.Threading.CancellationToken cancelToken) [0x002ab] in <a5c9e803bccb46f885a80a9923ad8150>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoPut (Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Interface.IBackend backend, System.Threading.CancellationToken cancelToken) [0x00426] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader+<>c__DisplayClass20_0.<UploadFileAsync>b__0 () [0x0010a] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoWithRetry (System.Func`1[TResult] method, Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x0017c] in <e60bc008dd1b454d861cfacbdd3760b9>:0 

docker logs duplicati

-------------------------------------
          _         ()
         | |  ___   _    __
         | | / __| | |  /  \
         | | \__ \ | | | () |
         |_| |___/ |_|  \__/


Brought to you by linuxserver.io
-------------------------------------

To support LSIO projects visit:
https://www.linuxserver.io/donate/
-------------------------------------
GID/UID
-------------------------------------

User uid:    1000
User gid:    1000
-------------------------------------

[cont-init.d] 10-adduser: exited 0.
[cont-init.d] 30-config: executing...
[cont-init.d] 30-config: exited 0.
[cont-init.d] 90-custom-folders: executing...
[cont-init.d] 90-custom-folders: exited 0.
[cont-init.d] 99-custom-scripts: executing...
[custom-init] no custom files found exiting...
[cont-init.d] 99-custom-scripts: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.

[davidtgbe]

Ok. I think I've found the problem.
Option binary-direct-mode=true seems to be a very demanding option in terms of memory requirements. I use it on other windows machines with OneDrive4B as destination with no problems. However, in the RPI, inspite of speeding up the process, it ends up consuming all the ram and then, duplicati crashes and restarts.

My solution: binary-direct-mode=false (which is the default mode). I think the process is slower, but it works so far.

Update:
Backup did not end sucessfully. After uploading all files, in the verification process, I got the error The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator. This seems to be a limitiation when using od4b endpoint. It has to be with the fact that sharepoint is not able to show more than 5000 items and I can even get this error acessing onedrive through its web page.

[davidtgbe]

One more update:
If I switch the endpoint to onedrivev2 (to try to overcome the maximum of 5000 items when using the od4b endpoint), leaving the rest of the backup job configuration untouched, I end getting again the memory related error:

...
2022-01-15 12:25:37 +01 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
System.AggregateException: One or more errors occurred. (Insufficient memory to continue the execution of the program.) (Insufficient memory to continue the execution of the program.) (Insufficient memory to continue the execution of the program.) (Insufficient memory to continue the execution of the program.) ---> System.OutOfMemoryException: Insufficient memory to continue the execution of the program.
  at (wrapper alloc) System.Object.AllocVector(intptr,intptr)
  at Duplicati.Library.Backend.MicrosoftGraphBackend.PutAsync (System.String remotename, System.IO.Stream stream, System.Threading.CancellationToken cancelToken) [0x007dc] in <dfa9574ee8124175856109d811c0b28b>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoPut (Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Interface.IBackend backend, System.Threading.CancellationToken cancelToken) [0x00426] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader+<>c__DisplayClass20_0.<UploadFileAsync>b__0 () [0x0010a] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoWithRetry (System.Func`1[TResult] method, Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x0017c] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoWithRetry (System.Func`1[TResult] method, Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x003a5] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.UploadFileAsync (Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x000da] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.UploadBlockAndIndexAsync (Duplicati.Library.Main.Operation.Backup.VolumeUploadRequest upload, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x00098] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.<Run>b__13_0 (<>f__AnonymousType12`1[<Input>j__TPar] self) [0x00780] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
   --- End of inner exception stack trace ---
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.<Run>b__13_0 (<>f__AnonymousType12`1[<Input>j__TPar] self) [0x005ba] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.<Run>b__13_0 (<>f__AnonymousType12`1[<Input>j__TPar] self) [0x007d7] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at CoCoL.AutomationExtensions.RunTask[T] (T channels, System.Func`2[T,TResult] method, System.Boolean catchRetiredExceptions) [0x000d5] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.FlushBackend (Duplicati.Library.Main.BackupResults result, CoCoL.IWriteChannel`1[T] uploadtarget, System.Threading.Tasks.Task uploader) [0x000f2] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00cb2] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
---> (Inner Exception #0) System.OutOfMemoryException: Insufficient memory to continue the execution of the program.
  at (wrapper alloc) System.Object.AllocVector(intptr,intptr)
  at Duplicati.Library.Backend.MicrosoftGraphBackend.PutAsync (System.String remotename, System.IO.Stream stream, System.Threading.CancellationToken cancelToken) [0x007dc] in <dfa9574ee8124175856109d811c0b28b>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoPut (Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Interface.IBackend backend, System.Threading.CancellationToken cancelToken) [0x00426] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader+<>c__DisplayClass20_0.<UploadFileAsync>b__0 () [0x0010a] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoWithRetry (System.Func`1[TResult] method, Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x0017c] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoWithRetry (System.Func`1[TResult] method, Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x003a5] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.UploadFileAsync (Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x000da] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.UploadBlockAndIndexAsync (Duplicati.Library.Main.Operation.Backup.VolumeUploadRequest upload, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x00098] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.<Run>b__13_0 (<>f__AnonymousType12`1[<Input>j__TPar] self) [0x00780] in <e60bc008dd1b454d861cfacbdd3760b9>:0 <---

---> (Inner Exception #1) System.OutOfMemoryException: Insufficient memory to continue the execution of the program.
  at (wrapper alloc) System.Object.AllocVector(intptr,intptr)
  at Duplicati.Library.Backend.MicrosoftGraphBackend.PutAsync (System.String remotename, System.IO.Stream stream, System.Threading.CancellationToken cancelToken) [0x007dc] in <dfa9574ee8124175856109d811c0b28b>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoPut (Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Interface.IBackend backend, System.Threading.CancellationToken cancelToken) [0x00426] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader+<>c__DisplayClass20_0.<UploadFileAsync>b__0 () [0x0010a] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoWithRetry (System.Func`1[TResult] method, Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x0017c] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoWithRetry (System.Func`1[TResult] method, Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x003a5] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.UploadFileAsync (Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x000da] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.UploadBlockAndIndexAsync (Duplicati.Library.Main.Operation.Backup.VolumeUploadRequest upload, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x00098] in <e60bc008dd1b454d861cfacbdd3760b9>:0 <---

---> (Inner Exception #2) System.OutOfMemoryException: Insufficient memory to continue the execution of the program.
  at (wrapper alloc) System.Object.AllocVector(intptr,intptr)
  at Duplicati.Library.Backend.MicrosoftGraphBackend.PutAsync (System.String remotename, System.IO.Stream stream, System.Threading.CancellationToken cancelToken) [0x007dc] in <dfa9574ee8124175856109d811c0b28b>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoPut (Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Interface.IBackend backend, System.Threading.CancellationToken cancelToken) [0x00426] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader+<>c__DisplayClass20_0.<UploadFileAsync>b__0 () [0x0010a] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoWithRetry (System.Func`1[TResult] method, Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x0017c] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoWithRetry (System.Func`1[TResult] method, Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x003a5] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.UploadFileAsync (Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x000da] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.UploadBlockAndIndexAsync (Duplicati.Library.Main.Operation.Backup.VolumeUploadRequest upload, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x00098] in <e60bc008dd1b454d861cfacbdd3760b9>:0 <---

---> (Inner Exception #3) System.OutOfMemoryException: Insufficient memory to continue the execution of the program.
  at (wrapper alloc) System.Object.AllocVector(intptr,intptr)
  at Duplicati.Library.Backend.MicrosoftGraphBackend.PutAsync (System.String remotename, System.IO.Stream stream, System.Threading.CancellationToken cancelToken) [0x007dc] in <dfa9574ee8124175856109d811c0b28b>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoPut (Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Interface.IBackend backend, System.Threading.CancellationToken cancelToken) [0x00426] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader+<>c__DisplayClass20_0.<UploadFileAsync>b__0 () [0x0010a] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoWithRetry (System.Func`1[TResult] method, Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x0017c] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoWithRetry (System.Func`1[TResult] method, Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x003a5] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.UploadFileAsync (Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x000da] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.UploadBlockAndIndexAsync (Duplicati.Library.Main.Operation.Backup.VolumeUploadRequest upload, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x00098] in <e60bc008dd1b454d861cfacbdd3760b9>:0 <---
...

So I'm stuck with this backup...

[warwickmm]

Thanks for reporting @davidtgbe. Can you confirm that your version of mono is 5.2.0 as stated above? Starting with version 2.0.6.1, Duplicati requires mono 5.10 or greater.

[davidtgbe]

Sure @warwickmm. I started with the oob linuxserver docker image which comes with Mono 5.2.0 built in. When I got these errors I tried updating the container with the latest Mono version, which is 6.12.0.122 and is the one I'm using right now.
You can check this on mi first post in this thread.

Summary:

• Using docker container from: https://hub.docker.com/r/linuxserver/duplicati
• RPI 4 with 4GB
• Duplicati version: 2.0.6.3_beta_2021-06-17
• Mono version: tried with 5.2.0 and latest 6.12.0.122 (currently using this version)
• Endpoints tried:
  • od4b:
    • memory exception solved not using binary direct mode (which is the default option) binary-direct-mode=false
    • 5000 items limitation error after correctly uploading all files
  • OnedriveV2:
    • Memory exception error

[warwickmm]

Thanks @davidtgbe. What's the total size of the source files you're backing up?

Also, can you try setting use-http-client=true in the OneDrive advanced options to see if that helps? Setting this value to false (the default) was thought to help with some earlier memory issues with mono, but it would be interesting to see if how it affects your situation:

If you're willing to see if this behavior also occurs with the official Docker image, that would be interesting as well.

/cc @tygill in case they have any additional thoughts.

[tygill]

I actually use a similar setup - Duplicati on a 4GB raspberry pi 4 backing up to OneDrive via the onedrivev2 backend (though I switched from the linuxserver.io to the official Docker container recently). I do have a few settings tuned to avoid memory problems though, and they seem to help:

asynchronous-concurrent-upload-limit=2
concurrency-block-hashers=1
concurrency-compressors=1
concurrency-max-threads=0 (default)
disable-file-scanner=true

I also use use-http-client=false on Mono, though it's possible that the HttpClient wasn't actually causing problems (when that is false, the OneDrive v2 backend uses the same infra most other Duplicati backends use). HttpClient is the newer API in dotnet, and might have benefits on Windows / .NET Core, but I'm not sure what it's status on Mono is.

[davidtgbe]

Sorry for the late response and thanks for your suggestions.
@warwickmm I tried your suggestion use-http-client=true and it worked fine. It succesfully completed the backup.
Uploading process to Onedrive was very slow (1.2~1.9MB/s) so I started to try to find the problem as internet BW was not my case.

I then realized that a lot of IOWaits were ocurring on SDCard. I supposed this was due to all the compress/encrypt operations that were being performed on /tmp (files like ´dup-XXXXXXXXX...´) before uploading files because my source files where on an external USB disk. I dug a bit and found this .

This also could lead to potential damage on a SD card as time passes due to wear levelling, so I've added a tmpfs volume to my dupplicati container. My RPI has 4GB of RAM so it can deal with this.

sudo mount -t tmpfs -o defaults,noatime,nosuid,nodev,noexec,mode=1777,size=512M tmpfs /home/pi/docker/duplicati/tmp/

docker-compose.yml

version: "2.1"
services:
  duplicati:
    image: ghcr.io/linuxserver/duplicati
    container_name: duplicati
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/Madrid
      - CLI_ARGS=
    volumes:
      - /home/pi/docker/duplicati/config:/config
      - /home/pi/docker/duplicati/backups:/backups
      - /home/pi/docker/duplicati/sources:/sources
      - /home/pi/docker/duplicati/tmp:/tmp
    ports:
      - 8200:8200
    restart: unless-stopped

This has reduced considerably the amount of IO operations performed on SDCard. Now, only DB operations perform RW operations on SDCard. However, speed has not improved despite my system now is not waiting for IOs or running out of RAM/CPU.

I then decided to check @tygill configuration to try to shed some light to this thread. @tygill configuration, which uses http-client=false as opposed to @warwickmm suggestion, also worked fine.

It is weird, maybe now runs ok with both configurations as a side effect of the tmpfs volume. I've changed nothing but this. Source files dataset has not been changed.

I will keep on digging on a way to improve upload speed.