Error while backing up to external app data via SFTP

[roever-rl]

Expected Behavior

Get a completed full backup over SFTP connection to external storage.

Current Behavior

Backup fails after backing up random number of chunks.

Failure Information (for bugs)

Steps to Reproduce
Install Nextcloud Hub II 23.0.3.
Install external storage and configure it with SFTP connection using username and password.
Install backup app 1.0.6 and set backups and temporary data to go to external storage.
Try to create a full backup.

Context

Please provide any relevant information about your setup. This is important in case the issue is not reproducible except for under certain conditions.

Nextcloud Hub II version: 23.0.3
Backup app version: 1.0.6
Nextcloud is installed on Ubuntu 20.04 LTS

I have configured app data to be on external storage configured with SFTP.
I can access the external storage with files.
Every time i run "sudo -u www-data php /var/www/nextcloud/occ backup:point:create" it will backup and create chunks on external storage.
I have a lot of data that is backed up so it takes several hours for a full backup.
But after a couple of hours it fails with message:
In ChunkService.php line 597:

Chunk data-8fe30df3-c083-4b2a-90ce-6878c0de32fe.zip not valid

Backup Output:

> maintenance mode: on
> initialization of the AppData
> initialization of the RestoringPoint: 20220414063407-full-lqJPD4Vv8mFlnvX based on NC23.0.3.2
> initialization of the storage
> preparation of the data to be stored in the restoring point
> preparation of internal data
> creating chunks
  * data: /opt/nextcloud/data/, 594386 files
    - data-bd4380e3-436a-4b08-9097-2187e099eb96
    - data-823febf1-db0a-4a67-a5fd-247e7be5fc54
    - data-460de213-0071-4874-a77b-9a4c3f584237
    - data-e7d87fbb-88bb-4bad-b7d1-c598a1d8615b
    - data-faae0aa3-5c23-43c5-8128-156c3047c8ac
    - data-23569b5b-ace1-4522-b496-5ce46816631f
    - data-8a5341ab-01f0-4021-9cf1-d4ae3841c171
    - data-c95887ca-27cb-458f-8243-bd5e2474c165
    - data-fa54cbbf-325e-4ed6-8fdc-7e70b9c6a474
    - data-3a67508b-4d3e-4e15-8374-bce11af48844
    - data-6a0fc0e4-4007-417c-95ca-f03217f5ff6e
    - data-a96fae05-59ca-4759-b6b7-245327a01b8e
    - data-715d3674-7144-4f26-aaa6-a75787776c48
    - data-f1ad0597-b6d7-46b8-a372-723acfb1ff7f
    - data-81993cdb-6a3a-4783-a66e-cf1b57b080ad
    - data-81016c23-c785-4213-82e7-093cca5fd4ab
    - data-cc1ac2b4-bef9-41aa-8561-6f54e6952d52
    - data-de2b3ff0-4303-4971-a015-0d3ebc2127ae
    - data-b5f905a4-be8d-48a2-994f-21bd0f8cdb29
    - data-1be3ce8c-cc2f-4a92-9c97-a8ea161dc8cc
    - data-4a53941c-0174-4ef7-a146-656eddc0f06a
    - data-489323de-33ba-4048-83ee-d1d89e2db5dd
    - data-a5268462-0469-416c-8d7a-45690f06beab
    - data-4ebc0d76-8b27-4569-920b-526214c5a7ad
    - data-148a2301-1bb8-4b0f-ae96-3ceedbe20e3e
    - data-f4c351dc-00ad-49d4-8cd5-f748103ea166
    - data-b4023dca-e092-411d-a000-faa150600c0b
    - data-8fe30df3-c083-4b2a-90ce-6878c0de32fe
> maintenance mode: off

In ChunkService.php line 597:

  Chunk data-8fe30df3-c083-4b2a-90ce-6878c0de32fe.zip not valid


backup:point:create [--output [OUTPUT]] [--generate-log] [--differential] [--] [<comment>]

Nextcloud log:

{"reqId":"di1OV9CSK6E4BqmFeNxD","level":3,"time":"2022-04-14T15:48:40+02:00","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":"fopen(sftpwrite://backups//nextcloudappdata/20220414063407-full-lqJPD4Vv8mFlnvX/data/data-8fe30df3-c083-4b2a-90ce-6878c0de32fe/.backup.data-8fe30df3-c083-4b2a-90ce-6878c0de32fe): failed to open stream: \"OCA\\Files_External\\Lib\\Storage\\SFTPWriteStream::stream_open\" call failed at /var/www/nextcloud/apps/files_external/lib/Lib/Storage/SFTP.php#387","userAgent":"--","version":"23.0.3.2","exception":{"Exception":"Error","Message":"fopen(sftpwrite://backups//nextcloudappdata/20220414063407-full-lqJPD4Vv8mFlnvX/data/data-8fe30df3-c083-4b2a-90ce-6878c0de32fe/.backup.data-8fe30df3-c083-4b2a-90ce-6878c0de32fe): failed to open stream: \"OCA\\Files_External\\Lib\\Storage\\SFTPWriteStream::stream_open\" call failed at /var/www/nextcloud/apps/files_external/lib/Lib/Storage/SFTP.php#387","Code":0,"Trace":[{"function":"onError","class":"OC\\Log\\ErrorHandler","type":"::"},{"file":"/var/www/nextcloud/apps/files_external/lib/Lib/Storage/SFTP.php","line":387,"function":"fopen"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Common.php","line":206,"function":"fopen","class":"OCA\\Files_External\\Lib\\Storage\\SFTP","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":258,"function":"file_put_contents","class":"OC\\Files\\Storage\\Common","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/PermissionsMask.php","line":112,"function":"file_put_contents","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":258,"function":"file_put_contents","class":"OC\\Files\\Storage\\Wrapper\\PermissionsMask","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Availability.php","line":274,"function":"file_put_contents","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":258,"function":"file_put_contents","class":"OC\\Files\\Storage\\Wrapper\\Availability","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/View.php","line":1169,"function":"file_put_contents","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/View.php","line":706,"function":"basicOperation","class":"OC\\Files\\View","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Node/Folder.php","line":198,"function":"file_put_contents","class":"OC\\Files\\View","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/SimpleFS/SimpleFolder.php","line":89,"function":"newFile","class":"OC\\Files\\Node\\Folder","type":"->"},{"file":"/var/www/nextcloud/apps/backup/lib/Service/ChunkService.php","line":551,"function":"newFile","class":"OC\\Files\\SimpleFS\\SimpleFolder","type":"->"},{"file":"/var/www/nextcloud/apps/backup/lib/Service/ChunkService.php","line":476,"function":"finalizeChunk","class":"OCA\\Backup\\Service\\ChunkService","type":"->"},{"file":"/var/www/nextcloud/apps/backup/lib/Service/ChunkService.php","line":378,"function":"generateChunk","class":"OCA\\Backup\\Service\\ChunkService","type":"->"},{"file":"/var/www/nextcloud/apps/backup/lib/Service/ChunkService.php","line":148,"function":"fillChunks","class":"OCA\\Backup\\Service\\ChunkService","type":"->"},{"file":"/var/www/nextcloud/apps/backup/lib/Service/PointService.php","line":258,"function":"createChunks","class":"OCA\\Backup\\Service\\ChunkService","type":"->"},{"file":"/var/www/nextcloud/apps/backup/lib/Command/PointCreate.php","line":120,"function":"create","class":"OCA\\Backup\\Service\\PointService","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/console/Command/Command.php","line":255,"function":"execute","class":"OCA\\Backup\\Command\\PointCreate","type":"->"},{"file":"/var/www/nextcloud/core/Command/Base.php","line":168,"function":"run","class":"Symfony\\Component\\Console\\Command\\Command","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/console/Application.php","line":1009,"function":"run","class":"OC\\Core\\Command\\Base","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/console/Application.php","line":273,"function":"doRunCommand","class":"Symfony\\Component\\Console\\Application","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/console/Application.php","line":149,"function":"doRun","class":"Symfony\\Component\\Console\\Application","type":"->"},{"file":"/var/www/nextcloud/lib/private/Console/Application.php","line":211,"function":"run","class":"Symfony\\Component\\Console\\Application","type":"->"},{"file":"/var/www/nextcloud/console.php","line":99,"function":"run","class":"OC\\Console\\Application","type":"->"},{"file":"/var/www/nextcloud/occ","line":11,"args":["/var/www/nextcloud/console.php"],"function":"require_once"}],"File":"/var/www/nextcloud/lib/private/Log/ErrorHandler.php","Line":92,"CustomMessage":"--"}}
{"reqId":"di1OV9CSK6E4BqmFeNxD","level":3,"time":"2022-04-14T15:48:40+02:00","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":"fwrite() expects parameter 1 to be resource, bool given at /var/www/nextcloud/lib/private/Files/Storage/Common.php#208","userAgent":"--","version":"23.0.3.2","exception":{"Exception":"Error","Message":"fwrite() expects parameter 1 to be resource, bool given at /var/www/nextcloud/lib/private/Files/Storage/Common.php#208","Code":0,"Trace":[{"function":"onError","class":"OC\\Log\\ErrorHandler","type":"::"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Common.php","line":208,"function":"fwrite"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":258,"function":"file_put_contents","class":"OC\\Files\\Storage\\Common","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/PermissionsMask.php","line":112,"function":"file_put_contents","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":258,"function":"file_put_contents","class":"OC\\Files\\Storage\\Wrapper\\PermissionsMask","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Availability.php","line":274,"function":"file_put_contents","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":258,"function":"file_put_contents","class":"OC\\Files\\Storage\\Wrapper\\Availability","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/View.php","line":1169,"function":"file_put_contents","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/View.php","line":706,"function":"basicOperation","class":"OC\\Files\\View","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Node/Folder.php","line":198,"function":"file_put_contents","class":"OC\\Files\\View","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/SimpleFS/SimpleFolder.php","line":89,"function":"newFile","class":"OC\\Files\\Node\\Folder","type":"->"},{"file":"/var/www/nextcloud/apps/backup/lib/Service/ChunkService.php","line":551,"function":"newFile","class":"OC\\Files\\SimpleFS\\SimpleFolder","type":"->"},{"file":"/var/www/nextcloud/apps/backup/lib/Service/ChunkService.php","line":476,"function":"finalizeChunk","class":"OCA\\Backup\\Service\\ChunkService","type":"->"},{"file":"/var/www/nextcloud/apps/backup/lib/Service/ChunkService.php","line":378,"function":"generateChunk","class":"OCA\\Backup\\Service\\ChunkService","type":"->"},{"file":"/var/www/nextcloud/apps/backup/lib/Service/ChunkService.php","line":148,"function":"fillChunks","class":"OCA\\Backup\\Service\\ChunkService","type":"->"},{"file":"/var/www/nextcloud/apps/backup/lib/Service/PointService.php","line":258,"function":"createChunks","class":"OCA\\Backup\\Service\\ChunkService","type":"->"},{"file":"/var/www/nextcloud/apps/backup/lib/Command/PointCreate.php","line":120,"function":"create","class":"OCA\\Backup\\Service\\PointService","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/console/Command/Command.php","line":255,"function":"execute","class":"OCA\\Backup\\Command\\PointCreate","type":"->"},{"file":"/var/www/nextcloud/core/Command/Base.php","line":168,"function":"run","class":"Symfony\\Component\\Console\\Command\\Command","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/console/Application.php","line":1009,"function":"run","class":"OC\\Core\\Command\\Base","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/console/Application.php","line":273,"function":"doRunCommand","class":"Symfony\\Component\\Console\\Application","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/console/Application.php","line":149,"function":"doRun","class":"Symfony\\Component\\Console\\Application","type":"->"},{"file":"/var/www/nextcloud/lib/private/Console/Application.php","line":211,"function":"run","class":"Symfony\\Component\\Console\\Application","type":"->"},{"file":"/var/www/nextcloud/console.php","line":99,"function":"run","class":"OC\\Console\\Application","type":"->"},{"file":"/var/www/nextcloud/occ","line":11,"args":["/var/www/nextcloud/console.php"],"function":"require_once"}],"File":"/var/www/nextcloud/lib/private/Log/ErrorHandler.php","Line":92,"CustomMessage":"--"}}
{"reqId":"di1OV9CSK6E4BqmFeNxD","level":3,"time":"2022-04-14T15:48:40+02:00","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":"fclose() expects parameter 1 to be resource, bool given at /var/www/nextcloud/lib/private/Files/Storage/Common.php#209","userAgent":"--","version":"23.0.3.2","exception":{"Exception":"Error","Message":"fclose() expects parameter 1 to be resource, bool given at /var/www/nextcloud/lib/private/Files/Storage/Common.php#209","Code":0,"Trace":[{"function":"onError","class":"OC\\Log\\ErrorHandler","type":"::"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Common.php","line":209,"function":"fclose"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":258,"function":"file_put_contents","class":"OC\\Files\\Storage\\Common","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/PermissionsMask.php","line":112,"function":"file_put_contents","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":258,"function":"file_put_contents","class":"OC\\Files\\Storage\\Wrapper\\PermissionsMask","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Availability.php","line":274,"function":"file_put_contents","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":258,"function":"file_put_contents","class":"OC\\Files\\Storage\\Wrapper\\Availability","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/View.php","line":1169,"function":"file_put_contents","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/View.php","line":706,"function":"basicOperation","class":"OC\\Files\\View","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Node/Folder.php","line":198,"function":"file_put_contents","class":"OC\\Files\\View","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/SimpleFS/SimpleFolder.php","line":89,"function":"newFile","class":"OC\\Files\\Node\\Folder","type":"->"},{"file":"/var/www/nextcloud/apps/backup/lib/Service/ChunkService.php","line":551,"function":"newFile","class":"OC\\Files\\SimpleFS\\SimpleFolder","type":"->"},{"file":"/var/www/nextcloud/apps/backup/lib/Service/ChunkService.php","line":476,"function":"finalizeChunk","class":"OCA\\Backup\\Service\\ChunkService","type":"->"},{"file":"/var/www/nextcloud/apps/backup/lib/Service/ChunkService.php","line":378,"function":"generateChunk","class":"OCA\\Backup\\Service\\ChunkService","type":"->"},{"file":"/var/www/nextcloud/apps/backup/lib/Service/ChunkService.php","line":148,"function":"fillChunks","class":"OCA\\Backup\\Service\\ChunkService","type":"->"},{"file":"/var/www/nextcloud/apps/backup/lib/Service/PointService.php","line":258,"function":"createChunks","class":"OCA\\Backup\\Service\\ChunkService","type":"->"},{"file":"/var/www/nextcloud/apps/backup/lib/Command/PointCreate.php","line":120,"function":"create","class":"OCA\\Backup\\Service\\PointService","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/console/Command/Command.php","line":255,"function":"execute","class":"OCA\\Backup\\Command\\PointCreate","type":"->"},{"file":"/var/www/nextcloud/core/Command/Base.php","line":168,"function":"run","class":"Symfony\\Component\\Console\\Command\\Command","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/console/Application.php","line":1009,"function":"run","class":"OC\\Core\\Command\\Base","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/console/Application.php","line":273,"function":"doRunCommand","class":"Symfony\\Component\\Console\\Application","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/console/Application.php","line":149,"function":"doRun","class":"Symfony\\Component\\Console\\Application","type":"->"},{"file":"/var/www/nextcloud/lib/private/Console/Application.php","line":211,"function":"run","class":"Symfony\\Component\\Console\\Application","type":"->"},{"file":"/var/www/nextcloud/console.php","line":99,"function":"run","class":"OC\\Console\\Application","type":"->"},{"file":"/var/www/nextcloud/occ","line":11,"args":["/var/www/nextcloud/console.php"],"function":"require_once"}],"File":"/var/www/nextcloud/lib/private/Log/ErrorHandler.php","Line":92,"CustomMessage":"--"}}

[AnAnalogGuy]

Have you tried if it's working locally?
If so, you might want to do a local backup and use restic to move your backup from local to your external storage via SFTP. Besides being a workaround for your problem, restic offers deduplication and therefore might reduce storage space and traffic.

[roever-rl]

Thank you. I'll try that.
I have another ptoblem where the latest version doesn't quite work with NC24 so i removed it temporarily. I am using a combination of VM backups and separate backups on the storage for now. I'll test it when i get back from a trip i'm going on.