Cannot restore backups #99

via-justa · 2021-12-21T14:46:33Z

Describe the bug
When downloading backup from nextcloud to HA receiving a 400 error code

Fail to upload backup to home assistant (HTTPError: Response code 400 (Bad Request)) !

When downloading the backup to the computer and uploading to HA via SMB supervisor report an error

21-12-21 15:34:26 ERROR (MainThread) [supervisor.backups.backup] Can't read backup tarfile /data/tmp/tmpaw_p2i6f/backup.tar: invalid header

To Reproduce
download backup from nextcloud to HA either via the GUI or manually

Expected behavior
Backups are downloaded successfully

System information

Home Assistant Version : 2021.12.4
Home Assistant Supervisor Version : supervisor-2021.12.2
Nextcloud Version : 22.2.0
Addon version : 0.16.4
If any, reverse proxy in fort on Nextcloud : It's an hosted service on Hetzner so I do not have access to the infrastructure to know that

Additional context
I managed to restore a few months ago without an issue with the same configuration on Nextcloud

The text was updated successfully, but these errors were encountered:

Sebclem · 2021-12-22T09:05:36Z

I think there is something wrong with your HA install.

Have you tried to do a manual snapshot ( in the supervisor panel, not with this addon ), manually downloading it et manually uploading it ?

Let me know is this work

palvarez89 · 2022-01-02T11:11:07Z

I'm having a similar (if not the same) issue when trying to restore these backups after re-installing HA, luckily I used a different SD card.

To give more information:

Create a backup using NC backup
Download backup from HA
Download backup from NC
Upload backup downloaded from HA works
Upload backup downloaded from NC fails
Upload backup from NC interface fails

Those failures fail with Fail to upload backup to home assistant (HTTPError: Response code 400 (Bad Request)).

When I try to see the contents of these 2 different files I get errors on the one downloaded from NC:

$ tar -tvf Manual_2021_12_0_2022_01_02_1026.tar 
drwx------ root/root         0 2022-01-02 10:27 ./
-rw-r--r-- root/root      1777 2022-01-02 10:26 ./229cc4d7_nextcloud_backup.tar.gz
-rw-r--r-- root/root      2467 2022-01-02 10:27 ./7ad98f9c_zigbee2mqtt.tar.gz
-rw-r--r-- root/root    292983 2022-01-02 10:26 ./a0d7b954_nginxproxymanager.tar.gz
-rw-r--r-- root/root      1575 2022-01-02 10:26 ./a0d7b954_nodered.tar.gz
-rw-r--r-- root/root    637163 2022-01-02 10:27 ./a0d7b954_unifi.tar.gz
-rw-r--r-- root/root       176 2022-01-02 10:27 ./addons_local.tar.gz
-rw------- root/root      2297 2022-01-02 10:27 ./backup.json
-rw-r--r-- root/root      1229 2022-01-02 10:26 ./core_check_config.tar.gz
-rw-r--r-- root/root     83556 2022-01-02 10:26 ./core_duckdns.tar.gz
-rw-r--r-- root/root      1518 2022-01-02 10:26 ./core_git_pull.tar.gz
-rw-r--r-- root/root   2059114 2022-01-02 10:26 ./core_mariadb.tar.gz
-rw-r--r-- root/root     10682 2022-01-02 10:26 ./core_mosquitto.tar.gz
-rw-r--r-- root/root      8492 2022-01-02 10:26 ./core_ssh.tar.gz
-rw-r--r-- root/root  13129050 2022-01-02 10:27 ./homeassistant.tar.gz
-rw-r--r-- root/root       173 2022-01-02 10:27 ./media.tar.gz
-rw-r--r-- root/root     13520 2022-01-02 10:27 ./share.tar.gz
-rw-r--r-- root/root     57795 2022-01-02 10:27 ./ssl.tar.gz

$ tar -tvf Manual-2021.12.0-2022-01-02_1026.tar 
tar: This does not look like a tar archive
tar: Skipping to next header
-rw-r--r-- root/root      1575 2022-01-02 10:26 ./a0d7b954_nodered.tar.gz
-rw-r--r-- root/root    637163 2022-01-02 10:27 ./a0d7b954_unifi.tar.gz
tar: Skipping to next header
tar: Exiting with failure status due to previous errors

How come these files are different? Are they modified in any way before uploading?

HA version: 2021.12.0
HA supervisor version: supervisor-2021.12.2
Raspberry PI 4 64 bits aarch64
Addon version: 0.16.4

Thanks!

Sebclem · 2022-01-02T11:36:32Z

Hum... Interesting.

The file is not modified, the add-on just download the snapshot from HA and upload it to Nexcloud, the only issue I see, is that there is something wrong when either downloading or uploading the snapshot.

Did you have any errors on the supervisor logs, add-on logs or Nexcloud logs ( or reverse proxy if you have any ) ?

palvarez89 · 2022-01-03T00:20:36Z

I've tried to debug this a bit more.

Some steps I've executed now to reproduce:

Local backup works:

Manually created backup
Downloaded using API: curl -X GET -H "Authorization: Bearer ${SUPERVISOR_TOKEN}" -H "Content-Type: application/json" http://hassio/backups/a47f21e7/download -o a47f21e7.tar
Verified I can read the file using tar -tvf a47f21e7.tar

Uploading the backup breaks the tar file.

Open NC Backup UI
Selected local backup and clicked Backup now
Downloaded backup from NC, and checked I can not read the contents using tar -tvf a47f21e7.tar

This upload process only generated these logs, no errors.

22-01-02 23:49:44 INFO (MainThread) [supervisor.api.middleware.security] /snapshots/a47f21e7/download access from 229cc4d7_nextcloud_backup
22-01-02 23:49:44 INFO (MainThread) [supervisor.api.backups] Downloading backup a47f21e7
22-01-02 23:50:11 INFO (MainThread) [supervisor.api.middleware.security] /snapshots access from 229cc4d7_nextcloud_backup

Just compared a bit the 2 files, and they are very differnt:

$ xxd a47f21e7-good.tar | head
00000000: 2e2f 2e2f 4050 6178 4865 6164 6572 0000  ././@PaxHeader..
00000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000030: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000040: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000050: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000060: 0000 0000 3030 3030 3030 3000 3030 3030  ....0000000.0000
00000070: 3030 3000 3030 3030 3030 3000 3030 3030  000.0000000.0000
00000080: 3030 3030 3033 3400 3030 3030 3030 3030  0000034.00000000
00000090: 3030 3000 3031 3032 3132 0020 7800 0000  000.010212. x...

$ xxd a47f21e7.tar | head
00000000: 1338 2ed7 18fd c220 2899 8033 46c5 049c  .8..... (..3F...
00000010: 09eb dcff 2189 5933 edfe 0fcd c3b3 9b80  ....!.Y3........
00000020: b349 df04 1edf 239b 33ff 3390 d3e8 fc0f  .I....#.3.3.....
00000030: 9106 f34a f77f 064a ab99 ff19 08af f239  ...J...J.......9
00000040: 7af2 a6db e63f 7bfe 67f3 9fca f91f 3431  z....?{.g.....41
00000050: fcc3 7f10 21d9 84fb 1bb2 f9cf e7fd 1f7d  ....!..........}
00000060: feb3 f7bf e9f2 9ffc 66e2 2783 7071 90ff  ........f.'.pq..
00000070: 9060 498c ffb4 c8ac cc7f 481d f9c5 f84f  .`I.......H....O
00000080: 515c 7a72 fbb4 9eff 9080 4e9a ff10 b5a8  Q\zr......N.....
00000090: c17f 6e8a f662 00d6 9bff f911 3096 fca7  ..n..b......0...

$ xxd a47f21e7.tar | grep PaxHeader | wc -l
22
$ xxd a47f21e7-good.tar | grep PaxHeader | wc -l
18

Sebclem · 2022-01-07T16:09:20Z

Thank for all this digup.

It's very strange, maybe there is something wrong during the snapshot download...
Did you see something special in the addon logs during the backup ?

Sebclem · 2022-01-07T16:11:48Z

You can change the login level in the addon config (in supervisor panel), change it form info to debug, maybe we can see something here

palvarez89 · 2022-02-10T00:18:38Z

Here some logs:

Startup

[core-ssh config]$ ha addons logs 229cc4d7_nextcloud_backup
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] 10-run: applying... 
[fix-attrs.d] 10-run: exited 1.
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 00-banner.sh: executing... 
-----------------------------------------------------------
 Add-on: Nextcloud Backup
 Easily backup your Home Assistant snapshots to Nextcloud
-----------------------------------------------------------
 Add-on version: 0.16.5
 You are running the latest version of this add-on.
 System: Home Assistant OS 7.0  (aarch64 / raspberrypi4-64)
 Home Assistant Core: 2021.12.0
 Home Assistant Supervisor: 2021.12.2
-----------------------------------------------------------
 Please, share the above information when looking for help
 or support in, e.g., GitHub, forums or the Discord chat.
-----------------------------------------------------------
[cont-init.d] 00-banner.sh: exited 0.
[cont-init.d] 01-log-level.sh: executing... 
Log level is set to DEBUG
[cont-init.d] 01-log-level.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[14:36:58] INFO: Starting Node...
> nexcloud-backup@0.8.0 start
> node ./bin/www
[2022-01-31 14:37:04] [info]: 	Satus : Go !
[2022-01-31 14:37:04] [info]: 	Initializing and checking webdav client...
[2022-01-31 14:37:05] [info]: 	Starting auto clean cron...
[2022-01-31 14:37:05] [info]: 	Starting Cron...
[2022-01-31 14:37:05] [info]: 	Hassio API : Go !
[2022-01-31 14:37:06] [debug]: 	Nextcloud connection:  Success !
[2022-01-31 14:37:06] [debug]: 	Path /Hassio Backup/ already exist.
[2022-01-31 14:37:07] [info]: 	Nextcloud connection : Go !
[2022-02-01 01:00:00] [debug]: 	Snapshot size: 165.81
(node:346) UnhandledPromiseRejectionWarning: undefined
(Use `node --trace-warnings ...` to show where the warning was created)
(node:346) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:346) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
[2022-02-02 01:00:00] [debug]: 	Snapshot size: 165.81
(node:346) UnhandledPromiseRejectionWarning: undefined
(node:346) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)

Uploading a previously done snapshot via Home Assistant UI:


[2022-02-09 21:46:40] [info]: 	Downloading snapshot a57da1f5...
[2022-02-09 21:46:40] [debug]: 	Snapshot size: 247.36
[2022-02-09 21:47:11] [info]: 	Download success !
[2022-02-09 21:47:11] [debug]: 	Snapshot dl size : 247.36328125
[2022-02-09 21:47:11] [info]: 	Uploading snap...
[2022-02-09 21:47:11] [debug]: 	...URI: http://host.hiden/remote.php/webdav/Hassio%20Backup/Manual/full_2022_02_09_entity_name_changes.tar

Uploading the backup back to Home Assistant:


[2022-02-09 23:05:46] [info]: 	Downloading backup...
[2022-02-09 23:05:46] [debug]: 	...URI: http://host.hiden/remote.php/webdav/Hassio%20Backup/Manual/full_2022_02_09_entity_name_changes.tar
[2022-02-09 23:06:42] [info]: 	Download success !
[2022-02-09 23:06:42] [debug]: 	Backup dl size : 247.36328125
[2022-02-09 23:06:42] [info]: 	Uploading backup...
[2022-02-09 23:07:15] [info]: 	Upload done...
[2022-02-09 23:07:15] [error]: 	Fail to upload backup to home assistant (HTTPError: Response code 400 (Bad Request)) !
(node:346) UnhandledPromiseRejectionWarning: Fail to upload backup to home assistant (HTTPError: Response code 400 (Bad Request)) !
(node:346) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 12)

Can confirm the tarball in Nextcloud is already corrupted, that's why the second part of the logs fails.

palvarez89 · 2022-02-10T01:49:39Z

When reading the upload code, I realised there is an SSL setting that I should have enabled (after reading the docs).

With this setting on, I managed to backup a snapshot and then restore it locally. I wonder if this fixes things too for @via-justa?

I have no idea why this was introducing these type of issues, and not just crashing when creating the connection with nextcloud or something.. 😖

Sebclem · 2022-02-14T10:43:35Z

As @via-justa is not responding, I'm closing this issue, feel free to re-open/comment is you sill have this issue

via-justa added the bug Something isn't working label Dec 21, 2021

Sebclem closed this as completed Feb 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot restore backups #99

Cannot restore backups #99

via-justa commented Dec 21, 2021

Sebclem commented Dec 22, 2021

palvarez89 commented Jan 2, 2022

Sebclem commented Jan 2, 2022

palvarez89 commented Jan 3, 2022

Sebclem commented Jan 7, 2022

Sebclem commented Jan 7, 2022

palvarez89 commented Feb 10, 2022 •

edited

Loading

palvarez89 commented Feb 10, 2022

Sebclem commented Feb 14, 2022

Cannot restore backups #99

Cannot restore backups #99

Comments

via-justa commented Dec 21, 2021

Sebclem commented Dec 22, 2021

palvarez89 commented Jan 2, 2022

Sebclem commented Jan 2, 2022

palvarez89 commented Jan 3, 2022

Sebclem commented Jan 7, 2022

Sebclem commented Jan 7, 2022

palvarez89 commented Feb 10, 2022 • edited Loading

palvarez89 commented Feb 10, 2022

Sebclem commented Feb 14, 2022

palvarez89 commented Feb 10, 2022 •

edited

Loading