Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Input/output error accessing NBD #56

Closed
bbs2web opened this issue Oct 7, 2019 · 9 comments
Assignees
Labels

Comments

@bbs2web
Copy link

@bbs2web bbs2web commented Oct 7, 2019

I've configured Benji to use a dedicated PostgreSQL database and storing backups in a Ceph radosgw using both compression and encryption. Backups and incremental backups complete without errors and I'm able to restore backups to new RBD images where a sha512sum matches the source and destination volumes.

Testing NBD exports is however unsuccessful, are there any caveats that I should be aware of?

Server:
(benji) [root@kvm1a ~]# benji nbd -a 192.168.1.60 -r INFO: $ /usr/local/benji/bin/benji nbd -a 192.168.1.60 -r INFO: Starting to serve NBD on 192.168.1.60:10809 INFO: Incoming connection from 192.168.1.61:58222. INFO: [192.168.1.61:58222] Negotiated export: V0000000085. INFO: [192.168.1.61:58222] Version V0000000085 has been opened. INFO: [192.168.1.61:58222] Export is read only. INFO: Active transforms for storage radosgw: zstd (zstd), aes_256_gcm (aes_256_gcm). ERROR: [192.168.1.61:58222] 0 bytes read on a total of 28 expected bytes

Client:
[admin@kvm1b ~]# nbd-client -N V0000000085 192.168.1.60 -p 10809 -b 512 -t 10 /dev/nbd5 Negotiation: ..size = 10240MB Connected /dev/nbd5 [admin@kvm1b ~]# parted /dev/nbd5 Warning: Error fsyncing/closing /dev/nbd5: Input/output error Retry/Ignore? i GNU Parted 3.2 Using /dev/nbd5 Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) p Error: /dev/nbd5: unrecognised disk label Model: Unknown (unknown) Disk /dev/nbd5: 10.7GB Sector size (logical/physical): 512B/512B Partition Table: unknown Disk Flags: (parted) q Warning: Error fsyncing/closing /dev/nbd5: Input/output error Retry/Ignore? i [admin@kvm1b ~]# nbd-client -d /dev/nbd5

Size of NBD block device appears to be correct...

Benji backup I'm attempting to mount via NBD:
(benji) [root@kvm1a ~]# benji ls | grep -e '+' -e name -e V0000000085 INFO: $ /usr/local/benji/bin/benji ls +---------------------+-------------+---------------------------+-----------------------+----------+------------+--------+-----------+---------+ | date | uid | name | snapshot_name | size | block_size | status | protected | storage | +---------------------+-------------+---------------------------+-----------------------+----------+------------+--------+-----------+---------+ | 2019-10-07T21:03:54 | V0000000085 | office-sip-disk0 | b-2019-10-07T21:03:49 | 10.0GiB | 4.0MiB | valid | False | radosgw | +---------------------+-------------+---------------------------+-----------------------+----------+------------+--------+-----------+---------+

@bbs2web

This comment has been minimized.

Copy link
Author

@bbs2web bbs2web commented Oct 7, 2019

Running benji with debug logging level does not appear to indicate any more detail as to the problem I'm encountering:
(benji) [root@kvm1a ~]# benji --log-level DEBUG nbd -a 192.168.1.60 -r INFO: $ /usr/local/benji/bin/benji --log-level DEBUG nbd -a 192.168.1.60 -r DEBUG: commands.nbd(**{'bind_address': '192.168.1.60', 'bind_port': 10809, 'read_only': True}) DEBUG: Using block hash BLAKE2b with kwargs {'digest_bits': 256}. DEBUG: Resolved schema for benji.io.file-v1: {'configuration': {'type': 'dict', 'required': True, 'empty': False, 'schema': {'simultaneousWrites': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}, 'simultaneousReads': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}}}}. DEBUG: Resolved schema for benji.io.rbd-v1: {'configuration': {'type': 'dict', 'required': True, 'empty': False, 'schema': {'simultaneousWrites': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}, 'simultaneousReads': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}, 'cephConfigFile': {'type': 'string', 'empty': False, 'default': '/etc/ceph/ceph.conf'}, 'clientIdentifier': {'type': 'string', 'empty': False, 'default': 'admin'}, 'newImageFeatures': {'type': 'list', 'empty': False, 'default': ['RBD_FEATURE_LAYERING'], 'schema': {'type': 'string', 'regex': '^RBD_FEATURE_.*'}}}}}. DEBUG: Resolved schema for benji.io.rbdaio-v1: {'configuration': {'type': 'dict', 'required': True, 'empty': False, 'schema': {'simultaneousWrites': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}, 'simultaneousReads': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}, 'cephConfigFile': {'type': 'string', 'empty': False, 'default': '/etc/ceph/ceph.conf'}, 'clientIdentifier': {'type': 'string', 'empty': False, 'default': 'admin'}, 'newImageFeatures': {'type': 'list', 'empty': False, 'default': ['RBD_FEATURE_LAYERING'], 'schema': {'type': 'string', 'regex': '^RBD_FEATURE_.*'}}}}}. DEBUG: Resolved schema for benji.transform.zstd-v1: {'configuration': {'schema': {'level': {'type': 'integer', 'empty': False, 'default': 3, 'min': 1}, 'dictDataFile': {'type': 'string', 'empty': False}}}}. DEBUG: Resolved schema for benji.transform.aes_256_gcm-v1: {'configuration': {'required': True, 'schema': {'masterKey': {'type': 'string', 'empty': False, 'required': True, 'excludes': ['kdfSalt', 'kdfIterations', 'password']}, 'kdfSalt': {'type': 'string', 'empty': False, 'required': True, 'excludes': ['masterKey'], 'dependencies': ['kdfIterations', 'password']}, 'kdfIterations': {'type': 'integer', 'empty': False, 'required': True, 'min': 1000, 'excludes': ['masterKey'], 'dependencies': ['kdfSalt', 'password']}, 'password': {'type': 'string', 'empty': False, 'required': True, 'minlength': 8, 'excludes': ['masterKey'], 'dependencies': ['kdfSalt', 'kdfIterations']}}}}. DEBUG: Resolved schema for benji.storage.base-v1: {'configuration': {'type': 'dict', 'empty': False, 'schema': {'activeTransforms': {'type': 'list', 'empty': False, 'schema': {'type': 'string', 'empty': False}}, 'simultaneousWrites': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}, 'simultaneousReads': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}, 'simultaneousRemovals': {'type': 'integer', 'empty': False, 'min': 1, 'default': 5}, 'bandwidthRead': {'type': 'integer', 'empty': False, 'min': 0, 'default': 0}, 'bandwidthWrite': {'type': 'integer', 'empty': False, 'min': 0, 'default': 0}, 'consistencyCheckWrites': {'type': 'boolean', 'empty': False, 'default': False}, 'hmac': {'type': 'dict', 'empty': False, 'schema': {'password': {'type': 'string', 'empty': False, 'required': True, 'minlength': 8}, 'kdfSalt': {'type': 'string', 'empty': False, 'required': True}, 'kdfIterations': {'type': 'integer', 'empty': False, 'required': True, 'min': 1000}}}}}}. DEBUG: Resolved schema for benji.storage.base.ReadCache-v1: {'configuration': {'type': 'dict', 'empty': False, 'schema': {'activeTransforms': {'type': 'list', 'empty': False, 'schema': {'type': 'string', 'empty': False}}, 'simultaneousWrites': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}, 'simultaneousReads': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}, 'simultaneousRemovals': {'type': 'integer', 'empty': False, 'min': 1, 'default': 5}, 'bandwidthRead': {'type': 'integer', 'empty': False, 'min': 0, 'default': 0}, 'bandwidthWrite': {'type': 'integer', 'empty': False, 'min': 0, 'default': 0}, 'consistencyCheckWrites': {'type': 'boolean', 'empty': False, 'default': False}, 'hmac': {'type': 'dict', 'empty': False, 'schema': {'password': {'type': 'string', 'empty': False, 'required': True, 'minlength': 8}, 'kdfSalt': {'type': 'string', 'empty': False, 'required': True}, 'kdfIterations': {'type': 'integer', 'empty': False, 'required': True, 'min': 1000}}}, 'readCache': {'type': 'dict', 'empty': False, 'schema': {'directory': {'type': 'string', 'required': True, 'empty': False, 'dependencies': ['maximumSize', 'shards']}, 'maximumSize': {'type': 'integer', 'required': True, 'min': 1, 'dependencies': ['directory', 'shards']}, 'shards': {'type': 'integer', 'required': True, 'min': 1, 'dependencies': ['directory', 'maximumSize']}}}}, 'required': True}}. DEBUG: Resolved schema for benji.storage.s3-v1: {'configuration': {'type': 'dict', 'empty': False, 'schema': {'activeTransforms': {'type': 'list', 'empty': False, 'schema': {'type': 'string', 'empty': False}}, 'simultaneousWrites': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}, 'simultaneousReads': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}, 'simultaneousRemovals': {'type': 'integer', 'empty': False, 'min': 1, 'default': 5}, 'bandwidthRead': {'type': 'integer', 'empty': False, 'min': 0, 'default': 0}, 'bandwidthWrite': {'type': 'integer', 'empty': False, 'min': 0, 'default': 0}, 'consistencyCheckWrites': {'type': 'boolean', 'empty': False, 'default': False}, 'hmac': {'type': 'dict', 'empty': False, 'schema': {'password': {'type': 'string', 'empty': False, 'required': True, 'minlength': 8}, 'kdfSalt': {'type': 'string', 'empty': False, 'required': True}, 'kdfIterations': {'type': 'integer', 'empty': False, 'required': True, 'min': 1000}}}, 'readCache': {'type': 'dict', 'empty': False, 'schema': {'directory': {'type': 'string', 'required': True, 'empty': False, 'dependencies': ['maximumSize', 'shards']}, 'maximumSize': {'type': 'integer', 'required': True, 'min': 1, 'dependencies': ['directory', 'shards']}, 'shards': {'type': 'integer', 'required': True, 'min': 1, 'dependencies': ['directory', 'maximumSize']}}}, 'awsAccessKeyId': {'type': 'string', 'empty': False, 'required': True, 'excludes': ['awsAccessKeyIdFile']}, 'awsAccessKeyIdFile': {'type': 'string', 'empty': False, 'required': True, 'excludes': ['awsAccessKeyId']}, 'awsSecretAccessKey': {'type': 'string', 'empty': False, 'required': True, 'excludes': ['awsSecretAccessKeyFile']}, 'awsSecretAccessKeyFile': {'type': 'string', 'required': True, 'empty': False, 'excludes': ['awsSecretAccessKey']}, 'regionName': {'type': 'string', 'empty': False}, 'endpointUrl': {'type': 'string', 'empty': False}, 'useSsl': {'type': 'boolean', 'empty': False, 'default': True}, 'addressingStyle': {'type': 'string', 'empty': False}, 'signatureVersion': {'type': 'string', 'empty': False}, 'bucketName': {'type': 'string', 'empty': False, 'required': True}, 'disableEncodingType': {'type': 'boolean', 'empty': False, 'default': False}}, 'required': True}}. DEBUG: Current database schema revision: 368014edd88c. DEBUG: Expected database schema revision: 368014edd88c. DEBUG: Using selector: EpollSelector INFO: Starting to serve NBD on 192.168.1.60:10809 INFO: Incoming connection from 192.168.1.61:60822. DEBUG: [192.168.1.61:60822]: opt=7, length=17, data=b'\x00\x00\x00\x0bV0000000085\x00\x00' DEBUG: [192.168.1.61:60822]: opt=1, length=11, data=b'V0000000085' INFO: [192.168.1.61:60822] Negotiated export: V0000000085. INFO: [192.168.1.61:60822] Version V0000000085 has been opened. INFO: [192.168.1.61:60822] Export is read only. ERROR: [192.168.1.61:60822] 0 bytes read on a total of 28 expected bytes

@bbs2web

This comment has been minimized.

Copy link
Author

@bbs2web bbs2web commented Oct 7, 2019

I'm running Proxmox 6, kernel 5.0.21-2-pve with Ceph Nautilus 14.2.4

/etc/benji.yaml
`configurationVersion: '1'
logFile: /var/log/benji.log
databaseEngine: postgresql://benji-kvm1:****************@postgresql.domain.com:5432/benji-backup1
defaultStorage: radosgw
storages:

  • name: radosgw
    module: s3
    storageId: 1
    configuration:
    simultaneousReads: 12
    simultaneousWrites: 12
    activeTransforms:
    - zstd
    - aes_256_gcm
    awsAccessKeyId: '****************'
    awsSecretAccessKey: '********************************'
    bucketName: benji-kvm1
    endpointUrl: https://backup1.domain.com:7480/
    ios:
  • name: file
    module: file
  • name: rbd
    module: rbdaio
    configuration:
    simultaneousReads: 12
    simultaneousWrites: 12
    cephConfigFile: /etc/ceph/ceph.conf
    clientIdentifier: admin
    newImageFeatures:
    - RBD_FEATURE_LAYERING
    - RBD_FEATURE_EXCLUSIVE_LOCK
    - RBD_FEATURE_STRIPINGV2
    transforms:
  • name: zstd
    module: zstd
    configuration:
    level: 4
  • name: aes_256_gcm
    module: aes_256_gcm
    configuration:
    kdfSalt: '********************************'
    kdfIterations: 1000
    password: '****************'`
@elemental-lf

This comment has been minimized.

Copy link
Owner

@elemental-lf elemental-lf commented Oct 7, 2019

There have been numerous problems with nbd-client not all of which we cloud track down yet. By using -b 512 and -t 10 you should already have prevent two of them. What version of nbd-client is Proxmox 6 running? We had problems with 3.18 which we could not get to work at all. Also see https://benji-backup.me/restore.html#nbd-server.

@elemental-lf elemental-lf self-assigned this Oct 7, 2019
@bbs2web

This comment has been minimized.

Copy link
Author

@bbs2web bbs2web commented Oct 7, 2019

Proxmox Virtual Environment (PVE) 6 has a Debian 10 (buster) base, so I get 3.18 when I simply 'apt-get install nbd-client'.

Would have thought Debian 10 to provide the latest, many thanks for your quick feedback!

@elemental-lf

This comment has been minimized.

Copy link
Owner

@elemental-lf elemental-lf commented Oct 7, 2019

Are you sure? I just looked on packages.debian.org and the current version for buster is 3.19-3.

@bbs2web

This comment has been minimized.

Copy link
Author

@bbs2web bbs2web commented Oct 7, 2019

Just re-installed it again, Debian labels the package as 3.19 but running it reports 3.18. Perhaps the version wasn't bumped somewhere:

[admin@kvm1a ~]# apt show nbd-client Package: nbd-client Version: 1:3.19-3 <snip> [admin@kvm1a ~]# nbd-client --help nbd-client version 3.18 <snip>

Going to try 3.20 from sid...

@bbs2web

This comment has been minimized.

Copy link
Author

@bbs2web bbs2web commented Oct 7, 2019

Working perfectly with nbd-client 3.20 package from Debian sid:
http://debian.mirror.ac.za/debian/pool/main/n/nbd/nbd-client_3.20-1_amd64.deb

@elemental-lf

This comment has been minimized.

Copy link
Owner

@elemental-lf elemental-lf commented Oct 7, 2019

Oh, well, seems that I need to add another known issue. I just looked at the Debian package patch and it resets the version from 3.19 to 3.18 for whatever reason I don't know. So it seems to be 3.19, even though it reports 3.18. To make matters worse Oliver or I myself tested 3.19 on RHEL or Fedora (I don't remember) and it worked out of the box at the time. But there might have been some backports. Or the Debian package patch breaks 3.19. This is getting tedious. But I'm glad that you're up and running now!

@elemental-lf

This comment has been minimized.

Copy link
Owner

@elemental-lf elemental-lf commented Oct 8, 2019

I've added another known issue for nbd-client to the documentation. This change currently is in https://github.com/elemental-lf/benji/tree/doc-update and will be merged later after other changes went in.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.