Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenEBS jiva volume continues to accept reads even after disk loss (simulated) #1387

Closed
ksatchit opened this Issue Mar 27, 2018 · 5 comments

Comments

Projects
None yet
4 participants
@ksatchit
Copy link
Member

commented Mar 27, 2018

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

What happened:

A disk failure/loss was simulated on a single-replica OpenEBS volume by performing a truncate command on the sparse file (volume-head-000.img) in the node where the replica was scheduled. Reads (via dd from the client node) on the volume were seen to continue on the iSCSI target - though the Controller logs showed that the iSCSI target was stopped with I/O errors due to loss of backend storage.

What you expected to happen:

The truncate command was used to create a sparse file of the same name with a minimal size of 1M, and the read requests were being received for offsets beyond the current size of the disk - which should have failed/errored out. However, these continued (possibly returning junk/invalid data)

How to reproduce it (as minimally and precisely as possible):

  • Created a volume with a single replica (5G) using a test PVC spec
  • Discovered, logged in to iSCSI volume, identified scsi device name & wrote some test data (dd)
  • Started continuous reads (dd) w/ bs=1M, count=1000
  • Performed truncated file create with same filename as disk file - truncate -s 1M volume-head-000.img
  • Controller stopped target, changed device mode to RO, threw I/O errors due to backend not available
time="2018-03-27T18:06:19Z" level=info msg="New Session initiator name:iqn.1993-08.org.debian:01:c4c2da36103e,target name:iqn.2016-09.com.openebs.jiva:pvc-6628f296-31e9-11e8-9309-000c298ff5fc,ISID:0x23d030000"
time="2018-03-27T18:06:19Z" level=error msg="non support"
time="2018-03-27T18:06:19Z" level=warning msg="check condition"
time="2018-03-27T18:06:19Z" level=warning msg="check condition"
time="2018-03-27T18:06:19Z" level=warning msg="check condition"
time="2018-03-27T18:07:14Z" level=error msg="Replicator.ReadAt:0 EOF"
time="2018-03-27T18:07:14Z" level=error msg="Setting replica tcp://10.44.0.5:9502 to ERR due to: EOF"
time="2018-03-27T18:07:14Z" level=info msg="Set replica tcp://10.44.0.5:9502 to mode ERR"
time="2018-03-27T18:07:14Z" level=error msg="I/O error: tcp://10.44.0.5:9502: EOF"
time="2018-03-27T18:07:14Z" level=info msg="stopping target iqn.2016-09.com.openebs.jiva:pvc-6628f296-31e9-11e8-9309-000c298ff5fc ..."
Closing Listener ...
time="2018-03-27T18:07:14Z" level=info msg="target iqn.2016-09.com.openebs.jiva:pvc-6628f296-31e9-11e8-9309-000c298ff5fc stopped"
time="2018-03-27T18:07:14Z" level=info msg="Removing backend: tcp://10.44.0.5:9502"
time="2018-03-27T18:07:14Z" level=info msg="Monitoring stopped tcp://10.44.0.5:9502"
time="2018-03-27T18:07:14Z" level=info msg="Closing: 10.44.0.5:9502"
time="2018-03-27T18:07:14Z" level=error msg="I/O error: No backend available"
time="2018-03-27T18:07:14Z" level=error msg="I/O error: No backend available"

Anything else we need to know?:

  • This test is part of a resiliency suite that aims to inject component failures. The actions in
    this activity are intended to simulate actual disk loss conditions for the OpenEBS jiva volume replica. The
    expectation is to fail-fast on such conditions.
@vishnuitta

This comment has been minimized.

Copy link
Member

commented Jan 17, 2019

@payes, @pawan had diagnosed it to the a behaviour w/ sparse files and the impact how we do then open/write operations on them.
Can this be verified with sparse files on a different disk, and by detaching the disk?

@vishnuitta vishnuitta assigned ksatchit and unassigned vishnuitta Jan 17, 2019

@ksatchit

This comment has been minimized.

Copy link
Member Author

commented Jan 18, 2019

Detaching the external disk upon which the jiva replica is created causes the following logs:

time="2019-01-18T12:22:49Z" level=info msg="Closing replica"
time="2019-01-18T12:22:49Z" level=error msg="failed to create temp file: volume.meta while encoding the data to file"
time="2019-01-18T12:22:49Z" level=info msg="Addreplica tcp://10.16.38.64:9502"
time="2019-01-18T12:22:49Z" level=error msg="open existing revision counter file failed"
time="2019-01-18T12:22:49Z" level=error msg="Error in initializing revision counter while creating temp replica"
time="2019-01-18T12:22:49Z" level=error msg="Error adding replica, err: failed to create temp replica, error: open /openebs/revision.counter: read-only file system, will retry"
time="2019-01-18T12:22:51Z" level=info msg="Closing replica"
time="2019-01-18T12:22:51Z" level=error msg="failed to create temp file: volume.meta while encoding the data to file"
time="2019-01-18T12:22:51Z" level=info msg="Addreplica tcp://10.16.38.64:9502"
time="2019-01-18T12:22:51Z" level=error msg="open existing revision counter file failed"
time="2019-01-18T12:22:51Z" level=error msg="Error in initializing revision counter while creating temp replica"
time="2019-01-18T12:22:51Z" level=error msg="Error adding replica, err: failed to create temp replica, error: open /openebs/revision.counter: read-only file system, will retry"
time="2019-01-18T12:22:53Z" level=info msg="Closing replica"
time="2019-01-18T12:22:53Z" level=error msg="failed to create temp file: volume.meta while encoding the data to file"
time="2019-01-18T12:22:53Z" level=info msg="Addreplica tcp://10.16.38.64:9502"
time="2019-01-18T12:22:53Z" level=error msg="open existing revision counter file failed"
time="2019-01-18T12:22:53Z" level=error msg="Error in initializing revision counter while creating temp replica"
time="2019-01-18T12:22:53Z" level=error msg="Error adding replica, err: failed to create temp replica, error: open /openebs/revision.counter: read-only file system, will retry"```

However the replica itself continues to run (which is undesirable)

Workaround: A re-attach, remount & subsequent restart of the pod will ensure the data replica resyncs (if quorum is intact).

Note: Any crashes (or deletes), if they occur will cause the local mount point to be used for this replica.

utkarshmani1997 added a commit to utkarshmani1997/jiva that referenced this issue Jan 24, 2019

[TA4759] fix(replica): handle case of disk-detach
- Issue Ref: openebs/openebs#1387
- Log and handle the case of disk-detach cases (input/output error)
  in case it's `EIO` error fatal the replica.

Note:
- What happens to running replica ?
  Replica will be fataled.
- What will happen upon replica restart after fatal?
  Replica will always be fataled until disk is attached
- What will happen to controller ?
  It will remove the fataled replica, and will continue to serve
  IO's if quorum is met.

Signed-off-by: Utkarsh Mani Tripathi <utkarsh.tripathi@mayadata.io>

@vishnuitta vishnuitta added this to the 0.8.1 milestone Jan 28, 2019

@vishnuitta

This comment has been minimized.

Copy link
Member

commented Jan 28, 2019

When the disk on which jiva replicas are created gets detached, any subsequent IOs on the disk returns EIO. PR#175 in jiva handles this by doing fatal of the replica, so that, when the disk gets attached, replica starts serving IOs.
This simulation of disk detach, by removing or truncating sparse files, will not help. File descriptor due to open by jiva replica will NOT allow to remove or truncate the sparse file, but, once the replica gets exited, sparse file gets removed.
Hence, closing this issue.

@vishnuitta vishnuitta closed this Jan 28, 2019

@vishnuitta

This comment has been minimized.

Copy link
Member

commented Jan 28, 2019

re-opening this to cherry-pick the fix to v0.8.x branch and assigning to @utkarshmani1997

@vishnuitta vishnuitta reopened this Jan 28, 2019

utkarshmani1997 added a commit to utkarshmani1997/jiva that referenced this issue Jan 29, 2019

[TA4759] fix(replica): handle case of disk-detach
- Issue Ref: openebs/openebs#1387
- Log and handle the case of disk-detach cases (input/output error)
  in case it's `EIO` error fatal the replica.

Note:
- What happens to running replica ?
  Replica will be fataled.
- What will happen upon replica restart after fatal?
  Replica will always be fataled until disk is attached
- What will happen to controller ?
  It will remove the fataled replica, and will continue to serve
  IO's if quorum is met.

Signed-off-by: Utkarsh Mani Tripathi <utkarsh.tripathi@mayadata.io>

utkarshmani1997 added a commit to utkarshmani1997/jiva that referenced this issue Jan 29, 2019

[TA4759] fix(replica): handle case of disk-detach
- Issue Ref: openebs/openebs#1387
- Log and handle the case of disk-detach cases (input/output error)
  in case it's `EIO` error fatal the replica.

Note:
- What happens to running replica ?
  Replica will be fataled.
- What will happen upon replica restart after fatal?
  Replica will always be fataled until disk is attached
- What will happen to controller ?
  It will remove the fataled replica, and will continue to serve
  IO's if quorum is met.

Signed-off-by: Utkarsh Mani Tripathi <utkarsh.tripathi@mayadata.io>

vishnuitta added a commit to openebs/jiva that referenced this issue Jan 29, 2019

fix(cherrypick): pingTimeout changes and disk detach to 0.8.1 (#180)
* [TA4284] fix(rpc): close connection at the client & server on errors or pingTimeout (#168)

Signed-off-by: Utkarsh Mani Tripathi <utkarsh.tripathi@mayadata.io>

* [TA4759] fix(replica): handle case of disk-detach

- Issue Ref: openebs/openebs#1387
- Log and handle the case of disk-detach cases (input/output error)
  in case it's `EIO` error fatal the replica.

Note:
- What happens to running replica ?
  Replica will be fataled.
- What will happen upon replica restart after fatal?
  Replica will always be fataled until disk is attached
- What will happen to controller ?
  It will remove the fataled replica, and will continue to serve
  IO's if quorum is met.

Signed-off-by: Utkarsh Mani Tripathi <utkarsh.tripathi@mayadata.io>
@vishnuitta

This comment has been minimized.

Copy link
Member

commented Feb 1, 2019

cherry-pick is also done.. hence, closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.