Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ElastiSearch on cStor volumes with XFS went to Read-Only. #2337

Open
kmova opened this Issue Jan 3, 2019 · 0 comments

Comments

Projects
None yet
2 participants
@kmova
Copy link
Member

commented Jan 3, 2019

This issue was reported by @davidkarlsen in OpenEBS Slack.

Description

Running ElasticSearch on OpenEBS 0.8 - on cStor Volumes. The StorageClass is set to use XFS. The volumes seem to quickly get into Read-Only.

U0jDQ]]: failed to write index state
logging/logging-elasticsearch-master-2[elasticsearch]: java.nio.file.FileSystemException: /usr/share/elasticsearch/data/nodes/0/indices/fu9IxoBfSLm8Kv0pkU0jDQ/_state/state-12.st.tmp: Read-only file system

The following logs were also seen on the node:

[60690.268780] Buffer I/O error on dev sdc, logical block 2097153, lost async page write
[60690.268881] sd 7:0:0:0: [sdc] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[60690.268886] sd 7:0:0:0: [sdc] CDB: Write(10) 2a 00 01 00 08 d0 00 00 40 00
[60690.268889] print_req_error: I/O error, dev sdc, sector 16779472
[60690.268961] Buffer I/O error on dev sdc, logical block 2097434, lost async page write
[60690.269052] Buffer I/O error on dev sdc, logical block 2097435, lost async page write
[60690.269143] Buffer I/O error on dev sdc, logical block 2097436, lost async page write
[60690.269234] Buffer I/O error on dev sdc, logical block 2097437, lost async page write
[60690.269324] Buffer I/O error on dev sdc, logical block 2097438, lost async page write
[60690.269415] Buffer I/O error on dev sdc, logical block 2097439, lost async page write
[60690.269505] Buffer I/O error on dev sdc, logical block 2097440, lost async page write
[60690.269606] sd 7:0:0:0: [sdc] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[60690.269611] sd 7:0:0:0: [sdc] CDB: Write(10) 2a 00 00 00 00 10 00 00 08 00
[60690.269614] print_req_error: I/O error, dev sdc, sector 16
[60690.274185] sd 7:0:0:0: rejecting I/O to offline device
[60690.274228] print_req_error: I/O error, dev sdc, sector 0
[60690.274281] sd 7:0:0:0: rejecting I/O to offline device
[60690.274308] print_req_error: I/O error, dev sdc, sector 0
[60690.274352] sd 7:0:0:0: rejecting I/O to offline device
[60690.274379] print_req_error: I/O error, dev sdc, sector 0
[60690.274489] sd 7:0:0:0: rejecting I/O to offline device
[60690.274517] print_req_error: I/O error, dev sdc, sector 0
[60690.274561] sd 7:0:0:0: rejecting I/O to offline device
[60690.274593] JBD2: Error -5 detected when updating journal superblock for sdc-8.
[60690.274627] Aborting journal on device sdc-8.
[60690.274659] sd 7:0:0:0: rejecting I/O to offline device
[60690.274689] JBD2: Error -5 detected when updating journal superblock for sdc-8.
[60690.274768] sd 7:0:0:0: rejecting I/O to offline device
[60690.274810] EXT4-fs error (device sdc): ext4_put_super:887: Couldn't clean up the journal
[60690.274848] EXT4-fs (sdc): Remounting filesystem read-only
[60690.274877] EXT4-fs (sdc): previous I/O error to superblock detected
[60690.274918] sd 7:0:0:0: rejecting I/O to offline device
[60690.478885] sd 7:0:0:0: [sdc] Synchronizing SCSI cache
[60690.478934] sd 7:0:0:0: [sdc] Synchronize Cache(10) failed: Result: hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK

Expected Behavior

The Volumes should not have gone into read-only. In case the volume did go into read-only due to valid errors, there should be a clear indication on the cause via alerts or troubleshooting steps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.