New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disk I/O fails with EREMOTEIO on Oracle OCI #2180

Closed
bgilbert opened this Issue Oct 4, 2017 · 16 comments

Comments

Projects
None yet
4 participants
@bgilbert
Member

bgilbert commented Oct 4, 2017

Issue Report

Bug

Container Linux Version

$ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1548.0.0
VERSION_ID=1548.0.0
BUILD_ID=2017-09-27-0012
PRETTY_NAME="Container Linux by CoreOS 1548.0.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

Environment

Oracle OCI

Expected Behavior

Disk I/O succeeds.

Actual Behavior

There are two cases, both involving EREMOTEIO (errno 121).

Case 1

[   14.084393] EXT4-fs (sda9): resizing filesystem from 553472 to 11602427 blocks
[   14.086381] EXT4-fs (sda9): resized filesystem to 11602427
[   14.520751] VFS: brelse: Trying to free free buffer
[   14.521336] ------------[ cut here ]------------
[   14.521818] WARNING: CPU: 0 PID: 846 at ../source/fs/buffer.c:1205 __brelse+0x23/0x30
[   14.522573] Modules linked in: xt_comment xt_owner iptable_mangle sb_edac edac_core kvm_intel kvm i2c_piix4 mousedev irqbypass psmouse evdev i2c_core efi_pstore efivars button sch_fq_codel nls_ascii nls_cp437 vfat fat ext4 crc16 mbcache jbd2 fscrypto dm_verity dm_bufio sd_mod iscsi_tcp libiscsi_tcp libiscsi crc32c_intel aesni_intel aes_x86_64 ata_piix crypto_simd cryptd glue_helper libata ixgbevf qemu_fw_cfg scsi_transport_iscsi scsi_mod dm_mirror dm_region_hash dm_log dm_mod dax
[   14.526698] CPU: 0 PID: 846 Comm: resize2fs Not tainted 4.13.3-coreos-r1 #1
[   14.527393] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 0.0.0 02/06/2015
[   14.528156] task: ffffa0b574335ac0 task.stack: ffffc04741338000
[   14.528708] RIP: 0010:__brelse+0x23/0x30
[   14.529107] RSP: 0018:ffffc0474133bc30 EFLAGS: 00010286
[   14.529634] RAX: 0000000000000027 RBX: ffffa0b57618a1a0 RCX: 0000000000000000
[   14.530338] RDX: 0000000000000000 RSI: ffffa0b57fc0df38 RDI: ffffa0b57fc0df38
[   14.531043] RBP: ffffc0474133bc30 R08: 0000000000000030 R09: 000000000000025c
[   14.531743] R10: ffffc0474133bc18 R11: 000000000000025b R12: ffffa0b57669ee88
[   14.532453] R13: ffffa0b4a05a9000 R14: 0000000000000035 R15: ffffa0b57618a1a0
[   14.533166] FS:  00007facdcca6740(0000) GS:ffffa0b57fc00000(0000) knlGS:0000000000000000
[   14.533958] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   14.534480] CR2: 00007f5096dc8160 CR3: 00000001f46eb000 CR4: 00000000001406f0
[   14.535132] Call Trace:
[   14.535400]  jbd2_journal_put_journal_head+0xc2/0x1a1 [jbd2]
[   14.535955]  __jbd2_journal_remove_checkpoint+0x5b/0x170 [jbd2]
[   14.536547]  jbd2_log_do_checkpoint+0x375/0x480 [jbd2]
[   14.537079]  ? wait_woken+0x80/0x80
[   14.537435]  jbd2_journal_flush+0x8b/0x180 [jbd2]
[   14.537901]  ? jbd2_journal_flush+0x8b/0x180 [jbd2]
[   14.538414]  ext4_ioctl+0x1163/0x15e0 [ext4]
[   14.538848]  do_vfs_ioctl+0x9f/0x5e0
[   14.539222]  ? getnstimeofday64+0xe/0x20
[   14.539619]  SyS_ioctl+0x79/0x90
[   14.539949]  do_syscall_64+0x5a/0x160
[   14.540344]  entry_SYSCALL64_slow_path+0x25/0x25
[   14.540771] RIP: 0033:0x7facdc160b97
[   14.541112] RSP: 002b:00007fff7a9ce238 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[   14.541791] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007facdc160b97
[   14.542500] RDX: 00007fff7a9ce350 RSI: 0000000040086610 RDI: 0000000000000004
[   14.543209] RBP: 00005556e6839030 R08: 0000000000000000 R09: 0000000000000029
[   14.543911] R10: 0000000000000000 R11: 0000000000000246 R12: 00005556e6839090
[   14.544612] R13: 0000000000000000 R14: 00005556e683c530 R15: 00007fff7a9ce350
[   14.545326] Code: 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 8b 47 60 85 c0 74 05 f0 ff 4f 60 c3 55 48 c7 c7 28 db 9f 97 31 c0 48 89 e5 e8 f0 74 e7 ff <0f> ff 5d c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 
[   14.547143] ---[ end trace cfe53a9cce2f416c ]---
[   14.598598]  connection1:0: detected conn error (1019)

(The last line of that log is not normally present, but happened to appear when I went to capture a sample.)

This is caused by extend-filesystems.service. journalctl -u extend-filesystems.service says:

Oct 04 21:47:30 localhost systemd[1]: Starting Extend Filesystems...
Oct 04 21:47:34 localhost extend-filesystems[803]: resize2fs 1.43.3 (04-Sep-2016)
Oct 04 21:47:34 localhost extend-filesystems[803]: resize2fs: Remote I/O error While checking for on-line resizing support
Oct 04 21:47:34 localhost extend-filesystems[803]: Filesystem at /dev/sda9 is mounted on /; on-line resizing required
Oct 04 21:47:34 localhost extend-filesystems[803]: old_desc_blocks = 1, new_desc_blocks = 6
Oct 04 21:47:34 localhost systemd[1]: extend-filesystems.service: Main process exited, code=exited, status=1/FAILURE
Oct 04 21:47:34 localhost systemd[1]: Failed to start Extend Filesystems.
Oct 04 21:47:34 localhost systemd[1]: extend-filesystems.service: Unit entered failed state.
Oct 04 21:47:34 localhost systemd[1]: extend-filesystems.service: Failed with result 'exit-code'.

Afterward, the root filesystem has not been resized. Manually running resize2fs on the root filesystem produces:

[  512.094780] EXT4-fs (sda9): resizing filesystem from 557056 to 11602427 blocks
[  512.351068] EXT4-fs (sda9): resized filesystem to 11602427

and resizes the filesystem.

Case 2

These log messages, appearing once per boot:

[   16.345604] EXT4-fs (sda9): Delayed block allocation failed for inode 59 at logical offset 382 with max blocks 2 with error 121
[   16.347144] EXT4-fs (sda9): This should not happen!! Data will be lost

Reproduction Steps

  1. Boot Container Linux.
  2. Check journal.

Other Information

This doesn't reproduce 100% reliably, but generally seems to happen at least once every two or three instance launches.

I have not seen this on bare metal, only VMs. I also reproduced case 1 with a 4.12.14 kernel as well as the 4.13.3 kernel above.

@bgilbert

This comment has been minimized.

Show comment
Hide comment
@bgilbert

bgilbert Oct 6, 2017

Member

This happens even with iscsid disabled. I also saw an instance of case 2 well after iscsid connection recovery finished.

Member

bgilbert commented Oct 6, 2017

This happens even with iscsid disabled. I also saw an instance of case 2 well after iscsid connection recovery finished.

@bgilbert

This comment has been minimized.

Show comment
Hide comment
@bgilbert

bgilbert Oct 6, 2017

Member

Some context on EREMOTEIO: torvalds/linux@63583cc

Member

bgilbert commented Oct 6, 2017

Some context on EREMOTEIO: torvalds/linux@63583cc

@bgilbert

This comment has been minimized.

Show comment
Hide comment
@bgilbert

bgilbert Oct 16, 2017

Member

I'm now unable to reproduce this either with a newly-built image or with an existing image that was failing before. Something may have changed in the implementation of the OCI iSCSI target. kola now has additional checks (coreos/mantle#733) that should catch the problem if it shows up again, so I'll close for now. Please reopen if the problem recurs.

Member

bgilbert commented Oct 16, 2017

I'm now unable to reproduce this either with a newly-built image or with an existing image that was failing before. Something may have changed in the implementation of the OCI iSCSI target. kola now has additional checks (coreos/mantle#733) that should catch the problem if it shows up again, so I'll close for now. Please reopen if the problem recurs.

@bgilbert bgilbert closed this Oct 16, 2017

@arithx arithx reopened this Nov 29, 2017

@arithx

This comment has been minimized.

Show comment
Hide comment
@arithx

arithx Nov 29, 2017

Nov 29 00:23:16 localhost systemd[1]: Starting Open-iSCSI...
Nov 29 00:23:16 localhost iscsid[1310]: iSCSI logger with pid=1311 started!
Nov 29 00:23:19 localhost kernel: VFS: brelse: Trying to free free buffer
Nov 29 00:23:19 localhost kernel: ------------[ cut here ]------------
Nov 29 00:23:19 localhost kernel: WARNING: CPU: 0 PID: 1268 at ../../../../../../../../usr/src/linux-4.14-rc8-coreos/fs/buffer.c:1205 __
Nov 29 00:23:19 localhost kernel: Modules linked in: xt_comment xt_owner iptable_mangle sb_edac edac_core kvm_intel mousedev psmouse i2c
Nov 29 00:23:19 localhost kernel: CPU: 0 PID: 1268 Comm: resize2fs Not tainted 4.14.0-rc8-coreos #1
Nov 29 00:23:19 localhost kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 0.0.0 02/06/2015
Nov 29 00:23:19 localhost kernel: task: ffff96ae83ea8000 task.stack: ffffbd2f47bb8000
Nov 29 00:23:19 localhost kernel: RIP: 0010:__brelse+0x23/0x30
Nov 29 00:23:19 localhost kernel: RSP: 0018:ffffbd2f47bbbc30 EFLAGS: 00010286
Nov 29 00:23:19 localhost kernel: RAX: 0000000000000027 RBX: ffff96ae862f92d8 RCX: 0000000000000000
Nov 29 00:23:19 localhost kernel: RDX: 0000000000000000 RSI: ffff96ae8ee0df38 RDI: ffff96ae8ee0df38
Nov 29 00:23:19 localhost kernel: RBP: ffffbd2f47bbbc30 R08: 0000000000000030 R09: 00000000000002dd
Nov 29 00:23:19 localhost kernel: R10: ffffbd2f47bbbc18 R11: 00000000000002dc R12: ffff96ae862df348
Nov 29 00:23:19 localhost kernel: R13: ffff96ae7e63f700 R14: 0000000000000038 R15: ffff96ae862f92d8
Nov 29 00:23:19 localhost kernel: FS:  00007f671eca7740(0000) GS:ffff96ae8ee00000(0000) knlGS:0000000000000000
Nov 29 00:23:19 localhost kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 29 00:23:19 localhost kernel: CR2: 000055606b888170 CR3: 0000001bc1239003 CR4: 00000000001606f0
Nov 29 00:23:19 localhost kernel: Call Trace:
Nov 29 00:23:19 localhost kernel:  jbd2_journal_put_journal_head+0xc2/0x1a1 [jbd2]
Nov 29 00:23:19 localhost kernel:  __jbd2_journal_remove_checkpoint+0x5b/0x170 [jbd2]
Nov 29 00:23:19 localhost kernel:  jbd2_log_do_checkpoint+0x375/0x480 [jbd2]
Nov 29 00:23:19 localhost kernel:  ? wait_woken+0x80/0x80
Nov 29 00:23:19 localhost kernel:  jbd2_journal_flush+0x8b/0x180 [jbd2]
Nov 29 00:23:19 localhost kernel:  ? jbd2_journal_flush+0x8b/0x180 [jbd2]
Nov 29 00:23:19 localhost kernel:  ext4_ioctl+0x1163/0x15e0 [ext4]
Nov 29 00:23:19 localhost kernel:  do_vfs_ioctl+0x9f/0x5e0
Nov 29 00:23:19 localhost kernel:  SyS_ioctl+0x79/0x90
Nov 29 00:23:19 localhost kernel:  do_syscall_64+0x5e/0x1d0
Nov 29 00:23:19 localhost kernel:  entry_SYSCALL64_slow_path+0x25/0x25
Nov 29 00:23:19 localhost kernel: RIP: 0033:0x7f671e161b97
Nov 29 00:23:19 localhost kernel: RSP: 002b:00007ffc2d5e5eb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Nov 29 00:23:19 localhost kernel: RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007f671e161b97
Nov 29 00:23:19 localhost kernel: RDX: 00007ffc2d5e5fd0 RSI: 0000000040086610 RDI: 0000000000000004
Nov 29 00:23:19 localhost kernel: RBP: 000055be35bf1030 R08: 0000000000000000 R09: 0000000000000029
Nov 29 00:23:19 localhost kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000055be35bf1090
Nov 29 00:23:19 localhost kernel: R13: 0000000000000000 R14: 000055be35bf4530 R15: 00007ffc2d5e5fd0
Nov 29 00:23:19 localhost kernel: Code: 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 8b 47 60 85 c0 74 05 f0 ff 4f 60 c3 55 48 c7 c7 e0 d4 a0 
Nov 29 00:23:19 localhost kernel: ---[ end trace 705dec4c1ea2ed2d ]---
Nov 29 00:23:19 localhost locksmithd[1302]: No configured reboot window
Nov 29 00:23:17 localhost iscsid[1311]: iSCSI daemon with pid=1312 started!
Nov 29 00:23:19 localhost extend-filesystems[1219]: resize2fs: Remote I/O error While checking for on-line resizing support
Nov 29 00:23:19 localhost extend-filesystems[1219]: Filesystem at /dev/sda9 is mounted on /; on-line resizing required
Nov 29 00:23:19 localhost extend-filesystems[1219]: old_desc_blocks = 1, new_desc_blocks = 6
Nov 29 00:23:19 localhost locksmithd[1302]: locksmithd starting currentOperation="UPDATE_STATUS_IDLE" strategy="reboot"
Nov 29 00:23:19 localhost systemd[1]: Starting Network Name Resolution...
Nov 29 00:23:19 localhost systemd[1]: Starting Permit User Sessions...
Nov 29 00:23:19 localhost systemd[1]: extend-filesystems.service: Main process exited, code=exited, status=1/FAILURE
Nov 29 00:23:19 localhost systemd[1]: extend-filesystems.service: Failed with result 'exit-code'.
Nov 29 00:23:19 localhost systemd[1]: Failed to start Extend Filesystems.
lines 1183-1260/1494 84%

Just hit this on Alpha 1590.0.0.

arithx commented Nov 29, 2017

Nov 29 00:23:16 localhost systemd[1]: Starting Open-iSCSI...
Nov 29 00:23:16 localhost iscsid[1310]: iSCSI logger with pid=1311 started!
Nov 29 00:23:19 localhost kernel: VFS: brelse: Trying to free free buffer
Nov 29 00:23:19 localhost kernel: ------------[ cut here ]------------
Nov 29 00:23:19 localhost kernel: WARNING: CPU: 0 PID: 1268 at ../../../../../../../../usr/src/linux-4.14-rc8-coreos/fs/buffer.c:1205 __
Nov 29 00:23:19 localhost kernel: Modules linked in: xt_comment xt_owner iptable_mangle sb_edac edac_core kvm_intel mousedev psmouse i2c
Nov 29 00:23:19 localhost kernel: CPU: 0 PID: 1268 Comm: resize2fs Not tainted 4.14.0-rc8-coreos #1
Nov 29 00:23:19 localhost kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 0.0.0 02/06/2015
Nov 29 00:23:19 localhost kernel: task: ffff96ae83ea8000 task.stack: ffffbd2f47bb8000
Nov 29 00:23:19 localhost kernel: RIP: 0010:__brelse+0x23/0x30
Nov 29 00:23:19 localhost kernel: RSP: 0018:ffffbd2f47bbbc30 EFLAGS: 00010286
Nov 29 00:23:19 localhost kernel: RAX: 0000000000000027 RBX: ffff96ae862f92d8 RCX: 0000000000000000
Nov 29 00:23:19 localhost kernel: RDX: 0000000000000000 RSI: ffff96ae8ee0df38 RDI: ffff96ae8ee0df38
Nov 29 00:23:19 localhost kernel: RBP: ffffbd2f47bbbc30 R08: 0000000000000030 R09: 00000000000002dd
Nov 29 00:23:19 localhost kernel: R10: ffffbd2f47bbbc18 R11: 00000000000002dc R12: ffff96ae862df348
Nov 29 00:23:19 localhost kernel: R13: ffff96ae7e63f700 R14: 0000000000000038 R15: ffff96ae862f92d8
Nov 29 00:23:19 localhost kernel: FS:  00007f671eca7740(0000) GS:ffff96ae8ee00000(0000) knlGS:0000000000000000
Nov 29 00:23:19 localhost kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 29 00:23:19 localhost kernel: CR2: 000055606b888170 CR3: 0000001bc1239003 CR4: 00000000001606f0
Nov 29 00:23:19 localhost kernel: Call Trace:
Nov 29 00:23:19 localhost kernel:  jbd2_journal_put_journal_head+0xc2/0x1a1 [jbd2]
Nov 29 00:23:19 localhost kernel:  __jbd2_journal_remove_checkpoint+0x5b/0x170 [jbd2]
Nov 29 00:23:19 localhost kernel:  jbd2_log_do_checkpoint+0x375/0x480 [jbd2]
Nov 29 00:23:19 localhost kernel:  ? wait_woken+0x80/0x80
Nov 29 00:23:19 localhost kernel:  jbd2_journal_flush+0x8b/0x180 [jbd2]
Nov 29 00:23:19 localhost kernel:  ? jbd2_journal_flush+0x8b/0x180 [jbd2]
Nov 29 00:23:19 localhost kernel:  ext4_ioctl+0x1163/0x15e0 [ext4]
Nov 29 00:23:19 localhost kernel:  do_vfs_ioctl+0x9f/0x5e0
Nov 29 00:23:19 localhost kernel:  SyS_ioctl+0x79/0x90
Nov 29 00:23:19 localhost kernel:  do_syscall_64+0x5e/0x1d0
Nov 29 00:23:19 localhost kernel:  entry_SYSCALL64_slow_path+0x25/0x25
Nov 29 00:23:19 localhost kernel: RIP: 0033:0x7f671e161b97
Nov 29 00:23:19 localhost kernel: RSP: 002b:00007ffc2d5e5eb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Nov 29 00:23:19 localhost kernel: RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007f671e161b97
Nov 29 00:23:19 localhost kernel: RDX: 00007ffc2d5e5fd0 RSI: 0000000040086610 RDI: 0000000000000004
Nov 29 00:23:19 localhost kernel: RBP: 000055be35bf1030 R08: 0000000000000000 R09: 0000000000000029
Nov 29 00:23:19 localhost kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000055be35bf1090
Nov 29 00:23:19 localhost kernel: R13: 0000000000000000 R14: 000055be35bf4530 R15: 00007ffc2d5e5fd0
Nov 29 00:23:19 localhost kernel: Code: 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 8b 47 60 85 c0 74 05 f0 ff 4f 60 c3 55 48 c7 c7 e0 d4 a0 
Nov 29 00:23:19 localhost kernel: ---[ end trace 705dec4c1ea2ed2d ]---
Nov 29 00:23:19 localhost locksmithd[1302]: No configured reboot window
Nov 29 00:23:17 localhost iscsid[1311]: iSCSI daemon with pid=1312 started!
Nov 29 00:23:19 localhost extend-filesystems[1219]: resize2fs: Remote I/O error While checking for on-line resizing support
Nov 29 00:23:19 localhost extend-filesystems[1219]: Filesystem at /dev/sda9 is mounted on /; on-line resizing required
Nov 29 00:23:19 localhost extend-filesystems[1219]: old_desc_blocks = 1, new_desc_blocks = 6
Nov 29 00:23:19 localhost locksmithd[1302]: locksmithd starting currentOperation="UPDATE_STATUS_IDLE" strategy="reboot"
Nov 29 00:23:19 localhost systemd[1]: Starting Network Name Resolution...
Nov 29 00:23:19 localhost systemd[1]: Starting Permit User Sessions...
Nov 29 00:23:19 localhost systemd[1]: extend-filesystems.service: Main process exited, code=exited, status=1/FAILURE
Nov 29 00:23:19 localhost systemd[1]: extend-filesystems.service: Failed with result 'exit-code'.
Nov 29 00:23:19 localhost systemd[1]: Failed to start Extend Filesystems.
lines 1183-1260/1494 84%

Just hit this on Alpha 1590.0.0.

@goltermann

This comment has been minimized.

Show comment
Hide comment
@goltermann

goltermann Dec 4, 2017

I had our engineers take a look at this. We've seen occasional issues, and suggest some tweaks to the iSCSI configuration:

In iscsid.conf, set the replacement_timeout to 6000:
node.session.timeo.replacement_timeout = 6000

Redhat and derivatives (CentOS, Oracle Linux, etc.) use dracut in their initramfs and require us to pass this in as a kernel parameter as well, or the initramfs doesn't use the setting for the initial iSCSI connection it establishes (via iscsi-start):
iscsi_param=node.session.timeo.replacement_timeout=6000

We also advise turning off noop timeouts for the root volume only:

node.conn[0].timeo.noop_out_interval = 0
node.conn[0].timeo.noop_out_timeout = 0

On RedHat based distributions iscsi drops a file in:
/var/lib/iscsi/nodes/iqn.2015-02.oracle.boot:uefi/169.254.0.2,3260,1/default
And we drop a modified file in place that has those noop settings in it.

goltermann commented Dec 4, 2017

I had our engineers take a look at this. We've seen occasional issues, and suggest some tweaks to the iSCSI configuration:

In iscsid.conf, set the replacement_timeout to 6000:
node.session.timeo.replacement_timeout = 6000

Redhat and derivatives (CentOS, Oracle Linux, etc.) use dracut in their initramfs and require us to pass this in as a kernel parameter as well, or the initramfs doesn't use the setting for the initial iSCSI connection it establishes (via iscsi-start):
iscsi_param=node.session.timeo.replacement_timeout=6000

We also advise turning off noop timeouts for the root volume only:

node.conn[0].timeo.noop_out_interval = 0
node.conn[0].timeo.noop_out_timeout = 0

On RedHat based distributions iscsi drops a file in:
/var/lib/iscsi/nodes/iqn.2015-02.oracle.boot:uefi/169.254.0.2,3260,1/default
And we drop a modified file in place that has those noop settings in it.

@bgilbert

This comment has been minimized.

Show comment
Hide comment
@bgilbert

bgilbert Dec 5, 2017

Member

We already configure similar settings before starting iscsid. That does leave a window during boot during which the default timeout settings are in use. However, I just built a custom image with a modified iscsistart command that sets those options from the beginning:

ExecStart=/usr/sbin/iscsistart -i iqn.2015-02.oracle.boot:instance -t iqn.2015-02.oracle.boot:uefi -a 169.254.0.2 -g 1 -P node.conn[0].timeo.noop_out_interval=0 -P node.conn[0].timeo.noop_out_timeout=0 -P node.session.timeo.replacement_timeout=86400

I've confirmed that iscsistart is correctly parsing those options, and the problem still occurs. I also tried disabling iscsid and rebooting, and the problem still occurred, so the act of starting iscsid during boot is not the cause.

Member

bgilbert commented Dec 5, 2017

We already configure similar settings before starting iscsid. That does leave a window during boot during which the default timeout settings are in use. However, I just built a custom image with a modified iscsistart command that sets those options from the beginning:

ExecStart=/usr/sbin/iscsistart -i iqn.2015-02.oracle.boot:instance -t iqn.2015-02.oracle.boot:uefi -a 169.254.0.2 -g 1 -P node.conn[0].timeo.noop_out_interval=0 -P node.conn[0].timeo.noop_out_timeout=0 -P node.session.timeo.replacement_timeout=86400

I've confirmed that iscsistart is correctly parsing those options, and the problem still occurs. I also tried disabling iscsid and rebooting, and the problem still occurred, so the act of starting iscsid during boot is not the cause.

@bgilbert

This comment has been minimized.

Show comment
Hide comment
@bgilbert

bgilbert Dec 5, 2017

Member

A network trace shows the target returning ILLEGAL REQUEST / INVALID COMMAND OPERATION CODE in response to a WRITE SAME. The kernel has code to handle this case, but apparently it's not completely functioning; I'm investigating whether torvalds/linux@d5ce4c3 fixes it. Meanwhile, writing 0 to /sys/devices/platform/host2/session1/target2:0:0/2:0:0:1/scsi_disk/2:0:0:1/max_write_same_blocks appears to avoid the problem.

Member

bgilbert commented Dec 5, 2017

A network trace shows the target returning ILLEGAL REQUEST / INVALID COMMAND OPERATION CODE in response to a WRITE SAME. The kernel has code to handle this case, but apparently it's not completely functioning; I'm investigating whether torvalds/linux@d5ce4c3 fixes it. Meanwhile, writing 0 to /sys/devices/platform/host2/session1/target2:0:0/2:0:0:1/scsi_disk/2:0:0:1/max_write_same_blocks appears to avoid the problem.

@bgilbert

This comment has been minimized.

Show comment
Hide comment
@bgilbert

bgilbert Dec 6, 2017

Member

That patch, together with its immediate parent, does appear to fix the problem.

Member

bgilbert commented Dec 6, 2017

That patch, together with its immediate parent, does appear to fix the problem.

@charandas

This comment has been minimized.

Show comment
Hide comment
@charandas

charandas Jan 1, 2018

@bgilbert I am experiencing similar on vSphere. Doing a manual resize2fs works after the fact. I see that the alpha release with your fixes is available here, just wanna confirm whether the above fix is oracle specific or not before trying.

charandas commented Jan 1, 2018

@bgilbert I am experiencing similar on vSphere. Doing a manual resize2fs works after the fact. I see that the alpha release with your fixes is available here, just wanna confirm whether the above fix is oracle specific or not before trying.

@bgilbert

This comment has been minimized.

Show comment
Hide comment
@bgilbert

bgilbert Jan 2, 2018

Member

@charandas The fix is not specific to Oracle. If you still see the problem on 1632.0.0 or later, please let us know.

Member

bgilbert commented Jan 2, 2018

@charandas The fix is not specific to Oracle. If you still see the problem on 1632.0.0 or later, please let us know.

@charandas

This comment has been minimized.

Show comment
Hide comment
@charandas

charandas Jan 3, 2018

@bgilbert I have been running with 1632.0.0 and finding it ok so far. I cannot however use this release since it bumps the docker to 17.11.0-ce which is incompatible with rancher 1.6 that I am using.

I have two questions for you:

  1. Are fixes ever backported in your release cycle?
  2. If answer to 1 is no, can I choose to run with an older docker (the one in current coreos stable) on 1632.0.0?

Thanks.

charandas commented Jan 3, 2018

@bgilbert I have been running with 1632.0.0 and finding it ok so far. I cannot however use this release since it bumps the docker to 17.11.0-ce which is incompatible with rancher 1.6 that I am using.

I have two questions for you:

  1. Are fixes ever backported in your release cycle?
  2. If answer to 1 is no, can I choose to run with an older docker (the one in current coreos stable) on 1632.0.0?

Thanks.

@bgilbert

This comment has been minimized.

Show comment
Hide comment
@bgilbert

bgilbert Jan 3, 2018

Member
  1. Yes, sometimes. In this case I think it's likely that we'll backport.
  2. On 1632, you can also select Docker 1.12.
Member

bgilbert commented Jan 3, 2018

  1. Yes, sometimes. In this case I think it's likely that we'll backport.
  2. On 1632, you can also select Docker 1.12.
@charandas

This comment has been minimized.

Show comment
Hide comment
@charandas

charandas Jan 4, 2018

charandas commented Jan 4, 2018

@charandas

This comment has been minimized.

Show comment
Hide comment
@charandas

charandas Jan 4, 2018

@bgilbert lastly, I want to confirm whether there is not a similar way as documented for Docker 1.12 to select a docker with 17.x range - I am asking because Docker 17.09 works perfectly for me, and for working around the above bug, it does not make complete sense to drop all the way back to Docker 1.12.

Of course, cannot wait for Torcx release upcoming in May 18.

charandas commented Jan 4, 2018

@bgilbert lastly, I want to confirm whether there is not a similar way as documented for Docker 1.12 to select a docker with 17.x range - I am asking because Docker 17.09 works perfectly for me, and for working around the above bug, it does not make complete sense to drop all the way back to Docker 1.12.

Of course, cannot wait for Torcx release upcoming in May 18.

@charandas

This comment has been minimized.

Show comment
Hide comment
@charandas

charandas Jan 4, 2018

Ah, I see the "What about Docker 17.03?" part on the blog post you shared. It would probably apply to my question too - which is revert back to 1.12 until Torcx get released.

In that case, I would go ahead and do that, but would be great as you mentioned to get this fix backported.

charandas commented Jan 4, 2018

Ah, I see the "What about Docker 17.03?" part on the blog post you shared. It would probably apply to my question too - which is revert back to 1.12 until Torcx get released.

In that case, I would go ahead and do that, but would be great as you mentioned to get this fix backported.

@bgilbert

This comment has been minimized.

Show comment
Hide comment
@bgilbert

bgilbert Jan 4, 2018

Member

At present there's not an official way to run versions of Docker other than the two in any given release. However, as it turns out, the EREMOTEIO fix should reach stable within a couple days.

Member

bgilbert commented Jan 4, 2018

At present there's not an official way to run versions of Docker other than the two in any given release. However, as it turns out, the EREMOTEIO fix should reach stable within a couple days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment