Skip to content
This repository has been archived by the owner on Oct 16, 2020. It is now read-only.

Multiple network interfaces fail to initialize correctly in EC2. #992

Open
mik373 opened this issue Nov 19, 2015 · 22 comments
Open

Multiple network interfaces fail to initialize correctly in EC2. #992

mik373 opened this issue Nov 19, 2015 · 22 comments
Labels
area/usability component/networkd kind/question kind/support team/os

Comments

@mik373
Copy link

mik373 commented Nov 19, 2015 •
edited by crawford
Loading

Scenario:

  • EC2 instance is launched as part of the autoscaling group
  • Auto-scaling group lifecycle hook sends an SNS message which is listened to my a Lambda
  • The lambda attaches a Network interface with a known private IP as eth1
  • When the instance comes up about one out of three times I am not able to ssh into it using the public ip of the default eth0 ENI that was assigned to it by EC2
  • Unattaching the eth1 via EC2 UI and then attaching it back fixes the problem and allows me to ssh.
    The debug log for the case I can't ssh is below:
core@ip-172-20-0-64 ~ $ cat /tmp/net/logs/*
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 4.2.2-coreos-r1 (buildbot@ip-10-204-3-57) (gcc version 4.9.3 (Gentoo Hardened 4.9.3 p1.2, pie-0.6.3) ) #2 SMP Sat Oct 24 19:42:01 UTC 2015
[    0.000000] Command line: BOOT_IMAGE=/coreos/vmlinuz-b console=ttyS0,115200n8 console=tty0 root=LABEL=ROOT mount.usr=PARTUUID=e03dd35c-7c2d-4a47-b3fe-27f15780a57c modprobe.blacklist=xen_fbfront net.ifnames=0 systemd.journald.forward_to_console=yes
[    0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100
[    0.000000] x86/fpu: Supporting XSAVE feature 0x01: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x02: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x04: 'AVX registers'
[    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 0x340 bytes, using 'standard' format.
[    0.000000] x86/fpu: Using 'eager' FPU context switches.
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009e000-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000efffffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000fc000000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000020fffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.4 present.
[    0.000000] DMI: Xen HVM domU, BIOS 4.2.amazon 05/06/2015
[    0.000000] Hypervisor detected: Xen
[    0.000000] Xen version 4.2.
[    0.000000] Xen Platform PCI: I/O protocol version 1
[    0.000000] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs.
[    0.000000] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks.
               You might have to change the root device
               from /dev/hd[a-d] to /dev/xvd[a-d]
               in your root= kernel command line option
[    0.000000] HVMOP_pagetable_dying not supported
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] e820: last_pfn = 0x210000 max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: write-back
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-BFFFF write-combining
[    0.000000]   C0000-FFFFF write-back
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 0000F0000000 mask 3FFFF8000000 uncachable
[    0.000000]   1 base 0000F8000000 mask 3FFFFC000000 uncachable
[    0.000000]   2 disabled
[    0.000000]   3 disabled
[    0.000000]   4 disabled
[    0.000000]   5 disabled
[    0.000000]   6 disabled
[    0.000000]   7 disabled
[    0.000000] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WC  UC- WT
[    0.000000] e820: last_pfn = 0xf0000 max_arch_pfn = 0x400000000
[    0.000000] Base memory trampoline at [ffff880000097000] 97000 size 28672
[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[    0.000000]  [mem 0x00000000-0x000fffff] page 4k
[    0.000000] BRK [0x0366d000, 0x0366dfff] PGTABLE
[    0.000000] BRK [0x0366e000, 0x0366efff] PGTABLE
[    0.000000] BRK [0x0366f000, 0x0366ffff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0x20fe00000-0x20fffffff]
[    0.000000]  [mem 0x20fe00000-0x20fffffff] page 2M
[    0.000000] BRK [0x03670000, 0x03670fff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0x200000000-0x20fdfffff]
[    0.000000]  [mem 0x200000000-0x20fdfffff] page 2M
[    0.000000] init_memory_mapping: [mem 0x1e0000000-0x1ffffffff]
[    0.000000]  [mem 0x1e0000000-0x1ffffffff] page 2M
[    0.000000] BRK [0x03671000, 0x03671fff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0x00100000-0xefffffff]
[    0.000000]  [mem 0x00100000-0x001fffff] page 4k
[    0.000000]  [mem 0x00200000-0xefffffff] page 2M
[    0.000000] init_memory_mapping: [mem 0x100000000-0x1dfffffff]
[    0.000000]  [mem 0x100000000-0x1dfffffff] page 2M
[    0.000000] BRK [0x03672000, 0x03672fff] PGTABLE
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x00000000000EA020 000024 (v02 Xen   )
[    0.000000] ACPI: XSDT 0x00000000FC00F5A0 000054 (v01 Xen    HVM      00000000 HVML 00000000)
[    0.000000] ACPI: FACP 0x00000000FC00F260 0000F4 (v04 Xen    HVM      00000000 HVML 00000000)
[    0.000000] ACPI: DSDT 0x00000000FC0035E0 00BBF6 (v02 Xen    HVM      00000000 INTL 20090123)
[    0.000000] ACPI: FACS 0x00000000FC0035A0 000040
[    0.000000] ACPI: FACS 0x00000000FC0035A0 000040
[    0.000000] ACPI: APIC 0x00000000FC00F360 0000D8 (v02 Xen    HVM      00000000 HVML 00000000)
[    0.000000] ACPI: HPET 0x00000000FC00F4B0 000038 (v01 Xen    HVM      00000000 HVML 00000000)
[    0.000000] ACPI: WAET 0x00000000FC00F4F0 000028 (v01 Xen    HVM      00000000 HVML 00000000)
[    0.000000] ACPI: SSDT 0x00000000FC00F520 000031 (v02 Xen    HVM      00000000 INTL 20090123)
[    0.000000] ACPI: SSDT 0x00000000FC00F560 000031 (v02 Xen    HVM      00000000 INTL 20090123)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at [mem 0x0000000000000000-0x000000020fffffff]
[    0.000000] NODE_DATA(0) allocated [mem 0x20fff4000-0x20fffafff]
[    0.000000]  [ffffea0000000000-ffffea00083fffff] PMD -> [ffff880207600000-ffff88020f5fffff] on node 0
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.000000]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x000000020fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000001000-0x000000000009dfff]
[    0.000000]   node   0: [mem 0x0000000000100000-0x00000000efffffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x000000020fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x000000020fffffff]
[    0.000000] On node 0 totalpages: 2097053
[    0.000000]   DMA zone: 64 pages used for memmap
[    0.000000]   DMA zone: 22 pages reserved
[    0.000000]   DMA zone: 3997 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 15296 pages used for memmap
[    0.000000]   DMA32 zone: 978944 pages, LIFO batch:31
[    0.000000]   Normal zone: 17408 pages used for memmap
[    0.000000]   Normal zone: 1114112 pages, LIFO batch:31
[    0.000000] ACPI: PM-Timer IO Port: 0xb008
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-47
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 low level)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 low level)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 low level)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ5 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] ACPI: IRQ10 used by override.
[    0.000000] ACPI: IRQ11 used by override.
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[    0.000000] smpboot: Allowing 15 CPUs, 13 hotplug CPUs
[    0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff]
[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff]
[    0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff]
[    0.000000] PM: Registered nosave memory: [mem 0xf0000000-0xfbffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfc000000-0xffffffff]
[    0.000000] e820: [mem 0xf0000000-0xfbffffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on Xen HVM
[    0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
[    0.000000] setup_percpu: NR_CPUS:128 nr_cpumask_bits:128 nr_cpu_ids:15 nr_node_ids:1
[    0.000000] PERCPU: Embedded 32 pages/cpu @ffff88020fc00000 s91480 r8192 d31400 u131072
[    0.000000] pcpu-alloc: s91480 r8192 d31400 u131072 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 --
[    0.000000] xen: PV spinlocks enabled
[    0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 2064263
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: rootflags=rw mount.usrflags=ro BOOT_IMAGE=/coreos/vmlinuz-b console=ttyS0,115200n8 console=tty0 root=LABEL=ROOT mount.usr=PARTUUID=e03dd35c-7c2d-4a47-b3fe-27f15780a57c modprobe.blacklist=xen_fbfront net.ifnames=0 systemd.journald.forward_to_console=yes
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Memory: 8149408K/8388212K available (5314K kernel code, 1050K rwdata, 2396K rodata, 27080K init, 940K bss, 238804K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=15, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  Build-time adjustment of leaf fanout to 64.
[    0.000000]  RCU restricting CPUs from NR_CPUS=128 to nr_cpu_ids=15.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=15
[    0.000000] NR_IRQS:8448 nr_irqs:952 16
[    0.000000] xen:events: Using 2-level ABI
[    0.000000] xen:events: Xen HVM callback vector for event delivery is enabled
[    0.000000] Console: colour VGA+ 80x25
[    0.000000] console [tty0] enabled
[    0.000000] Cannot get hvm parameter CONSOLE_EVTCHN (18): -22!
[    0.000000] console [ttyS0] enabled
[    0.000000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 30580167144 ns
[    0.000000] hpet clockevent registered
[    0.000000] tsc: Detected 2400.040 MHz processor
[    0.005000] Calibrating delay loop (skipped), value calculated using timer frequency.. 4800.08 BogoMIPS (lpj=2400040)
[    0.012003] pid_max: default: 32768 minimum: 301
[    0.015008] ACPI: Core revision 20150619
[    0.023563] ACPI: All ACPI Tables successfully acquired
[    0.027019] Security Framework initialized
[    0.030003] SELinux:  Initializing.
[    0.033006] SELinux:  Starting in permissive mode
[    0.033438] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[    0.039716] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
[    0.044450] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes)
[    0.048009] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes)
[    0.053199] Initializing cgroup subsys blkio
[    0.056004] Initializing cgroup subsys memory
[    0.059009] Initializing cgroup subsys devices
[    0.062003] Initializing cgroup subsys freezer
[    0.065004] Initializing cgroup subsys net_cls
[    0.069003] Initializing cgroup subsys perf_event
[    0.072003] Initializing cgroup subsys net_prio
[    0.075053] CPU: Physical Processor ID: 0
[    0.078778] mce: CPU supports 2 MCE banks
[    0.081019] Last level iTLB entries: 4KB 1024, 2MB 1024, 4MB 1024
[    0.085002] Last level dTLB entries: 4KB 1024, 2MB 1024, 4MB 1024, 1GB 4
[    0.089189] Freeing SMP alternatives memory: 20K (ffffffff8357a000 - ffffffff8357f000)
[    0.098561] ftrace: allocating 21487 entries in 84 pages
[    0.120671] x2apic: IRQ remapping doesn't support X2APIC mode
[    0.124003] Switched APIC routing to physical flat.
[    0.129000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=0 pin2=0
[    0.143602] clocksource: xen: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[    0.150007] Xen: using vcpuop timer interface
[    0.150015] installing Xen timer for CPU 0
[    0.152051] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz (fam: 06, model: 3f, stepping: 02)
[    0.156027] cpu 0 spinlock event irq 53
[    0.157027] Performance Events: unsupported p6 CPU model 63 no PMU driver, software events only.
[    0.160520] NMI watchdog: disabled (cpu0): hardware events not enabled
[    0.161006] NMI watchdog: Shutting down hard lockup detector on all cpus
[    0.162076] installing Xen timer for CPU 1
[    0.163060] x86: Booting SMP configuration:
[    0.164006] .... node  #0, CPUs:        #1cpu 1 spinlock event irq 59
[    0.170030] x86: Booted up 1 node, 2 CPUs
[    0.171005] smpboot: Total of 2 processors activated (9600.16 BogoMIPS)
[    0.172173] devtmpfs: initialized
[    0.176115] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[    0.183064] pinctrl core: initialized pinctrl subsystem
[    0.186179] NET: Registered protocol family 16
[    0.192006] cpuidle: using governor ladder
[    0.197005] cpuidle: using governor menu
[    0.198112] ACPI: bus type PCI registered
[    0.199006] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    0.200076] dca service started, version 1.12.1
[    0.201216] PCI: Using configuration type 1 for base access
[    0.206057] ACPI: Added _OSI(Module Device)
[    0.207009] ACPI: Added _OSI(Processor Device)
[    0.208005] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.209004] ACPI: Added _OSI(Processor Aggregator Device)
[    0.211389] xen: --> pirq=16 -> irq=9 (gsi=9)
[    0.214431] ACPI: Interpreter enabled
[    0.215008] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20150619/hwxface-580)
[    0.218004] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20150619/hwxface-580)
[    0.221013] ACPI: (supports S0 S3 S4 S5)
[    0.222004] ACPI: Using IOAPIC for interrupt routing
[    0.223027] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    0.262563] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    0.263011] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI]
[    0.264011] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM
[    0.265051] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
[    0.266814] acpiphp: Slot [0] registered
[    0.267759] acpiphp: Slot [3] registered
[    0.268321] acpiphp: Slot [4] registered
[    0.269371] acpiphp: Slot [5] registered
[    0.270251] acpiphp: Slot [6] registered
[    0.271296] acpiphp: Slot [7] registered
[    0.272248] acpiphp: Slot [8] registered
[    0.273244] acpiphp: Slot [9] registered
[    0.274237] acpiphp: Slot [10] registered
[    0.275250] acpiphp: Slot [11] registered
[    0.276251] acpiphp: Slot [12] registered
[    0.277260] acpiphp: Slot [13] registered
[    0.278290] acpiphp: Slot [14] registered
[    0.279256] acpiphp: Slot [15] registered
[    0.280250] acpiphp: Slot [16] registered
[    0.281264] acpiphp: Slot [17] registered
[    0.282285] acpiphp: Slot [18] registered
[    0.283252] acpiphp: Slot [19] registered
[    0.284245] acpiphp: Slot [20] registered
[    0.285255] acpiphp: Slot [21] registered
[    0.286254] acpiphp: Slot [22] registered
[    0.287252] acpiphp: Slot [23] registered
[    0.288264] acpiphp: Slot [24] registered
[    0.289253] acpiphp: Slot [25] registered
[    0.290251] acpiphp: Slot [26] registered
[    0.291265] acpiphp: Slot [27] registered
[    0.292257] acpiphp: Slot [28] registered
[    0.293253] acpiphp: Slot [29] registered
[    0.294261] acpiphp: Slot [30] registered
[    0.295251] acpiphp: Slot [31] registered
[    0.296263] PCI host bridge to bus 0000:00
[    0.297006] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.298005] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    0.299004] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    0.300005] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    0.301004] pci_bus 0000:00: root bus resource [mem 0xf0000000-0xfbffffff window]
[    0.302121] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000
[    0.303449] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100
[    0.305249] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180
[    0.306246] pci 0000:00:01.1: reg 0x20: [io  0xc100-0xc10f]
[    0.306749] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io  0x01f0-0x01f7]
[    0.307005] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io  0x03f6]
[    0.308004] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io  0x0170-0x0177]
[    0.309004] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io  0x0376]
[    0.310497] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000
[    0.310580] * Found PM-Timer Bug on the chipset. Due to workarounds for a bug,
               * this clock source is slow. Consider trying other clock sources
[    0.312005] pci 0000:00:01.3: quirk: [io  0xb000-0xb03f] claimed by PIIX4 ACPI
[    0.313715] pci 0000:00:02.0: [1013:00b8] type 00 class 0x030000
[    0.314000] pci 0000:00:02.0: reg 0x10: [mem 0xf0000000-0xf1ffffff pref]
[    0.314140] pci 0000:00:02.0: reg 0x14: [mem 0xf3010000-0xf3010fff]
[    0.315668] pci 0000:00:03.0: [8086:10ed] type 00 class 0x020000
[    0.316437] pci 0000:00:03.0: reg 0x10: [mem 0xf3000000-0xf3003fff 64bit pref]
[    0.317059] pci 0000:00:03.0: reg 0x1c: [mem 0xf3004000-0xf3007fff 64bit pref]
[    0.318420] pci 0000:00:04.0: [8086:10ed] type 00 class 0x020000
[    0.319119] pci 0000:00:04.0: reg 0x10: [mem 0xf3008000-0xf300bfff 64bit pref]
[    0.319884] pci 0000:00:04.0: reg 0x1c: [mem 0xf300c000-0xf300ffff 64bit pref]
[    0.321799] pci 0000:00:1f.0: [5853:0001] type 00 class 0xff8000
[    0.322082] pci 0000:00:1f.0: reg 0x10: [io  0xc000-0xc0ff]
[    0.322266] pci 0000:00:1f.0: reg 0x14: [mem 0xf2000000-0xf2ffffff pref]
[    0.323696] ACPI: PCI Interrupt Link [LNKA] (IRQs *5 10 11)
[    0.326165] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
[    0.329170] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
[    0.332249] ACPI: PCI Interrupt Link [LNKD] (IRQs *5 10 11)
[    0.348689] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.349058] xen:balloon: Initialising balloon driver
[    0.352024] xen_balloon: Initialising balloon driver
[    0.353050] init_memory_mapping: [mem 0x210000000-0x217ffffff]
[    0.353121] vgaarb: setting as boot device: PCI:0000:00:02.0
[    0.353122] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
[    0.353124] vgaarb: loaded
[    0.353124] vgaarb: bridge control possible 0000:00:02.0
[    0.353172] PCI: Using ACPI for IRQ routing
[    0.353173] PCI: pci_cache_line_size set to 64 bytes
[    0.353831] e820: reserve RAM buffer [mem 0x0009e000-0x0009ffff]
[    0.354168] HPET: 3 timers in total, 0 timers will be used for per-cpu timer
[    0.354189] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[    0.354190] hpet0: 3 comparators, 64-bit 62.500000 MHz counter
[    0.385006]  [mem 0x210000000-0x217ffffff] page 2M
[    0.385979]  [ffffea0008400000-ffffea00085fffff] PMD -> [ffff880205e00000-ffff880205ffffff] on node 0
[    0.386000] clocksource: Switched to clocksource xen
[    0.393933] pnp: PnP ACPI init
[    0.396344] system 00:00: [mem 0x00000000-0x0009ffff] could not be reserved
[    0.400460] system 00:00: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.400543] system 00:01: [io  0x08a0-0x08a3] has been reserved
[    0.404217] system 00:01: [io  0x0cc0-0x0ccf] has been reserved
[    0.409132] system 00:01: [io  0x04d0-0x04d1] has been reserved
[    0.412814] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.412840] xen: --> pirq=17 -> irq=8 (gsi=8)
[    0.412858] pnp 00:02: Plug and Play ACPI device, IDs PNP0b00 (active)
[    0.412877] xen: --> pirq=18 -> irq=12 (gsi=12)
[    0.412890] pnp 00:03: Plug and Play ACPI device, IDs PNP0f13 (active)
[    0.412904] xen: --> pirq=19 -> irq=1 (gsi=1)
[    0.412916] pnp 00:04: Plug and Play ACPI device, IDs PNP0303 PNP030b (active)
[    0.412930] xen: --> pirq=20 -> irq=6 (gsi=6)
[    0.412931] pnp 00:05: [dma 2]
[    0.412944] pnp 00:05: Plug and Play ACPI device, IDs PNP0700 (active)
[    0.412964] xen: --> pirq=21 -> irq=4 (gsi=4)
[    0.412975] pnp 00:06: Plug and Play ACPI device, IDs PNP0501 (active)
[    0.413011] system 00:07: [io  0x10c0-0x1141] has been reserved
[    0.416729] system 00:07: [io  0xb044-0xb047] has been reserved
[    0.420391] system 00:07: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.434571] pnp: PnP ACPI: found 8 devices
[    0.444997] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    0.451559] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
[    0.451561] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
[    0.451562] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[    0.451564] pci_bus 0000:00: resource 7 [mem 0xf0000000-0xfbffffff window]
[    0.451599] NET: Registered protocol family 2
[    0.454855] TCP established hash table entries: 65536 (order: 7, 524288 bytes)
[    0.459842] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.464007] TCP: Hash tables configured (established 65536 bind 65536)
[    0.467884] UDP hash table entries: 4096 (order: 5, 131072 bytes)
[    0.471557] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes)
[    0.475561] NET: Registered protocol family 1
[    0.478492] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[    0.482174] pci 0000:00:01.0: PIIX3: Enabling Passive Release
[    0.485842] pci 0000:00:01.0: Activating ISA DMA hang workarounds
[    0.489899] pci 0000:00:02.0: Video device with shadowed ROM
[    0.490153] PCI: CLS 0 bytes, default 64
[    0.863068] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    0.866986] software IO TLB [mem 0xec000000-0xf0000000] (64MB) mapped at [ffff8800ec000000-ffff8800efffffff]
[    0.873499] RAPL PMU detected, API unit is 2^-32 Joules, 3 fixed counters 655360 ms ovfl timer
[    0.879278] hw unit of domain pp0-core 2^-14 Joules
[    0.882554] hw unit of domain package 2^-14 Joules
[    0.885899] hw unit of domain dram 2^-16 Joules
[    0.889359] futex hash table entries: 4096 (order: 6, 262144 bytes)
[    0.893230] Initialise system trusted keyring
[    0.896248] audit: initializing netlink subsys (disabled)
[    0.899694] audit: type=2000 audit(1447957006.383:1): initialized
[    0.903779] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.909030] VFS: Disk quotas dquot_6.6.0
[    0.911851] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.916087] SELinux:  Registering netfilter hooks
[    0.916934] Key type asymmetric registered
[    0.919882] Asymmetric key parser 'x509' registered
[    0.924079] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[    0.931017] io scheduler noop registered
[    0.934504] io scheduler deadline registered
[    0.937384] io scheduler cfq registered (default)
[    0.940632] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.944092] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[    0.948091] intel_idle: does not run on family 6 model 63
[    0.948265] GHES: HEST is not enabled!
[    0.950979] ioatdma: Intel(R) QuickData Technology Driver 4.00
[    0.954872] xen: --> pirq=22 -> irq=47 (gsi=47)
[    0.954953] xen:grant_table: Grant tables using version 1 layout
[    0.958718] Grant table initialized
[    0.961376] Cannot get hvm parameter CONSOLE_EVTCHN (18): -22!
[    0.965069] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.995270] 00:06: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[    1.001171] i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
[    1.008189] serio: i8042 KBD port at 0x60,0x64 irq 1
[    1.011399] serio: i8042 AUX port at 0x60,0x64 irq 12
[    1.014861] rtc_cmos 00:02: rtc core: registered rtc_cmos as rtc0
[    1.018746] rtc_cmos 00:02: alarms up to one day, 114 bytes nvram, hpet irqs
[    1.024737] NET: Registered protocol family 10
[    1.029011] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0
[    1.029184] NET: Registered protocol family 17
[    1.029447] Loading compiled-in X.509 certificates
[    1.030485] Loaded X.509 cert 'Build time autogenerated kernel key: cb4c49a7d0f54acc380d8e47053c385ae2bab225'
[    1.030495] registered taskstats version 1
[    1.030963] xenbus_probe_frontend: Device with no driver: device/vfb/0
[    1.030963] xenbus_probe_frontend: Device with no driver: device/vbd/51712
[    1.030964] xenbus_probe_frontend: Device with no driver: device/vbd/51744
[    1.030965] xenbus_probe_frontend: Device with no driver: device/pci/0
[    1.035791] rtc_cmos 00:02: setting system clock to 2015-11-19 18:16:46 UTC (1447957006)
[    1.078926] Freeing unused kernel memory: 27080K (ffffffff81b08000 - ffffffff8357a000)
[    1.084173] Write protecting the kernel read-only data: 10240k
[    1.090235] Freeing unused kernel memory: 820K (ffff880001533000 - ffff880001600000)
[    1.097499] Freeing unused kernel memory: 1700K (ffff880001857000 - ffff880001a00000)
[    1.108742] ip_tables: (C) 2000-2006 Netfilter Core Team
[    1.114319] random: systemd urandom read with 33 bits of entropy available
[    1.119156] systemd[1]: systemd 225 running in system mode. (-PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT -GNUTLS -ACL +XZ -LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD -IDN)
[    1.132209] systemd[1]: Detected virtualization xen.
[    1.136747] systemd[1]: Detected architecture x86-64.
[    1.140379] systemd[1]: Running in initial RAM disk.
[    1.147712] systemd[1]: No hostname configured.
[    1.150745] systemd[1]: Set hostname to <localhost>.
[    1.153962] systemd[1]: Initializing machine ID from random generator.
[    1.193752] systemd[1]: Reached target Local File Systems.
[    1.200888] systemd[1]: Created slice -.slice.
[    1.206943] systemd[1]: Listening on udev Kernel Socket.
[    1.214962] systemd[1]: Created slice System Slice.
[    1.221032] systemd[1]: Reached target Slices.
[    1.226924] systemd[1]: Listening on Journal Socket (/dev/log).
[    1.235906] systemd[1]: Created slice system-systemd\x2dfsck.slice.
[    1.242950] systemd[1]: Listening on Journal Audit Socket.
[    1.249393] systemd[1]: Listening on udev Control Socket.
[    1.269473] systemd[1]: Reached target Swap.
[    1.275251] systemd[1]: Reached target Timers.
[    1.281678] systemd[1]: Listening on Journal Socket.
[    1.288447] systemd[1]: Starting Apply Kernel Variables...
[    1.295460] systemd[1]: Starting dracut cmdline hook...
[    1.303170] systemd[1]: Reached target Sockets.
[    1.312321] systemd[1]: Starting Journal Service...
[    1.322510] systemd[1]: Starting Create list of required static device nodes for the current kernel...
[    1.337006] systemd[1]: Reached target Encrypted Volumes.
[    1.354500] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[    1.365420] systemd[1]: Reached target Paths.
[    1.373449] systemd[1]: Started Journal Service.
[    1.379526] audit: type=1130 audit(1447957006.848:2): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-journald comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    1.394140] audit: type=1130 audit(1447957006.863:3): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-sysctl comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    1.409637] audit: type=1130 audit(1447957006.878:4): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-cmdline comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    1.423918] audit: type=1130 audit(1447957006.892:5): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=kmod-static-nodes comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    1.468149] audit: type=1130 audit(1447957006.937:6): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-tmpfiles-setup-dev comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    1.491207] audit: type=1130 audit(1447957006.960:7): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-udev comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    1.516801] audit: type=1130 audit(1447957006.985:8): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-udevd comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    1.575323] audit: type=1130 audit(1447957007.044:9): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-trigger comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    1.645109] audit: type=1130 audit(1447957007.114:10): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-udev-trigger comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    1.647239] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2
[    1.647243] ACPI: Power Button [PWRF]
[    1.647301] input: Sleep Button as /devices/LNXSYSTM:00/LNXSLPBN:00/input/input3
[    1.647303] ACPI: Sleep Button [SLPF]
[    1.765172] SCSI subsystem initialized
[    1.770975] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.12.1-k
[    1.779036] ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
[    1.793483] piix4_smbus 0000:00:01.3: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr
[    1.803488] microcode: CPU0 sig=0x306f2, pf=0x1, revision=0x25
[    1.807784] microcode: CPU1 sig=0x306f2, pf=0x1, revision=0x25
[    1.828958] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input4
[    1.836749] microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
[    1.838847] ixgbevf 0000:00:03.0: 02:13:80:52:46:6f
[    1.838848] ixgbevf 0000:00:03.0: MAC: 1
[    1.838849] ixgbevf 0000:00:03.0: Intel(R) 82599 Virtual Function
[    1.853652] ixgbevf 0000:00:04.0: 02:a9:44:00:d4:09
[    1.853654] ixgbevf 0000:00:04.0: MAC: 1
[    1.853655] ixgbevf 0000:00:04.0: Intel(R) 82599 Virtual Function
[    1.864864] libata version 3.00 loaded.
[    1.873651] ata_piix 0000:00:01.1: version 2.13
[    1.875814] tsc: Refined TSC clocksource calibration: 2400.000 MHz
[    1.880383] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns
[    1.891816] AVX2 version of gcm_enc/dec engaged.
[    1.895925] AES CTR mode by8 optimization enabled
[    1.900062] scsi host0: ata_piix
[    1.903592] scsi host1: ata_piix
[    1.906845] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc100 irq 14
[    1.912152] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc108 irq 15
[    1.919908] mousedev: PS/2 mouse device common for all mice
[    1.936803] blkfront: xvda: barrier or flush: disabled; persistent grants: enabled; indirect descriptors: enabled;
[    1.951941]  xvda: xvda1 xvda2 xvda3 xvda4 xvda6 xvda7 xvda9
[    1.957950] blkfront: xvdc: barrier or flush: disabled; persistent grants: enabled; indirect descriptors: enabled;
[    2.153703] random: nonblocking pool is initialized
[    2.618977] EXT4-fs (xvda9): mounted filesystem with ordered data mode. Opts: (null)
[    2.699972] EXT4-fs (xvda4): mounted filesystem without journal. Opts: (null)
[    3.387548] systemd-journald[103]: Received SIGTERM from PID 1 (n/a).
[    4.224851] SELinux: 4096 avtab hash slots, 13308 rules.
[    4.226318] SELinux: 4096 avtab hash slots, 13308 rules.
[    4.226676] SELinux:  6 users, 6 roles, 1323 types, 54 bools, 1 sens, 1024 cats
[    4.226679] SELinux:  92 classes, 13308 rules
[    4.227200] SELinux:  Completing initialization.
[    4.227201] SELinux:  Setting up existing superblocks.
[    4.236018] systemd[1]: Successfully loaded SELinux policy in 103.670ms.
[    4.260328] systemd[1]: Relabelled /dev and /run in 5.530ms.
[    6.376720] audit_printk_skb: 159 callbacks suppressed
[    6.381119] audit: type=1130 audit(1447957011.845:64): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=systemd-udevd comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    6.557208] audit: type=1130 audit(1447957012.026:65): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=systemd-journald comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    6.612811] audit: type=1107 audit(1447957012.081:66): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='Unknown class service exe="/usr/lib64/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'
[    6.647038] systemd-journald[466]: Received request to flush runtime journal from PID 1
[    7.043755] audit: type=1130 audit(1447957012.512:67): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=systemd-journal-flush comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    9.971159] audit: type=1130 audit(1447957015.440:68): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=systemd-udev-settle comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    9.985235] EXT4-fs (xvda6): mounted filesystem with ordered data mode. Opts: commit=600
[   10.157361] audit: type=1130 audit(1447957015.626:69): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=lvm2-activation-early comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   10.157378] audit: type=1131 audit(1447957015.626:70): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=lvm2-activation-early comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   10.245528] audit: type=1130 audit(1447957015.714:71): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=lvm2-activation comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   10.258212] audit: type=1131 audit(1447957015.727:72): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=lvm2-activation comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   10.338047] audit: type=1130 audit(1447957015.806:73): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=coreos-tmpfiles comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   12.244338] ixgbevf 0000:00:04.0: NIC Link is Up 10 Gbps
[   12.245533] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[   12.259149] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   12.259655] ixgbevf 0000:00:03.0: NIC Link is Up 10 Gbps
[   12.267285] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   12.267342] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:13:80:52:46:6f brd ff:ff:ff:ff:ff:ff
    inet 172.20.0.64/24 brd 172.20.0.255 scope global dynamic eth0
       valid_lft 3600sec preferred_lft 3600sec
    inet6 fe80::13:80ff:fe52:466f/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:a9:44:00:d4:09 brd ff:ff:ff:ff:ff:ff
    inet 172.20.0.202/24 brd 172.20.0.255 scope global dynamic eth1
       valid_lft 3600sec preferred_lft 3600sec
    inet6 fe80::a9:44ff:fe00:d409/64 scope link
       valid_lft forever preferred_lft forever
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 02:13:80:52:46:6f brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 02:a9:44:00:d4:09 brd ff:ff:ff:ff:ff:ff
default via 172.20.0.1 dev eth1  proto dhcp  src 172.20.0.202  metric 1024
default via 172.20.0.1 dev eth0  proto dhcp  src 172.20.0.64  metric 1024
172.20.0.0/24 dev eth1  proto kernel  scope link  src 172.20.0.202
172.20.0.0/24 dev eth0  proto kernel  scope link  src 172.20.0.64
172.20.0.1 dev eth1  proto dhcp  scope link  src 172.20.0.202  metric 1024
172.20.0.1 dev eth0  proto dhcp  scope link  src 172.20.0.64  metric 1024
0:  from all lookup local
32766:  from all lookup main
32767:  from all lookup default
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         172.20.0.1      0.0.0.0         UG        0 0          0 eth1
0.0.0.0         172.20.0.1      0.0.0.0         UG        0 0          0 eth0
172.20.0.0      0.0.0.0         255.255.255.0   U         0 0          0 eth1
172.20.0.0      0.0.0.0         255.255.255.0   U         0 0          0 eth0
172.20.0.1      0.0.0.0         255.255.255.255 UH        0 0          0 eth1
172.20.0.1      0.0.0.0         255.255.255.255 UH        0 0          0 eth0
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         ip-172-20-0-1.u 0.0.0.0         UG    1024   0        0 eth1
default         ip-172-20-0-1.u 0.0.0.0         UG    1024   0        0 eth0
ip-172-20-0-0.u *               255.255.255.0   U     0      0        0 eth1
ip-172-20-0-0.u *               255.255.255.0   U     0      0        0 eth0
ip-172-20-0-1.u *               255.255.255.255 UH    1024   0        0 eth1
ip-172-20-0-1.u *               255.255.255.255 UH    1024   0        0 eth0

It seems like a race condition in CoreOS determining and initializing eth0 and eth1 that causes the routing to be broken.
@mik373
Copy link
Author

mik373 commented Nov 21, 2015

Update:

We managed to get ssh to the public IP of default eth0 working consistently by setting eth0's default gateway with a unit:

    - name: restart_network.service
      command: start
      content: |
        [Unit]
        Description=Set the Gateways
        After=network-online.target
        Wants=network-online.target
        Before=docker.service
        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/usr/bin/route del default eth1
        ExecStart=/usr/bin/route add default gw 172.20.0.1 eth0

The only issue we have right now is that the datadog agent 2 out of 3 times can't talk out to the datadog service:

2015-11-20 23:54:01 UTC | ERROR | dd.forwarder | forwarder(ddagent.py:267) | Response: HTTPResponse(_body=None,buffer=None,code=599,effective_url='https://5-5-1-app.agent.datadoghq.com/intake/?api_key=d9c988c950eb837f5583e676509734a9',error=HTTPError('HTTP 599: Timeout',),headers={},reason='Unknown',request=<tornado.httpclient.HTTPRequest object at 0x7f579c699fd0>,request_time=20.00114893913269,time_i

The datadog is started by this unit:

- name: datadog-agent.service
      command: start
      content: |
        [Unit]
        Description=Datadog
        After=docker.service
        Requires=docker.service

        [Service]
        Restart=always
        EnvironmentFile=/etc/etcd-environment
        ExecStartPre=-/usr/bin/docker kill dd-agent
        ExecStartPre=-/usr/bin/docker rm dd-agent
        ExecStartPre=/usr/bin/sleep 30
        ExecStart=/usr/bin/docker run -h %H --name dd-agent \
            --add-host=etcd:$${ETCD_LOCAL_HOST} \
            -v /var/run/docker.sock:/var/run/docker.sock \
            -v /sys/fs/cgroup:/host/sys/fs/cgroup:ro \
            -v /proc/:/host/proc/:ro \
            -e API_KEY=key \
            registry:${dd_agent_version}
        ExecStop=/usr/bin/docker stop dd-agent
        [X-Fleet]
        MachineMetadata=role=worker
        Global=True

Interestingly enough every time the container get the IP of

core@ip-172-20-0-11 ~ $ docker exec -it dd-agent ip route
default via 172.17.42.1 dev eth0

the agent can't talk out to the datadog service but if the IP issues is 172.17.42.2, the service is reachable.

@crawford
Copy link
Contributor

The most appropriate method for configuring those interfaces is to provide your own .network configs. The "Match" section can be used to selectively apply configs to the various interfaces.

@crawford
Copy link
Contributor

crawford commented Dec 1, 2015

@mik373 Were you able to get this working with the networkd configs?

@crawford crawford closed this as completed Dec 2, 2015
@mik373
Copy link
Author

mik373 commented Dec 3, 2015

I can't use the static IPs configs for two reasons:

  1. The ip is not known at the configuration time since the ENI is dynamically assigned by a lambda to the new instance
  2. Even if I knew the static IP and assigned it to this node, how would I propagate that knowledge into my cluster?

@crawford
Copy link
Contributor

crawford commented Dec 3, 2015

You should be able to define .network configs for each interface which enables DHCP. For the public interface gateway, use a lower routing metric to ensure egress packets deterministically use that interface.

@crawford crawford reopened this Dec 3, 2015
@mik373
Copy link
Author

mik373 commented Dec 3, 2015

So my etcd cluster with my config works about 80% of the time. The other 20% the interfaces are initialized in the order that creates asymmetrical ip routs and the cluster members can't dial each other.
It seems that my issue might have to do with eth0 and eth1 coming from the same subnet which confuses the routes. I am trying to use a different subnet now for the launched instances but ssh times out when that's the case. Anything special I have to do on CoreOS level for ssh to work? The ingress rules are configured correctly.

@crawford
Copy link
Contributor

crawford commented Feb 9, 2016

@mik373 Sorry, I just noticed there was an open question from you. No, nothing special is needed on CoreOS for SSH to work. Are you still having trouble with this?

@vaijab
Copy link

vaijab commented Apr 29, 2016

I am having same or very similar issue.

My setup is fairly similar, I have a bunch of instances with a single network interface to start with and then there is a daemon which attaches an additional ENI (eth1).

I found that systemd-networkd fails to bring up eth1 properly. I believe I am hitting this issue: systemd/systemd#1784

So I have the following hack to make sure that eth1 comes up:

[Unit]
Description=Brings up eth1 when networkd fails to bring it up                                                                  
[Service]                                                                                                                      
ExecStart=/usr/bin/bash -c 'while true; do ip -o -4 link show | grep -q "eth1:.*state DOWN" && ip link set up dev eth1; sleep 60; done'

The other problem that I just noticed is that if I reboot an instance which has two ENIs (eth0 and eth1) then the instance comes up with no network working, apart from eth1 because of the above hack.

This is quite a serious problem, because it prevents us from using CoreOS with more than one network interface on EC2.

@marcovnyc
Copy link

I don't know if this can help anyone but I have instances on AWS with two interfaces. I was having the same problem when the eth1 would become active and then the computer rebooted I would lose network connectivity. The second interface adds another route and it messes with your eth0 setup , I added this to my /etc/systemd/network

[Match]
Name=eth1

[Network]
DHCP=ipv4

[DHCP]
UseDNS=false
SendHostname=true
UseRoutes=false
RouteMetric=2000

I believe if you use static ips and use a higher Route Metric that can help you also to not lose connectivity .

@crawford
Copy link
Contributor

crawford commented May 2, 2016

@vaijab can you give this another shot with the latest Alpha? That ships with a much newer version of systemd. @marcovnyc's suggestion to set the route metric is also interesting and might help out. I haven't had a chance to look into this yet.

@vaijab
Copy link

vaijab commented May 6, 2016

Thanks @crawford. This is what I have in my user-data to make it work:

    # This is a dirty workaround hack until this has been fixed: https://github.com/systemd/systemd/issues/1784
    - name: networkd-restart.service
      command: start
      enable: true
      content: |
        [Unit]
        Description=Restart systemd-networkd when DOWN interface is found
        [Service]
        ExecStart=/usr/bin/bash -c 'while true; do ip -o -4 link show | grep -q "eth[0-1]:.*state DOWN" && systemctl restart systemd-networkd; sleep 60; done'
        Restart=always
        RestartSec=10
    - name: 20-eth1.network
      runtime: false
      content: |
        [Match]
        Name=eth1
        [Network]
        DHCP=ipv4
        [DHCP]
        UseDNS=false
        SendHostname=true
        UseRoutes=false
        RouteMetric=2048

@crawford
Copy link
Contributor

Is this issue still present with systemd 231?

@crawford
Copy link
Contributor

Closing due to inactivity.

@joshuabaird
Copy link

This is still an issue in 1911.4.0 as far as I can tell.

@bgilbert bgilbert reopened this Dec 13, 2018
@idleyoungman idleyoungman mentioned this issue Feb 15, 2019
Closed
@idleyoungman
Copy link

idleyoungman commented Feb 15, 2019 •
edited
Loading

We've found this is an issue when using CoreOS (1911.3.0 at time of writing) with https://github.com/aws/amazon-vpc-cni-k8s/ in EC2.

When enough pods are scheduled onto an instance, additional interfaces/ENIs are created. Pod IPs are drawn from a pool of secondary IPs attached to each interface as an implementation detail of the Amazon VPC CNI. These new interfaces learn default routes via DHCP with a metric of 1024. After a reboot, the order of the default routes is undetermined and the node is then unreachable via the eth0 IP address if a non-eth0 default is "first" in the kernel's route table (ip route show | grep default or similar to check).

We are working around currently by lowering the metric for the eth0 default route w/ an /etc/systemd/network/10-eth0-default-pref.network systemd-networkd unit file like:

[Match]
Name=eth0

[Network]
DHCP=ipv4

[DHCP]
RouteMetric=512

@yuwata
Copy link

yuwata commented Feb 22, 2019

Is systemd-udevd running in the system?
(I've asked the same question in systemd/systemd#1784. Sorry for multi-posting.)

@bgilbert
Copy link
Contributor

@yuwata systemd-udevd does run on Container Linux.

@yuwata
Copy link

yuwata commented Feb 25, 2019 •
edited
Loading

@bgilbert Thanks. I'd like to ask one more thing: please provide the results of systemd-detect-virt and systemd-detect-virt --container.

@yuwata yuwata mentioned this issue Feb 25, 2019
Closed
@yuwata
Copy link

yuwata commented Feb 26, 2019

BTW, if you think this is a bug in networkd or udevd, then please open a new issue in systemd and provide debugging logs of the daemons: booting with systemd.log_level=debug udev.log_priority=debug and journalctl -b -u systemd-networkd.service -u systemd-udevd.service --no-hostname may be sufficient. Thank you.

@yuwata
Copy link

yuwata commented Mar 6, 2019 •
edited
Loading

Not sure, but systemd/systemd#11881 may fix this issue.

@sethp-nr sethp-nr mentioned this issue Mar 13, 2019
Closed
@MMichael-S
Copy link

Hi Experts
I am using coreos CoreOS-stable-2135.5.0-hvm (ami-049ed451bb483d4be) and found this issue still exists. Is there a corresponding solution and bug fix plan?

@MMichael-S MMichael-S mentioned this issue Jul 18, 2019
Closed
@bartelsb
Copy link

bartelsb commented Oct 3, 2019

Seems to still be a problem in CoreOS-stable-2191.5.0-hvm (ami-038cea5071a5ee580).

@jayanthvn jayanthvn mentioned this issue Sep 24, 2021
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area/usability component/networkd kind/question kind/support team/os
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@idleyoungman @bgilbert @crawford @vaijab @bartelsb @marcovnyc @mik373 @joshuabaird @yuwata @MMichael-S