Kernel panic in ip_get_pmtu caused by NULL pointer

[stevenwilliamson]

> ::status
debugging crash dump vmcore.0 (64-bit) from 0c-c4-7a-a8-78-49
operating system: 5.11 joyent_20170105T023718Z (i86pc)
image uuid: (not set)
panic message: BAD TRAP: type=e (#pf Page fault) rp=ffffff00f5a20670 addr=4 occurred in module "ip" due to a NULL pointer dereference
dump content: kernel pages only

Panic caused by NULL pointer in ip_get_pmtu:

> $C
ffffff00f5a207b0 ip_get_pmtu+0x198(ffffffe0df726dc0)
ffffff00f5a20890 ip_set_destination_v4+0x40c(ffffff00f5a208cc, 60000e0, 60000e0, ffffffe0df726dc0, 0, 13, 0)
ffffff00f5a20950 ip_attr_connect+0xd2(ffffff3258d1a240, ffffffe0df726dc0, ffffff00f5a209e0, ffffff00f5a209d0, ffffff00f5a209c0, 0, ffffff00f5a209e0, 0, 13)
ffffff00f5a20a50 icmp_output_hdrincl+0x21c(ffffff3258d1a240, ffffff33e5439600, ffffff32268d37d0, 2b3b)
ffffff00f5a20af0 rawip_send+0x111(ffffff3258d1a240, ffffff33e5439600, ffffff00f5a20e80, ffffff32268d37d0)
ffffff00f5a20b80 so_sendmsg+0x247(ffffff3227f28090, ffffff00f5a20e80, ffffff00f5a20e20, ffffff32268d37d0)
ffffff00f5a20be0 socket_sendmsg+0x48(ffffff3227f28090, ffffff00f5a20e80, ffffff00f5a20e20, ffffff32268d37d0)
ffffff00f5a20c70 sendit+0x162(7, ffffff00f5a20e80, ffffff00f5a20e20, 8000)
ffffff00f5a20f00 sendmsg+0x15b(7, fffffd7fffdff940, 8000)
ffffff00f5a20f10 sys_syscall+0x1a2()
>

ffffffe0df726dc0::print ip_xmit_attr_t
{
    ixa_flags = 0x80044000
    ixa_free_flags = 0
    ixa_refcnt = 0
    ixa_xmit_hint = 0xe64b1a33
    ixa_pktlen = 0x78
    ixa_zoneid = 0
    ixa_ire = 0
    ixa_ire_generation = 0x1
    ixa_nce = 0
    ixa_dce = 0
    ixa_dce_generation = 0x1
    ixa_src_generation = 0x17f382
    ixa_src_preferences = 0
    ixa_pmtu = 0
    ixa_fragsize = 0x5dc
    ixa_use_min_mtu = '\0'
    ixa_postfragfn = ip_postfrag_loopcheck
    ixa_nexthop_v6 = ::
    ixa_no_loop_zoneid = 0
    ixa_scopeid = 0
    ixa_broadcast_ttl = 0
    ixa_multicast_ttl = 0x1
    ixa_multicast_ifindex = 0
    ixa_multicast_ifaddr = 0.0.0.0
    ixa_raw_cksum_offset = 0
    ixa_ident = 0
    ixa_conn_id = 0xffffff3258d19dc0
    .... snipped for clarity

> ::cpuinfo
 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  0 fffffffffbc33020  1f    0    0  60   no    no t-31   ffffff00f4b2bc40 fsflush
  1 ffffff31cf27db00  1f    1    0  -1   no    no t-1    ffffff00f4171c40 (idle)
  2 ffffff31cf26e500  1f    0    0  -1   no    no t-1    ffffff00f41f1c40 (idle)
  3 ffffff31cf26d000  1f    0    0  -1   no    no t-5    ffffff00f402ac40 (idle)
  4 ffffff31cf267ac0  1f    0    0  -1   no    no t-0    ffffff00f4339c40 (idle)
  5 ffffff31cf3ea080  1f    0    0  -1   no    no t-7    ffffff00f4485c40 (idle)
  6 ffffff31cf3e4a80  1f    0    0  -1   no    no t-17   ffffff00f450bc40 (idle)
  7 fffffffffbc3d920  1b    0    0  59   no    no t-0    ffffff32167d9860 ospfd
  8 ffffff31cf3de040  1f    0    0  -1   no    no t-6    ffffff00f4617c40 (idle)
  9 ffffff31cf3dab00  1f    0    0  -1   no    no t-5    ffffff00f469dc40 (idle)
 10 ffffff31cf3d5500  1f    0    0  -1   no    no t-5    ffffff00f4723c40 (idle)
 11 ffffff31cf3d4000  1f    0    0  -1   no    no t-14   ffffff00f47a9c40 (idle)
 12 ffffff31cf3ceac0  1f    0    0  59   no    no t-0    ffffff3217b99820 ospfd
 13 ffffff31cf52b580  1f    1    0  -1   no    no t-10   ffffff00f48bbc40 (idle)
 14 ffffff31cf52a080  1f    1    0  -1   no    no t-13   ffffff00f4941c40 (idle)
 15 ffffff31cf528a80  1f    0    0  -1   no    no t-1    ffffff00f49c7c40 (idle)


 > ffffff32167d9860::findstack
stack pointer for thread ffffff32167d9860: ffffff00f5a20450
  ffffff00f5a204c0 param_preset()
  ffffff00f5a20550 die+0xdf()
  ffffff00f5a20660 trap+0xe18()
  ffffff00f5a20670 0xfffffffffb8001d6()
  ffffff00f5a207b0 ip_get_pmtu+0x198()
  ffffff00f5a20890 ip_set_destination_v4+0x40c()
  ffffff00f5a20950 ip_attr_connect+0xd2()
  ffffff00f5a20a50 icmp_output_hdrincl+0x21c()
  ffffff00f5a20af0 rawip_send+0x111()
  ffffff00f5a20b80 so_sendmsg+0x247()
  ffffff00f5a20be0 socket_sendmsg+0x48()
  ffffff00f5a20c70 sendit+0x162()
  ffffff00f5a20f00 sendmsg+0x15b()
  ffffff00f5a20f10 sys_syscall+0x1a2()




/*
 * Get the PMTU for the attributes. Handles both IPv4 and IPv6.
 * Assumes that ixa_ire, dce, and nce have already been set up.
 *
 * The caller has set IXAF_PMTU_DISCOVERY if path MTU discovery is desired.
 * We avoid path MTU discovery if it is disabled with ndd.
 * Furtermore, if the path MTU is too small, then we don't set DF for IPv4.
 *
 * NOTE: We also used to turn it off for source routed packets. That
 * is no longer required since the dce is per final destination.
 */
uint_t
ip_get_pmtu(ip_xmit_attr_t *ixa)
{
	ip_stack_t	*ipst = ixa->ixa_ipst;
	dce_t		*dce;
	nce_t		*nce;
	ire_t		*ire;
	uint_t		pmtu;

	ire = ixa->ixa_ire;
	dce = ixa->ixa_dce;
	nce = ixa->ixa_nce;

	/*
	 * If path MTU discovery has been turned off by ndd, then we ignore
	 * any dce_pmtu and for IPv4 we will not set DF.
	 */
	if (!ipst->ips_ip_path_mtu_discovery)
		ixa->ixa_flags &= ~IXAF_PMTU_DISCOVERY;

	pmtu = IP_MAXPACKET;
	/*
	 * Decide whether whether IPv4 sets DF
	 * For IPv6 "no DF" means to use the 1280 mtu
	 */
	if (ixa->ixa_flags & IXAF_PMTU_DISCOVERY) {
		ixa->ixa_flags |= IXAF_PMTU_IPV4_DF;
	} else {
		ixa->ixa_flags &= ~IXAF_PMTU_IPV4_DF;
		if (!(ixa->ixa_flags & IXAF_IS_IPV4))
			pmtu = IPV6_MIN_MTU;
	}

	/* Check if the PMTU is to old before we use it */
	if ((dce->dce_flags & DCEF_PMTU) &&
	    TICK_TO_SEC(ddi_get_lbolt64()) - dce->dce_last_change_time >

^ Looks like were hitting the dce NULL pointer here.
As per the comment at the top of the function ip_get_pmtu expects dce/nce/ire to already be setup, all of them are 0.

ip_set_destination_v4+0x40c(ffffff00f5a208cc, 60000e0, 60000e0, ffffffe0df726dc0, 0, 13)

> ffffff00f5a208cc::print ipaddr_t
10.24.93.100

Side note: looking at the routers logs in question in this example it started to log errors for rtm_write() on 2017-03-28.

2017-03-28T01:29:04+00:00 localhost zebra[11009]: [ID 405691 daemon.error] kernel_rtm_ipv4: 10.24.100.254/32: rtm_write() unexpectedly returned -4 for command RTM_DELETE
2017-03-28T01:29:04+00:00 localhost zebra[11009]: [ID 405691 daemon.error] kernel_rtm_ipv4: 10.24.100.254/32: rtm_write() unexpectedly returned -2 for command RTM_ADD
2017-03-28T01:29:04+00:00 localhost zebra[11009]: [ID 405691 daemon.error] kernel_rtm_ipv4: 10.24.100.254/32: rtm_write() unexpectedly returned -2 for command RTM_ADD

It was happy logging these errors until 2017-03-31T12:58:24 when the panic occured. The router came back up after the panic and has not logged the errors since.

I can't see how dce ends up been NULL from skimming ip_set_destination_v4.

The dump can be uploaded somewhere secure to aid in debugging if needs be. Any pointers appreciated.

[pfmooney]

Hi. We would love to get access to the dump for further debugging. Would a signed upload URL into Manta work for you?

[stevenwilliamson]

@pfmooney yup that should be fine.

[pfmooney]

Open as OS-6049

[stevenwilliamson]

@pfmooney two other seemingly random panics since this one on the same hardware along with some memory errors reported by fmdump -e. Though seemingly not enough to trip fma to log an actual fault.

Looking pretty clear this is bad memory/hardware now so closing.