amd64: allow parallel shootdown IPIs

Stop using smp_ipi_mtx to protect global shootdown state, and
move/multiply the global state into pcpu.  Now each CPU can initiate
shootdown IPI independently from other CPUs.  Initiator enters
critical section, then fills its local PCPU shootdown info
(pc_smp_tlb_XXX), then clears scoreboard generation at location (cpu,
my_cpuid) for each target cpu.  After that IPI is sent to all targets
which scan for zeroed scoreboard generation words.  Upon finding such
word the shootdown data is read from corresponding cpu' pcpu, and
generation is set.  Meantime initiator loops waiting for all zeroed
generations in scoreboard to update.

Initiator does not disable interrupts, which should allow
non-invalidation IPIs from deadlocking, it only needs to disable
preemption to pin itself to the instance of the pcpu smp_tlb data.

The generation is set before the actual invalidation is performed in
handler. It is safe because target CPU cannot return to userspace
before handler finishes. In principle only NMI can preempt the
handler, but NMI would see the kernel handler frame and not touch
not-invalidated user page table.

Handlers loop until they do not see zeroed scoreboard generations.
This, together with hardware keeping one pending IPI in LAPIC IRR
should prevent lost shootdowns.

Notes.
1. The code does protect writes to LAPIC ICR with exclusion. I believe
   this is fine because we in fact do not send IPIs from interrupt
   handlers. More for !x2APIC mode where ICR access for write requires
   two registers write, we disable interrupts around it. If considered
   incorrect, I can add per-cpu spinlock around ipi_send().
2. Scoreboard lines owned by given target CPU can be padded to the
   cache line, to reduce ping-pong.

Reviewed by:	markj (previous version)
Discussed with:	alc
Tested by:	pho
Sponsored by:	The FreeBSD Foundation
MFC after:	3 weeks
Differential revision:	https://reviews.freebsd.org/D25510

sys/amd64/amd64/apic_vector.S

@@ -171,63 +171,13 @@ IDTVEC(spuriousint)
 	.text
 
 	SUPERALIGN_TEXT
-invltlb_ret:
-	call	as_lapic_eoi
-	jmp	ld_regs
-
-	SUPERALIGN_TEXT
-	INTR_HANDLER invltlb
-	call	invltlb_handler
-	jmp	invltlb_ret
-
-	INTR_HANDLER invltlb_pcid
-	call	invltlb_pcid_handler
-	jmp	invltlb_ret
-
-	INTR_HANDLER invltlb_invpcid_nopti
-	call	invltlb_invpcid_handler
-	jmp	invltlb_ret
-
-	INTR_HANDLER invltlb_invpcid_pti
-	call	invltlb_invpcid_pti_handler
-	jmp	invltlb_ret
-
-/*
- * Single page TLB shootdown
- */
-	INTR_HANDLER invlpg
-	call	invlpg_handler
-	jmp	invltlb_ret
-
-	INTR_HANDLER invlpg_invpcid
-	call	invlpg_invpcid_handler
-	jmp	invltlb_ret
-
-	INTR_HANDLER invlpg_pcid
-	call	invlpg_pcid_handler
-	jmp	invltlb_ret
-
-/*
- * Page range TLB shootdown.
- */
-	INTR_HANDLER invlrng
-	call	invlrng_handler
-	jmp	invltlb_ret
-
-	INTR_HANDLER invlrng_invpcid
-	call	invlrng_invpcid_handler
-	jmp	invltlb_ret
-
-	INTR_HANDLER invlrng_pcid
-	call	invlrng_pcid_handler
-	jmp	invltlb_ret
-
 /*
- * Invalidate cache.
+ * IPI handler for cache and TLB shootdown
  */
-	INTR_HANDLER invlcache
-	call	invlcache_handler
-	jmp	invltlb_ret
+	INTR_HANDLER invlop
+	call	invlop_handler
+	call	as_lapic_eoi
+	jmp	ld_regs
 
 /*
  * Handler for IPIs sent via the per-cpu IPI bitmap.

sys/amd64/amd64/db_interface.c

@@ -107,5 +107,4 @@ db_show_mdpcpu(struct pcpu *pc)
 	db_printf("gs32p        = %p\n", pc->pc_gs32p);
 	db_printf("ldt          = %p\n", pc->pc_ldt);
 	db_printf("tss          = %p\n", pc->pc_tss);
-	db_printf("tlb gen      = %u\n", pc->pc_smp_tlb_done);
 }

sys/amd64/amd64/machdep.c

@@ -1562,6 +1562,7 @@ amd64_bsp_pcpu_init1(struct pcpu *pc)
 	PCPU_SET(ldt, (struct system_segment_descriptor *)&gdt[GUSERLDT_SEL]);
 	PCPU_SET(fs32p, &gdt[GUFS32_SEL]);
 	PCPU_SET(gs32p, &gdt[GUGS32_SEL]);
+	PCPU_SET(smp_tlb_gen, 1);
 }
 
 void