Add msgsync to doorbell wakeup logic to avoid weak consistency bug

POWER9 added a new sync mode called 'msgsync' that is required
to avoid weak consistency issues when you are using doorbell
(msgsnd) functions.

See POWER ISA Section 5.9.2 for details, excerpt here:
The ordering done by sync (and ptesync) provides
the appearance of "causality" across a sequence of
msgsnd instructions, as in the following example.
"msgsnd->T1" means "msgsnd instruction target-
ting thread T1". "<DHDI 0>" means "occurrence of
Directed Hypervisor Doorbell interrupt caused by
msgsnd executed on T0". On T0, register r1 is
assumed to contain the value 1.
  T0           T1           T2
  std r1,X     <DHDI 0>     <DHDI 1>
  sync         msgsnd->T2   msgsync
  msgsnd->T1                ld r1,X
In this example, T2's load from X must return 1.

The change here adds the msgsync call to the code that executes
any time we handle a doorbell interrupt.  In addition there is a
POWER9 DD2 errata that indicates we also require a lwsync to
ensure consistency.

Change-Id: Ib0f3571926d71efcbffa205093278e2a1d58df85
CQ: SW454611
Reviewed-on: http://rchgit01.rchland.ibm.com/gerrit1/70648
Reviewed-by: Nicholas E. Bofferding <bofferdn@us.ibm.com>
Reviewed-by: Dean Sanner <dsanner@us.ibm.com>
Tested-by: Jenkins Server <pfd-jenkins+hostboot@us.ibm.com>
Tested-by: Jenkins OP Build CI <op-jenkins+hostboot@us.ibm.com>
Tested-by: FSP CI Jenkins <fsp-CI-jenkins+hostboot@us.ibm.com>
Tested-by: Jenkins OP HW <op-hw-jenkins+hostboot@us.ibm.com>
Reviewed-by: William G. Hoffa <wghoffa@us.ibm.com>

src/include/arch/ppc.H

@@ -5,7 +5,7 @@
 /*                                                                        */
 /* OpenPOWER HostBoot Project                                             */
 /*                                                                        */
-/* Contributors Listed Below - COPYRIGHT 2011,2018                        */
+/* Contributors Listed Below - COPYRIGHT 2011,2019                        */
 /* [+] Google Inc.                                                        */
 /* [+] International Business Machines Corp.                              */
 /*                                                                        */
@@ -225,6 +225,20 @@ inline void eieio()
     asm volatile("eieio" ::: "memory");
 }
 
+ALWAYS_INLINE
+inline void msgsync()
+{
+    // See POWER ISA 5.9.2 for details
+    //asm volatile("msgsync" ::: "memory");
+    asm volatile(".long 0x7C0006EC"); // Our GCC doesn't support 'msgsync' yet
+    //  [011111 ///// ///// ///// 11011 10110/]
+
+    // There is a P9 DD2 workaround that a lwsync is also required
+    //  after a msgsync
+    asm volatile("lwsync" ::: "memory");
+}
+
+
 ALWAYS_INLINE
 inline uint64_t getHMER()
 {

src/kernel/syscall.C

@@ -5,7 +5,7 @@
 /*                                                                        */
 /* OpenPOWER HostBoot Project                                             */
 /*                                                                        */
-/* Contributors Listed Below - COPYRIGHT 2010,2018                        */
+/* Contributors Listed Below - COPYRIGHT 2010,2019                        */
 /* [+] International Business Machines Corp.                              */
 /*                                                                        */
 /*                                                                        */
@@ -52,6 +52,12 @@
 extern "C"
 void kernel_execute_hyp_doorbell()
 {
+    // Per POWER ISA Section 5.9.2, to avoid any weak consistency
+    //  issues we must use a msgsync instruction before consuming
+    //  any data set by a different thread following a doorbell
+    //  wakeup.
+    msgsync();
+
     task_t* t = TaskManager::getCurrentTask();
     task_t* l_task_post = nullptr;
     doorbell_clear();