From 7e598bb2b1f157043efef305f7e0565db17c0c4e Mon Sep 17 00:00:00 2001 From: "haixia.cheng@microsoft.com" Date: Thu, 8 Jul 2021 10:39:48 +0800 Subject: [PATCH 1/4] WLS on AKS: apply mountOptions "nobrl" to automation script. --- .../azure-file-pv-template.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/kubernetes/samples/scripts/create-weblogic-domain-on-azure-kubernetes-service/azure-file-pv-template.yaml b/kubernetes/samples/scripts/create-weblogic-domain-on-azure-kubernetes-service/azure-file-pv-template.yaml index b8cbba5fb7e..db1b5dbcf98 100644 --- a/kubernetes/samples/scripts/create-weblogic-domain-on-azure-kubernetes-service/azure-file-pv-template.yaml +++ b/kubernetes/samples/scripts/create-weblogic-domain-on-azure-kubernetes-service/azure-file-pv-template.yaml @@ -24,3 +24,4 @@ spec: - uid=1000 - gid=1000 - mfsymlinks + - nobrl From acc8a48be07fe4fe6a9eaf8b99650e03695711f3 Mon Sep 17 00:00:00 2001 From: "haixia.cheng@microsoft.com" Date: Fri, 9 Jul 2021 12:19:10 +0800 Subject: [PATCH 2/4] Add notes for `nobrl` tag. --- .../includes/create-aks-cluster-storage.txt | 2 ++ 1 file changed, 2 insertions(+) diff --git a/documentation/staging/content/samples/simple/azure-kubernetes-service/includes/create-aks-cluster-storage.txt b/documentation/staging/content/samples/simple/azure-kubernetes-service/includes/create-aks-cluster-storage.txt index a8c96aa7637..105b304cfc9 100644 --- a/documentation/staging/content/samples/simple/azure-kubernetes-service/includes/create-aks-cluster-storage.txt +++ b/documentation/staging/content/samples/simple/azure-kubernetes-service/includes/create-aks-cluster-storage.txt @@ -137,6 +137,8 @@ spec: - nobrl ``` +**Note:** This sample enables tag `nobrl` to prevent sending byte range lock requests to the server. If not using `nobrl`, file locking issues will happen during domain lifecycle management. If you don't want to enable it for specific requirements, you can apply the workarounds of [Handling NFS Locking Errors](https://docs.oracle.com/en/middleware/fusion-middleware/weblogic-server/12.2.1.4/perfm/storetune.html#GUID-A1126893-C60F-4C39-BFCF-0D0F2436478D). + We have provided another configuration file `pvc.yaml` for the `PersistentVolumeClaim`. Both `pv.yaml` and `pvc.yaml` have exactly the same content for `storageClassName` attributes. This is required. We set the same value to the `metadata` property in both files. The following content is an example that uses the persistent volume claim name `wls-azurefile`. ```yaml From 48f29262f1fae94b871b8c5246442512c8bda12b Mon Sep 17 00:00:00 2001 From: Ed Burns Date: Tue, 13 Jul 2021 10:06:43 +0800 Subject: [PATCH 3/4] Reword helpful content from @tbarnes-us. Signed-off-by: Ed Burns Changes to be committed: modified: create-aks-cluster-storage.txt --- .../includes/create-aks-cluster-storage.txt | 47 ++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-) diff --git a/documentation/staging/content/samples/simple/azure-kubernetes-service/includes/create-aks-cluster-storage.txt b/documentation/staging/content/samples/simple/azure-kubernetes-service/includes/create-aks-cluster-storage.txt index 105b304cfc9..626c20e6613 100644 --- a/documentation/staging/content/samples/simple/azure-kubernetes-service/includes/create-aks-cluster-storage.txt +++ b/documentation/staging/content/samples/simple/azure-kubernetes-service/includes/create-aks-cluster-storage.txt @@ -137,7 +137,52 @@ spec: - nobrl ``` -**Note:** This sample enables tag `nobrl` to prevent sending byte range lock requests to the server. If not using `nobrl`, file locking issues will happen during domain lifecycle management. If you don't want to enable it for specific requirements, you can apply the workarounds of [Handling NFS Locking Errors](https://docs.oracle.com/en/middleware/fusion-middleware/weblogic-server/12.2.1.4/perfm/storetune.html#GUID-A1126893-C60F-4C39-BFCF-0D0F2436478D). +**Note:** This sample includes `nobrl` in the `mountOptions` to disable byte range file locking on the `azurefile` storage class. This is necessary as of this writing because the `azurefile` storage class does not support advisory byte range locking. This approach is documented in the [Azure Kubernetes Service FAQ](https://docs.microsoft.com/en-us/azure/aks/troubleshooting#what-are-the-default-mountoptions-when-using-azure-files). + +-------- + +##### Additional important file locking information + +Some action must be taken to deal with the presence or absence of advisory byte range locking when running WebLogic on Kubernetes. Failure to address this issue will cause WebLogic file store locking issues during domain lifecycle management. When this happens, WebLogic will not be able to start. + +If it is not possible to use a different file system that fully supports advisory byte range locking, such as [NFS file shares in Azure Files](https://docs.microsoft.com/en-us/azure/storage/files/files-nfs-protocol#regional-availability), be aware that disabling locking risks data corruption and additional steps are required to mitigate this risk, as shown in the next section. + +Here are several different approaches to disable file locking. + +- When using the `azurefile` storage class, you can universally disable locking on the entire file system by enabling the `nobrl` mount option, as shown above. + + - Note that this affects all software using the same file system. Steps to reduce the consequent corruption risk will vary based on software, and will differ from the steps used for WebLogic. + +- You can disable file locking in the WebLogic configuration for each default file store, custom file store, and JMS paging store by following the steps outlined in [Handling NFS Locking Errors](https://docs.oracle.com/en/middleware/fusion-middleware/weblogic-server/12.2.1.4/perfm/storetune.html#GUID-A1126893-C60F-4C39-BFCF-0D0F2436478D). + + - When using the operator, you can provide this configuration without needing to modify your original configuration using [configuration overrides](https://oracle.github.io/weblogic-kubernetes-operator/userguide/managing-domains/configoverrides/) for Domain on PV or Domain in Image, or [runtime updates](https://oracle.github.io/weblogic-kubernetes-operator/userguide/managing-domains/model-in-image/runtime-updates/) for Model in Image + - Note that this can be a substantial amount of work and error prone as it requires configuration updates for each individual default store, custom file store, and JMS paging store. + +- You can disable all file store locks on a particular WebLogic server JVM by _both_ applying patch `32471832` and setting `-Dweblogic.store.file.LockEnabled=false`. When using the operator, you can set command line values using the `JAVA_OPTIONS` env var in `spec.serverPod.env` domain resource attribute. This will work for any operator domain home source type. + +##### Mitigating corruption risk when locking is disabled + +It is important to mitigate the risk of WebLogic data corruption when locking is disabled. + +- Do not configure [service migration](https://docs.oracle.com/en/middleware/fusion-middleware/weblogic-server/12.2.1.4/clust/service_migration.html#GUID-AEECC92D-88DA-4E07-8C42-A24DBC1C7076) for WebLogic JTA default file stores or custom file stores. + + - File store service migration is not supported when file locking is disabled because it relies on file locks for safe behavior. + + - Service migration is a WebLogic high availability option that is typically configured to enable data recovery on surviving WebLogic servers in a cluster upon an unexpected WebLogic server failure. It is also used to enable JMS and JTA data recovery from WebLogic servers that are shutdown due to a cluster shrink. + +- If at all possible, do not store important data in WebLogic default or custom file stores. + +- Configure WebLogic JMS, JTA, and EJB Timers to use database storage instead of file storage. + + - For example, use the 'TLOG-in-DB' feature for JTA and custom database stores instead of file stores for JMS. + + - This will require a change to your original WebLogic configuration. It is not practical to use operator configuration overrides or runtime updates for this purpose because the changes are too extensive. + +- Note that it is fine to configure service migration for database stores even when file locking is disabled. + +- Take additional steps in your CI/CD processes to guard against the user errors that file locks normally help prevent. In particular, ensure you have procedures in place to prevent administrators or testers from mistakenly starting a duplicate WebLogic domain in the same shared file system. + +-------- We have provided another configuration file `pvc.yaml` for the `PersistentVolumeClaim`. Both `pv.yaml` and `pvc.yaml` have exactly the same content for `storageClassName` attributes. This is required. We set the same value to the `metadata` property in both files. The following content is an example that uses the persistent volume claim name `wls-azurefile`. From 9e2dc7cfc737a75bce5478670b478a0b88e45091 Mon Sep 17 00:00:00 2001 From: Ed Burns Date: Tue, 13 Jul 2021 14:00:17 -0700 Subject: [PATCH 4/4] On branch galiacheng-main - Apply suggestions from @rosemarymarano. modified: documentation/staging/content/samples/simple/azure-kubernetes-service/includes/create-aks-cluster-storage.txt - Apply suggestions from @rosemarymarano. --- .../includes/create-aks-cluster-storage.txt | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/documentation/staging/content/samples/simple/azure-kubernetes-service/includes/create-aks-cluster-storage.txt b/documentation/staging/content/samples/simple/azure-kubernetes-service/includes/create-aks-cluster-storage.txt index 626c20e6613..6aa2f3d350d 100644 --- a/documentation/staging/content/samples/simple/azure-kubernetes-service/includes/create-aks-cluster-storage.txt +++ b/documentation/staging/content/samples/simple/azure-kubernetes-service/includes/create-aks-cluster-storage.txt @@ -137,7 +137,7 @@ spec: - nobrl ``` -**Note:** This sample includes `nobrl` in the `mountOptions` to disable byte range file locking on the `azurefile` storage class. This is necessary as of this writing because the `azurefile` storage class does not support advisory byte range locking. This approach is documented in the [Azure Kubernetes Service FAQ](https://docs.microsoft.com/en-us/azure/aks/troubleshooting#what-are-the-default-mountoptions-when-using-azure-files). +**Note:** This sample includes `nobrl` in the `mountOptions` to disable byte range file locking on the `azurefile` storage class. Currently, this is necessary because the `azurefile` storage class does not support advisory byte range locking. This approach is documented in the [Azure Kubernetes Service FAQ](https://docs.microsoft.com/en-us/azure/aks/troubleshooting#what-are-the-default-mountoptions-when-using-azure-files). -------- @@ -149,7 +149,7 @@ If it is not possible to use a different file system that fully supports advisor Here are several different approaches to disable file locking. -- When using the `azurefile` storage class, you can universally disable locking on the entire file system by enabling the `nobrl` mount option, as shown above. +- When using the `azurefile` storage class, you can universally disable locking on the entire file system by enabling the `nobrl` mount option, as shown previously. - Note that this affects all software using the same file system. Steps to reduce the consequent corruption risk will vary based on software, and will differ from the steps used for WebLogic. @@ -158,7 +158,7 @@ Here are several different approaches to disable file locking. - When using the operator, you can provide this configuration without needing to modify your original configuration using [configuration overrides](https://oracle.github.io/weblogic-kubernetes-operator/userguide/managing-domains/configoverrides/) for Domain on PV or Domain in Image, or [runtime updates](https://oracle.github.io/weblogic-kubernetes-operator/userguide/managing-domains/model-in-image/runtime-updates/) for Model in Image - Note that this can be a substantial amount of work and error prone as it requires configuration updates for each individual default store, custom file store, and JMS paging store. -- You can disable all file store locks on a particular WebLogic server JVM by _both_ applying patch `32471832` and setting `-Dweblogic.store.file.LockEnabled=false`. When using the operator, you can set command line values using the `JAVA_OPTIONS` env var in `spec.serverPod.env` domain resource attribute. This will work for any operator domain home source type. +- You can disable all file store locks on a particular WebLogic server JVM by _both_ applying patch `32471832` and setting `-Dweblogic.store.file.LockEnabled=false`. When using the operator, you can set command-line values using the `JAVA_OPTIONS` env var in `spec.serverPod.env` domain resource attribute. This will work for any operator domain home source type. ##### Mitigating corruption risk when locking is disabled @@ -168,7 +168,7 @@ It is important to mitigate the risk of WebLogic data corruption when locking is - File store service migration is not supported when file locking is disabled because it relies on file locks for safe behavior. - - Service migration is a WebLogic high availability option that is typically configured to enable data recovery on surviving WebLogic servers in a cluster upon an unexpected WebLogic server failure. It is also used to enable JMS and JTA data recovery from WebLogic servers that are shutdown due to a cluster shrink. + - Service migration is a WebLogic high availability option that is typically configured to enable data recovery on surviving WebLogic Servers in a cluster upon an unexpected WebLogic server failure. It is also used to enable JMS and JTA data recovery from WebLogic Servers that are shutdown due to a cluster shrink. - If at all possible, do not store important data in WebLogic default or custom file stores.