Apply resources limits to wls 14.1.1.0 to solve the evicted pod issue. #103

galiacheng · 2021-12-28T03:12:10Z

Description

This pr is a workaround for the evicted issue on WLS 14c. We have discussion on this Slack thread

The WLS14c admin pod become evicted at age 2/3 days with error like Container weblogic-server was using 3300040Ki, which exceeds its request of 1610612736.

The admin pod used memory more than the request studently, see the monitoring screenshot

I am working with Anil to find out the root cause.

Per our observation, applying the resources.limits can resolve the issue, we decided to apply the settings by default.
As the issue does not happen to 12c, so we agree to only apply to 14c, otherwise it will cause the deployment slower.

Test

Test on WLS14c: https://github.com/galiacheng/weblogic-azure/actions/runs/1628854471
Test on WLS 12c:

Signed-off-by: galiacheng <haixia.cheng@microsoft.com> Changes to be committed: new file: weblogic-azure-aks/src/main/arm/scripts/applyGuaranteedQos.sh modified: weblogic-azure-aks/src/main/bicep/mainTemplate.bicep new file: weblogic-azure-aks/src/main/bicep/modules/_deployment-scripts/_ds-apply-guaranteed-qos.bicep Fix script update update update update introspectorJobActiveDeadlineSeconds debug update script remove debug code fix script fix timestamp fix domain uid fix interval create global const for JVM args. Signed-off-by: galiacheng <haixia.cheng@microsoft.com> Changes to be committed: modified: weblogic-azure-aks/src/main/arm/scripts/common.sh modified: weblogic-azure-aks/src/main/arm/scripts/genDomainConfig.sh

Signed-off-by: galiacheng <haixia.cheng@microsoft.com> Changes to be committed: modified: weblogic-azure-aks/pom.xml

tbarnes-us · 2021-12-29T18:21:49Z

weblogic-azure-aks/src/main/arm/scripts/applyGuaranteedQos.sh

+version="${version##*\#version\#\:}" # match #version#:, this is a special mark for the version output, please do not change it.
+echo_stdout ${version}
+
+if [ "${version#*WebLogic Server 14.1.1.0}" != "$version" ]; then


I have not tried to fully understand this script so I could be wrong, but even so, I have two comments:

To future proof, assuming the problem could be in all future 14.x versions, should the version check be a "greater than or equal"? I think you can leverage sort --version-sort for this.

Would the check be better off running before deploying the domain resource? E.g. a kubectl run equivalent of docker run SOME_IMAGE_NAME sh -c 'source $ORACLE_HOME/wlserver/server/bin/setWLSEnv.sh > /dev/null 2>&1 && java weblogic.version'.

Thanks for your review @tbarnes-us Wish you a Merry Christmas though it's a late greeting.

For #1, you are right, we should consider future version, while I would like to only apply the workaround to 14.1.1.0 now, as we can not make sure the issue will happen in the future version, and we may have other configuration to resolve the issue after we find out the root cause. Also cc @edburns

For #2, we can check the version before deploying the domain resource. Considering the usage of post deployment for data source set up, we separate the processes of basic cluster set up and other configuration that configured via configmap which requires a restart. Current work flow is:

Deploy the basic without data source

[Optional if the customer select data source] deploy configmap for data source and patch the cluster, this will cause a restart

Apply resource limits to WLS 14c.

The domain deployment will be much slower by applying resource limits (2min/pod to 8min/pod). So we apply the setting after there is not further restart(rolling update) process.

For getting version using docker run, this is a great idea, without executing the script using kubectl exec.
We have to run the command in a machine with docker deamon, I will check and add another commit.

For #2 I was thinking of kubectl run. This implicitly launches a temporary pod in the k8s cluster, and so has the advantage of using the same docker image cache as the k8s cluster. One issue is that it may need a pull secret - not sure how that'd be done.

For #2:

while I would like to only apply the workaround to 14.1.1.0 now

That's fine by me - as long as one of us remembers in the case that we still need the work-around once 14.1.2 comes out.

For #2:

while I would like to only apply the workaround to 14.1.1.0 now

That's fine by me - as long as one of us remembers in the case that we still need the work-around once 14.1.2 comes out.

We can create an issue to track that, let's discuss with Ed when he is back next week.

For #2 I was thinking of kubectl run. This implicitly launches a temporary pod in the k8s cluster, and so has the advantage of using the same docker image cache as the k8s cluster. One issue is that it may need a pull secret - not sure how that'd be done.

We have the domain running when running the shell script to apply resources limits, I enhanced the script using java weblogic.version, please find 5523347

Signed-off-by: galiacheng <haixia.cheng@microsoft.com>

galiacheng · 2022-01-04T08:17:20Z

Test: https://github.com/galiacheng/weblogic-azure/actions/runs/1642380893

edburns · 2022-01-05T22:05:51Z

weblogic-azure-aks/src/main/arm/scripts/applyGuaranteedQos.sh

+version=${stringArray[2]}
+echo_stdout "WebLogic Server version: ${version}"
+
+if [ "${version#*14.1.1.0}" != "$version" ]; then


Please include this comment:

# Temporary workaround for https://github.com/oracle/weblogic-kubernetes-operator/issues/2693

edburns · 2022-01-05T22:08:48Z

weblogic-azure-aks/src/main/arm/scripts/applyGuaranteedQos.sh

+    resources: 
+      requests:
+        cpu: "${cpu}m"
+        memory: "${memoryRequest}"


As far as I can tell, we are not modifying the memoryRequest from its previous value, retrieved on line 64, though we are modifying the cpu value. Is that what we want? I thought the essence of the workaround was to increase the cpu and memory requests and limits?

Hello @edburns per my observations, 1.5Gi is enough, so will not increase the memory in this workaround.

edburns · 2022-01-05T22:10:06Z

weblogic-azure-aks/src/main/arm/scripts/common.sh

 export constClusterName='cluster-1'
 export constClusterT3AddressEnvName="T3_TUNNELING_CLUSTER_ADDRESS"
 export constDefaultJavaOptions="-Dlog4j2.formatMsgNoLookups=true -Dweblogic.StdoutDebugEnabled=false" # the java options will be applied to the cluster
+export constDefaultJVMArgs="-Djava.security.egd=file:/dev/./urandom -XX:MinRAMPercentage=25.0 -XX:MaxRAMPercentage=50.0 " # the JVM options will be applied to the cluster


Are we sure we want to do this for all wls versions? The other script only applies to 14c, but this change applies unconditionally. Is that what we want?

Yes, you are right, we should specify -Xms256m -Xmx512m for lower version. Thanks for the finding.

edburns · 2022-01-05T22:11:18Z

weblogic-azure-aks/src/main/bicep/mainTemplate.bicep

 }

+/*
+* Apply resource limits to WebLogic Server 14.1.1.0.


Include this in the comment.

* Temporary workaround for https://github.com/oracle/weblogic-kubernetes-operator/issues/2693

edburns

Please review my comments before merging. I assume you will consider them and apply them if necessary.

Signed-off-by: galiacheng <haixia.cheng@microsoft.com>

…56m -Xmx512m. Signed-off-by: galiacheng <haixia.cheng@microsoft.com>

…zure into fix-wls14c-evicted

Signed-off-by: galiacheng <haixia.cheng@microsoft.com> Changes to be committed: modified: weblogic-azure-aks/src/main/arm/scripts/applyGuaranteedQos.sh modified: weblogic-azure-aks/src/main/bicep/mainTemplate.bicep modified: weblogic-azure-aks/src/main/bicep/modules/_deployment-scripts/_ds-apply-guaranteed-qos.bicep

galiacheng · 2022-01-07T07:04:59Z

Hello @edburns @tbarnes-us There is a bug, querying the version from admin pod causes failure, there is no admin pod for slim image, but we can make sure there is at least one managed server pod, as the UI definition requires at least 1 replicas. I fixed it by running the command in one of the managed server pod, see 0082b26.

galiacheng · 2022-01-07T07:05:25Z

Test: https://github.com/galiacheng/weblogic-azure/actions/runs/1666320984

galiacheng added 2 commits December 28, 2021 11:01

increase pom version to 1.0.30

7dad213

Signed-off-by: galiacheng <haixia.cheng@microsoft.com> Changes to be committed: modified: weblogic-azure-aks/pom.xml

galiacheng marked this pull request as draft December 28, 2021 03:18

galiacheng marked this pull request as ready for review December 28, 2021 07:58

tbarnes-us reviewed Dec 29, 2021

View reviewed changes

galiacheng changed the title ~~Apply resources limits to wls 14 to solve the evicted pod issue.~~ Apply resources limits to wls 14.1.1.0 to solve the evicted pod issue. Dec 31, 2021

get version with "java weblogic.version"

5523347

Signed-off-by: galiacheng <haixia.cheng@microsoft.com>

galiacheng force-pushed the fix-wls14c-evicted branch from 6ab60f0 to 5523347 Compare December 31, 2021 08:55

galiacheng added 3 commits December 31, 2021 17:00

typos

d84b00c

Signed-off-by: galiacheng <haixia.cheng@microsoft.com>

use exact version 14.1.1.0 in comment that has 14c.

4f37b4d

Signed-off-by: galiacheng <haixia.cheng@microsoft.com>

fix syntax error near unexpected token `)'

a399318

edburns mentioned this pull request Jan 5, 2022

Using WLS 14c with WLS k8s operator: prone to admin server pod eviction. oracle/weblogic-kubernetes-operator#2693

Closed

edburns reviewed Jan 5, 2022

View reviewed changes

edburns approved these changes Jan 5, 2022

View reviewed changes

galiacheng added 3 commits January 6, 2022 09:12

apply Ed's comments.

aee82e1

Signed-off-by: galiacheng <haixia.cheng@microsoft.com>

fix the default jvm memory args for lower version by specifying -Xms2…

8cd232d

…56m -Xmx512m. Signed-off-by: galiacheng <haixia.cheng@microsoft.com>

Merge branch 'fix-wls14c-evicted' of github.com:galiacheng/weblogic-a…

0b1e7c8

…zure into fix-wls14c-evicted

edburns approved these changes Jan 6, 2022

View reviewed changes

tbarnes-us approved these changes Jan 6, 2022

View reviewed changes

mriccell approved these changes Jan 8, 2022

View reviewed changes

mriccell merged commit 8742285 into oracle:main Jan 8, 2022

galiacheng deleted the fix-wls14c-evicted branch January 24, 2022 02:10

Apply resources limits to wls 14.1.1.0 to solve the evicted pod issue. #103

Apply resources limits to wls 14.1.1.0 to solve the evicted pod issue. #103

Uh oh!

Conversation

galiacheng commented Dec 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test

Uh oh!

Choose a reason for hiding this comment

Uh oh!

galiacheng Dec 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tbarnes-us Dec 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

galiacheng commented Jan 4, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edburns left a comment

Choose a reason for hiding this comment

Uh oh!

galiacheng commented Jan 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

galiacheng commented Jan 7, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

galiacheng commented Dec 28, 2021 •

edited

Loading

galiacheng Dec 30, 2021 •

edited

Loading

tbarnes-us Dec 30, 2021 •

edited

Loading

galiacheng commented Jan 7, 2022 •

edited

Loading