Weblogic Image: t3 to the admin port takes significant time if the Admin Container doesnt map the same port as the docker host

[vipinponnarath]

Hi,

Current setup
I am able to build the 12C(12.2.1) weblogic docker image and then create a sample domain 1221-domain:latest extending the oracle/weblogic :12.2.1-developer image. The resulting image is given below
REPOSITORY TAG IMAGE ID CREATED SIZE
1221-domain latest 595c72b4ce39 7 hours ago 1.29GB

I was also able to do port mappings to a different host port and the console was accessible on the mapped port. I then deployed one of our internal application which sets up some jms and jdbc resources and then deployed the application itself without any issues.
Note that I am only doing everything on just the AdminServer and not using the node manager and managed server domain.

The issue
The problem comes when I try to connect to the t3 protocol from a remote machine when the exposed port inside the container is different from the published port on the docker host. eg:- I spun up a sample container like this -

docker run -it --name WLS -p 8047:7001 -p 8048:8453 -e JAVA_OPTIONS="-Dweblogic.rjvm.enableprotocolswitch=true" myapp:v1
I can successfully login to the console using http://:8047/console
However, when i try to connect through t3 on 8047, it connects only after a significat time (15-20 mins). I initially thought this is something to do with my local test framework which I am using, but using Hermes JMS also returned the same behaviour.

I then spun up another container mapping exactly the same ports as the container exposes.
docker run -it --name SAMEPORT -p 7001:7001 -p 8453:8453 -e JAVA_OPTIONS="-Dweblogic.rjvm.enableprotocolswitch=true" myapp:v1

and then my test framework was able to immediately connect to through the t3 protocol to the AdminServer on 7001. I am not sure what is special about t3 ports vs the http ports. The console access works absolutely fine, but t3 needs the exact same port on the docker host as the container exposes.

my workaround
As a workaround, I am spinning up containers like below
docker run -it --name WLS -p 8047:8047 -p 8048:8453 -e JAVA_OPTIONS="-Dweblogic.rjvm.enableprotocolswitch=true" myapp:v1

and then 'exec' into the container and change the port in config.xml to 8047 and restart the container.

I am unable to pin point if this is a weblogic docker issue or something to do with the docker-proxy or my local set up.

Please help!!

Regards,
Vipin

[vipinponnarath]

Hi, It would be great if someone can look and provide a resolution to this issue.

regards,
Vipin

[brunoborges]

Hey @mriccell, mind to review this one please?

[jralmaraz]

Its a very interesting issue and your workaround matches a known behavior when using OTD to balance t3 connections (not sure if you have seen this article, but here it goes: http://pauldone.blogspot.co.nz/2013/04/load-balancing-t3-initial-context.html) and it might be the way to go.

Just one more interested in it and wanting to follow the update from the product perspective.

[mriccell]

The problem relates to t3 using the default channel to connect to the servers in the container where the internal server port and the external host port are different.

[shean-guangchang]

currently the workaround is to configure a channel and specify the external address and external port and then use the channel to make your t3 connection.

[mriccell]

@vipinponnarath make reference to the documentation https://docs.oracle.com/cd/E72987_01/wls/WLACH/taskhelp/channels/ConfigureCustomNetworkChannelsForNonclusteredServers.html leave the address blank and only configure your port to the host port that you are mapping. Can you please try it and let me know the results?

[vipinponnarath]

@mriccell - thanks for the response, I will try this and post the results here.

[vipinponnarath]

@mriccell I tried to configure a Custom Channel, but no luck here. I am not sure if I am doing anything wrong here. Here is a snapshot of the channel configuration. My admin server is accessible on the docker host port 9001. The container has the default port 7001.

[mriccell]

The Channel External port must match the port you are mapping the internal 7001 port of the WLS server to host. Example in your run -p 8047:7001, the internal port is 7001 the external port is 8047. The listen address is correct it should be blank.

[vipinponnarath]

Hello @mriccell ,

If you mean, I should use 8047 instead of 9001 for the Public port in the above snapshot, i just chose this time to spin up the container to publish the internal port 7001 on the host port 9001, like -p 9001:7001.

Do you also mean that I should configure the Listen Port in the above snapshot to also 7001? Note that this doesnt work due to a port conflict. I have attached a snapshot here.

[vipinponnarath]

@mriccell, do you have some time to check my updates above? I appreciate the help you are providing.

[vipinponnarath]

Hello @mriccell

Please can you check the above update and provide a resolution?

Regards,
Vipin

[shean-guangchang]

Could you post the WebLogic config.xml and the related docker config regarding network setup?

[shean-guangchang]

Beside that we can tell the above "Channel-t3" configuration you have it listening on 7001 and the weblogic server has so called default channel which is the port the server is listening on and by default it is using 7001 also so if you did not customize it then your weblogic server already configured to listening on port 7001 and that is why you are getting the error message. So, send the above info and describe what port you want the weblogic server to listen on and what port you want those ports to map out.

[anpanigr]

Configure a t3 channel with ListenPort set to 8002 ( other than Server Listen Port) and Public Port set to any valid Port e.g. 9002. See the snippet from config.xml file

<server>
    <name>AdminServer</name>
    <listen-port>8001</listen-port>
    <listen-address></listen-address>
    <network-access-point>
      <name>Channel-0</name>
      <protocol>t3</protocol>
      <listen-port>8002</listen-port>
      <public-port>9002</public-port>
      <http-enabled-for-this-protocol>true</http-enabled-for-this-protocol>
      <tunneling-enabled>false</tunneling-enabled>
      <outbound-enabled>false</outbound-enabled>
      <enabled>true</enabled>
      <two-way-ssl-enabled>false</two-way-ssl-enabled>
      <client-certificate-enforced>false</client-certificate-enforced>
    </network-access-point>
  </server>

Use the following Docker run command
docker run --name WLS -p 8001:8001 -p 3098:8002

[mriccell]

@vipinponnarath have you tried this configuration?

[vipinponnarath]

@mriccell @shean-guangchang @anpanigr ,

I tried this configuration and it is quite similar to the workaround that I posted while raising the issue itself. It works for only that specific listen port mapping(-p 3098:8002). The moment I spin up a second container and configure a channel again with another port mapping (say -p 4098:6002) my test suite gets stuck at this t3 connection stage and then it waits and waits and after a really long time (~20mins) it proceeds successfully. Just as reported in my original issue. What is it doing these minutes(just thinking loud)

Just that we all understand the use case here. I have a small docker server that can spin up several containers from the same weblogic image template. All these different weblogic containers, runs on the same docker host and they just maps to different host ports. But with our current situation, it seems I can only spin up one container from one docker host.

Again - note that it is only a problem if I use t3. Otherwise, I can run as many weblogic containers as i want and they all map to different ports on the same host and runs tests without any problem.

the config.xml from both the containers that i used for today's test is attached herewith.
Also the docker container network inspect for one of the container is given here, actually even if I use the default network, its still the same - so not sure if this file is of any help.
Additionally, the docker info results are also attached - if that helps.
Please let me know what additonal config you'd like to check.

regards,
Vipin

Docker.zip

[mriccell]

@vipinponnarath This is probably because you are using the 12.2.1 domain sample image where the domain is created at build time of the image. How are you changing the configuration when you spin a new container (managed server)?
In the 12.2.1.3 domain sample image the domain is externalized to a volume and the domain is created at run time. Any changes such as creating a new port or channel for a managed server will be reflected in the domain configuration for all servers running in the domain.
The default channel does not work when the host port is different from the server port, this is why you need the channel for t3.

[vipinponnarath]

Hello @mriccell - Not sure if you closed this because of not receiving a response from me for the question in your above email. Would you be kind enough to reopen this?
---->How are you changing the configuration when you spin a new container (managed server)?
Ans: I am not using managed server. I am only using AdminServer for my development purpose.

I have the prebuilt 12.2.1 domain image from which i am spinning up 2 containers as below:

Admin container1
docker run -it --name WLS1 -p 8047:7001 myapp:v1

Admin container 2
docker run -it --name WLS2 -p 8048:7001 myapp:v1

myapp:v1 is just extending 1221-domain in the samples folder and deploying my application to the server.

my expectation is that without any additional configuration both ports 8047 and 8048 should act like two different AdminServers. Infact it does that decently because i can access the console of both 8047 and 8048 successfully at the same time. the problem is only t3. As I wrote above, once I configure the t3 channel, to use a port other than server default port, it then only works with that port. If i configure a second t3 channel - the issue is same. So I am back to the problem of one docker host machine not being able to effectively serve two AdminServer containers with different t3 ports.

[vipinponnarath]

@mriccell
I wanted to try the 12.2.1.3 domain sample, but I cant find a way to extend the Dockerfile because of the fact that the domain is created at runtime. If I want to make an app deployment, i need to have a domain as pre-requisite. So I need to manually commit after creating the domain at runtime. This is not very efficient. Could you kindly reopen this issue?

[mriccell]

@vipinponnarath
In 12.2.1.3 sample domain the domain_home, logs, stores, and app deployment is externalized to a volume outside of containers. To deploy the application follow the following steps:

1. every container needs to have a shared application volume say /shared/apps
   2)each of these directories should map to an external volume /hosts/apps
2. Use any of the following ways to deploy the applications
   Administration console
   REST call
   WLST
   weblogic.Deployer

[vipinponnarath]

@mriccell
The question was not really on how to deploy - this is very clear to me. We are already having wlst scripts to deploy our application. My goal was to use Dockerfiles to call the wlst for doing the deployments.
eg:- I want to write a dockerfile as below:(just concept- ignore the syntax)
FROM 12213-domain:latest
RUN ./startWeblogic.sh &&
./waitForAdminServer.sh &&
./deployApp.py &&
./stopWeblogic.sh
CMD ./startWeblogic.sh

Now as you can see the 12213 image is creating domain only at runtime - which means my above approach wont work. We are right now prototyping on a fully automated pipeline where there is no manual interaction (like as you suggested above - a console based deployment). Anyway this is a different topic from the original issue in this thread and I will try to raise this separately in another thread.

Can you please tell me, from your point of view, is it possible to run two AdminServer containers on a single host? If yes, how does the t3 channel behave in such a situation?

[mriccell]

The 12.2.1.3 sample is intended to extend the WebLogic install image and create a domain with a cluster of managed servers. If you want to run only simple server (Admin Server ) topologies, you need to extend the WebLogic install image in https://github.com/oracle/docker-images/tree/master/OracleWebLogic/dockerfiles/12.2.1.3 either developer or generic version. This WebLogic install image can also be pulled from Oracle Container Registry or the Docker Store.
Hope this helps.

[mriccell]

Question about your port configuration when you say you configure a second t3 channel do you mean on the same server? Do you mind sharing your configuration?
Monica

[vipinponnarath]

@mriccell - Can you tell me exactly what configuration you would like to see? the weblogic config.xml? or some specific config to my docker setup?. I have already shared the two containers config xmls in one of the above discussion (Dec 19 update from me). But I can again redo all that and give you new config if you need - but please tell me exactly what you would like to have.

[mriccell]

I would think to run 2 Admin Servers each exposing a different t3 port you would need to write WLST code to create a t3 custom channel in each where you are mapping the internal t3 port of the server to a different host port for each server

Write WLST that would create config example of t3 custom channelo from @anpanigr above

[mriccell]

@vipinponnarath I have read with more care your problem. As I understand it you want to use the same exact application image and just map the Admin server port to different host ports. You are trying to run many Admin servers with the application deployed and have t3 calls to be made to different ports.
You said before you are using -e JAVA_OPTIONS="-Dweblogic.rjvm.enableprotocolswitch=true" are you sure that the servers that start have this JAVA_OPTION enabled? Can you show the container info?

[shean-guangchang]

I have a two server env and one server I use to run the weblogic docker image (12.2.1.3.0) and my command line is
docker run -it --name WLS -p 8047:7001 -e JAVA_OPTIONS="-Dweblogic.rjvm.enableprotocolswitch=true" weblogic-developer

an empty domain will be created as part of container startup...
Oracle WebLogic Server Auto Generated Empty Domain:

  ----> 'weblogic' admin password: xivp7VL0

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

domain_name : [base_domain]
admin_port : [7001]
domain_path : [/u01/oracle/user_projects/domains/base_domain]
production_mode : [prod]
admin password : [xivp7VL0]
admin name : [AdminServer]
admin username : [weblogic]

from the other machine I can make a t3 connection using "t3://machine:8047"
What I used is weblogic.WLST command and as long as you have the weblogic env then you can do
java weblogic.WLST and then it will enter an interactive mode
$ java weblogic.WLST

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

wls:/offline> connect()
Please enter your username :weblogic
weblogic
Please enter your password :
Please enter your server URL [t3://localhost:7001] :t3://adc00qxd:8047
t3://adc00qxd:8047
Connecting to t3://adc00qxd:8047 with userid weblogic ...

The CompatabilityMBeanServer is not initialized properly.
This might happen if the CompatabilityMBeanServer is
disabled via the JMXMBean.

To view the root cause exception use dumpStack()
debug patch for bug 25558644 WorkContextLocalMap for 12.1.3 generated on 2/7/2018
Apr 12, 2018 11:30:17 AM
FINE: Verifying debug logger in 25558644
Successfully connected to Admin Server "AdminServer" that belongs to domain "base_domain".

[shean-guangchang]

forgot to mention it took no time to establish the t3 connection.

[shean-guangchang]

If my example is matching what you did then it seems working for me. I have a different version of wls image (wls ps3 release) so could you try with that release?

[mriccell]

@vipinponnarath are you still experiencing this issue?

[mriccell]

Closing issue due to lack of response

[hythinkpad]

I also encountered the same problem, from the external use of the t3 protocol to access the weblogic application inside the container, but can not access, please let us know

[ankushgarg28794]

@mricell, I have deployed my war file on weblogic server docker container, does it reflect to the existing docker image for which docker container is running.

[barryguda-1]

As a work around I used @vipinponnarath 's solution and it worked seam lessly. Simply use your own custom ports for the exposing: internal admin server and change the same in the config.xml file and restart the container:
docker run -itd --name=weblogic_nig_6057 -p 6057:6057 -e JAVA_OPTIONS="-Dweblogic.rjvm.enableprotocolswitch=true"

[zappee]

I faced a similar issue. On my side to add -Dweblogic.rjvm.enableprotocolswitch=true JVM param to the WebLogic start script solved the issue. I did not create any additional channels. The -D param solved completely this Docker port mapping issue.