[oracle_weblogic] Add Integration Package with Admin Server Logs Data Stream #3066

kush-elastic · 2022-04-11T18:48:23Z

Enhancement

What does this PR do?

Generated the skeleton of Oracle WebLogic integration package.
Added 1 data stream (Admin Server Logs).
Added data collection logic for the data streams.
Added the ingest pipeline for the data streams.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yaml files.
Added dashboards and visualizations.
Added system test cases for the data stream.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Clone integrations repo.
Install elastic-package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/oracle_weblogic directory.
Run the following command to run tests.
elastic-package test

Related issues

Screenshots

elasticmachine · 2022-04-11T18:52:36Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-05-17T16:39:17.642+0000
Duration: 16 min 49 sec

Test stats 🧪

Test	Results
Failed	0
Passed	8
Skipped	0
Total	8

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

packages/oracle_weblogic/data_stream/admin_server/fields/fields.yml

elasticmachine · 2022-04-29T16:38:53Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`1/1`)	💚 3.589
Classes	100.0% (`1/1`)	💚 3.589
Methods	100.0% (`9/9`)	💚 11.711
Lines	100.0% (`56/56`)	💚 10.985
Conditionals	100.0% (`0/0`)	💚

packages/oracle_weblogic/data_stream/admin_server/elasticsearch/ingest_pipeline/default.yml

agithomas · 2022-04-30T08:31:57Z

packages/oracle_weblogic/data_stream/admin_server/manifest.yml

+        required: true
+        show_user: true
+        default:
+          - /tmp/service_logs/logs/AdminServer.log


In general, i have seen /user_projects/domains/wls_domain/servers/AdminServer.log or similar as the default path fof AdminServer log. You may validate this to correct the default path of AdminServer.log file

You may refer : https://docs.oracle.com/cd/E13222_01/wls/docs91/ConsoleHelp/taskhelp/logging/ChangeServerLogFileNamesAndLocations.html

agithomas

Looks good to me.

mtojek

AFAIR there are two types of logs: server and domain. What are format differences between admin_server, managed_server and domain logs? Can we squeeze them into a single data stream?

mtojek · 2022-05-06T10:36:30Z

packages/oracle_weblogic/_dev/deploy/docker/docker-compose.yml

+  weblogic-log:
+    image: alpine
+    volumes:
+      - ./sample_logs:/sample_logs:ro
+      - ${SERVICE_LOGS_DIR}:/var/log


It isn't the Oracle WebLogic. Please use the right Docker image with the application running.

That makes sense. However, I see in the documentation (Page 8) that we'll need to download supported supplements for Oracle WebLogic which seems to have a Oracle Account signed in as a prerequisite and accepting their terms and policies via GUI (which does not seem to be feasible to achieve here).

First, decide which installation type you want to use, either generic, quick, or slim installer, and then download the required Oracle WebLogic Server installer to the dockerfiles/12.2.1.4 folder.

@mtojek / @kush-elastic, do you have any ideas for the same?

Have you tried playing with any of these: dockerfiles?

Yes, we tried using this. But what is blocking us to use that docker image directly is while running that dockerfile present in repo, we need to download this supplements.
To download that supplements we need oracle login (manually).

It seems that we need access to container-registry.oracle.com to run:

docker pull container-registry.oracle.com/middleware/weblogic:12.2.1.4, right?

@v1v do you think that we can pull/copy them into our Docker registry? Let me justify as this is a longer path. If we want to maintain and develop observability tools for Oracle products, then we need an official/corp account there.

do you think that we can pull/copy them into our Docker registry?

I think so, if you can raise an issue in the obt-robots

Thanks, raised an internal issue.

@kush-elastic Please note here which images would be useful for testing.

Thanks @mtojek, We will check for the exact requirement over the docker image and the required supplements and will get back on this.

Hey @mtojek,

With reference to this discussion #3298

We had a query related to the sample docker files and container scripts used in the docker files, we will be copying and combining these two docker-files (File1, File2) into a single docker file as we want an application to be deployed on managed and admin server. We hope there won't be any consequences in combining these files and using them as one.

We will also be combining all the scripts under one folder from these two folders folder1 and folder2 as it is required for the current use case. we just wanted to confirm this as we can see the Copyright License on each of the files mentioned, hope that it won't cause any issue here as this are the samples files, or is their any other suggestions from your side.

As far as I can see, these files are distributed based on the UPL 1.0 license, which is considered safe according to the Open Source policy in Elastic.

Please add the NOTICE.txt file with a copy of the original license and the link to the docker-images repository.

mtojek · 2022-05-06T10:37:37Z

packages/oracle_weblogic/docs/README.md

+            "diagnostic_context_id": "1648117791865",
+            "machine_name": "wlsadmin",
+            "message_id": "BEA-002622",
+            "message_text": "The protocol \"admin\" is now configured.",


It isn't aligned with ECS, there is a log.message field.

@mtojek Tried to look for the log.message field but could not find it in the ECS mapping can you guide me to the specific ECS reference.

Ah, sorry, my bad. Not log.message but just message.

mtojek · 2022-05-06T10:39:29Z

packages/oracle_weblogic/data_stream/admin_server/elasticsearch/ingest_pipeline/default.yml

+
+    ignore_failure: true
+- script:
+    description: Drops null/empty values recursively


Why is it required? What exactly are you dropping?

This script is used for removing empty fields. So we are using it when we get empty value for any of the fields in the log message.

Why don't you use the drop_fields? https://www.elastic.co/guide/en/beats/metricbeat/current/drop-fields.html

In the drop_fields processor we will explicitly need to provide set of all the possible fields which can possibly have a NULL value, However the above script processor will go through every field in the document and check if any field is NULL or "" and will drop the fields accordingly. Still WDYT should we change this?

I would go with the prebuilt processor as this is less error-prone. Is it common to observe "" fields?

We tried going through drop_fields processor it seems that drop_fields is used to preprocess the document before it actually parse through the ingest pipeline. We also looked into the drop processor which actually drops the entire document and not just the empty fields. So can we prefer this Script instead of any other processor as we can find same script in previously created integrations as well

Ref:

https://github.com/elastic/integrations/blob/main/packages/aws/data_stream/cloudtrail/elasticsearch/ingest_pipeline/default.yml#:~:text=Drops%20null/empty%20values%20recursively

https://github.com/elastic/integrations/blob/main/packages/okta/data_stream/system/elasticsearch/ingest_pipeline/default.yml#:~:text=Drops%20null/empty%20values%20recursively

Ok, thanks for explaining!

@andrewkroh, QQ as you have definitely more experience with pipeline processing. Does it look like a good candidate for a new ES processor?

It's a common enough pattern that I think it could be processor. Often we see data sources that put values like "-" or empty strings into the event and we want to "clean" those from the event. Alternatively we could check every known field and conditionally drop it, but that becomes verbose.

mtojek · 2022-05-06T10:40:14Z

packages/oracle_weblogic/changelog.yml

+# newer versions go on top
+- version: "0.1.0"
+  changes:
+    - description: Oracle WebLogic integration package with Admin server datastream


nit: data stream

yug-rajani · 2022-05-06T11:10:11Z

packages/oracle_weblogic/docs/README.md

+
+### Admin Server logs
+
+The `admin_server` data stream collects Admin Server logs from Adminserver.log.


nit: Adminserver.log

yug-rajani · 2022-05-06T11:15:04Z

packages/oracle_weblogic/manifest.yml

+title: "Oracle WebLogic"
+version: 0.1.0
+license: basic
+description: "This Elastic integration collects logs and metrics from Oracle WebLogic."


Let's keep it consistent with other integrations. Please refer #2614 (comment)

yug-rajani · 2022-05-06T11:15:52Z

packages/oracle_weblogic/manifest.yml

@@ -0,0 +1,31 @@
+format_version: 1.0.0
+name: oracle_weblogic
+title: "Oracle WebLogic"


nit: title: Oracle WebLogic

yug-rajani · 2022-05-06T11:19:18Z

packages/oracle_weblogic/data_stream/admin_server/agent/stream/stream.yml.hbs

@@ -0,0 +1,23 @@
+paths:
+{{#each paths as |path i|}}


Unused variable: i

yug-rajani · 2022-05-06T11:19:27Z

packages/oracle_weblogic/data_stream/admin_server/agent/stream/stream.yml.hbs

+{{#if preserve_original_event}}
+  - preserve_original_event
+{{/if}}
+{{#each tags as |tag i|}}


Unused variable i

yug-rajani

Requested some changes

kush-elastic · 2022-05-06T12:37:27Z

AFAIR there are two types of logs: server and domain. What are format differences between admin_server, managed_server and domain logs? Can we squeeze them into a single data stream?

We discussed with @akshay-saraswat for the same and based on discussion we decided to separate logs to 4 logs data-stream.

kush-elastic · 2022-05-16T06:47:55Z

packages/oracle_weblogic/NOTICE.txt

@@ -0,0 +1,37 @@
+Repository: https://github.com/oracle/docker-images


Hey @mtojek, we have added the licensing details in this file as discussed here by referring to this integrations here.
Can you please take a look and let us know if it's fine and won't cause any issues going further?

Let's move this NOTICE.txt to the _dev/deploy as, as far as I can see, Oracle resources are only used for testing.

When we moved NOTICE.txt to _dev/deploy, we faced a lint error when linting the integration using elastic-package lint.

Error: linting package failed: found 1 validation error: 1. item [NOTICE.txt] is not allowed in folder [/home/devuser/kush/integrations/packages/oracle_weblogic/_dev/deploy]

Should we keep it as it is or do you suggest moving it somewhere else?

Let's move it close to the Dockerfile.

yug-rajani

LGTM, I am not sure on how the NOTICE.txt entry should be added as per #3066 (comment). Please wait for @mtojek's approval over the same.

mtojek · 2022-05-16T09:21:08Z

packages/oracle_weblogic/_dev/deploy/docker/oracle_weblogic/Dockerfile

+
+# Pull base image
+# ---------------
+FROM docker.elastic.co/observability-ci/weblogic:12.2.1.3-dev


Just wanted to clarify: did you copy all files from the original repository (docker-images) or implemented them manually?

Yes, we copied (clubbed file1 and file2 logically) the files form docker images repository and did some minor changes where required.

mtojek · 2022-05-16T09:21:45Z

packages/oracle_weblogic/_dev/deploy/docker/docker-compose.yml

+services:
+  wlsadmin:
+    build:
+      context: .


Why exactly do we need to build this image?

We are actually adding sample application along with Admin Server and Managed Server configuration. Also, we are downloading and configuring Jolokia for data collection. so we would need to build the image.

Ok, I see now.

One more question, are those scripts bound to a specific image version of Oracle? If so, it might be worth mentioning it. Maybe another CONTRIBUTING.md file?

No they aren't we tried to run the system test with same configuration using three docker images (three versions of weblogic)

container-registry.oracle.com/middleware/weblogic:14.1.1.0-dev-8 (14.1.1.0 version)

container-registry.oracle.com/middleware/weblogic:12.2.1.4-dev (12.2.1.4 version)

container-registry.oracle.com/middleware/weblogic:12.2.1.3-dev (12.2.1.3 version)

mtojek · 2022-05-16T09:22:36Z

packages/oracle_weblogic/_dev/deploy/docker/docker-compose.yml

+      context: .
+      args:
+        - APPLICATION_NAME=sample
+        - APPLICATION_PKG=archive.zip


It would be great if you can drop sources somewhere or, even better, use sources and build them in-the-fly.

Just to give you a quick background, we built the archieve.zip by running build.sh locally, we thought it would be a good idea to keep it here (because of its tiny size) rather than building it in-the-fly.

Just in case we run into any troubles building it in-the-fly, can you please elaborate how do we "drop sources somewhere"?

I mean a similar way to what Sunny implemented here: spring_boot. We don't need a full blown WAR application, just a simple and small PoC.

I don't think this is a full-blown WAR application, it's just a single page application which basically gives data related to server and it's pretty small in size.

That's the point, it's easier for developers to the main source than repack WAR, setup environment, maven deps, etc.

mtojek · 2022-05-16T09:24:13Z

packages/oracle_weblogic/NOTICE.txt

@@ -0,0 +1,37 @@
+Repository: https://github.com/oracle/docker-images


Let's move this NOTICE.txt to the _dev/deploy as, as far as I can see, Oracle resources are only used for testing.

kush-elastic · 2022-05-18T09:12:13Z

Thanks for the approval, @mtojek!
@agithomas, can you please take a second look at this PR as the code owner?

agithomas · 2022-05-18T09:18:17Z

Sure @kush-elastic . Checking ..

agithomas

Looks good to me

Add Integration Package with Admin Server Logs Datastream

fbb0aca

kush-elastic requested a review from a team as a code owner April 11, 2022 18:48

kush-elastic requested a review from mtojek April 11, 2022 18:50

Update changelog.yml and Dashboard Screenshot

aa10408

This was referenced Apr 11, 2022

[oracle_weblogic] Add Integration Package with Managed Server Logs Data Stream #3067

Merged

[oracle_weblogic] Add Integration Package with Access Logs Data Stream #3068

Merged

[oracle_weblogic] Add Integration Package with Domain Logs Data Stream #3069

Merged

Update CODEOWNER details

7a93457

masci added the Integration:oracle_weblogic Oracle WebLogic label Apr 13, 2022

kush-elastic mentioned this pull request Apr 19, 2022

[oracle_weblogic] Add Integration Package with Deployed Application Data Stream #3121

Merged

4 tasks

kush-elastic added 2 commits April 23, 2022 15:47

NITS And Change ECS version 8.2

9d33c75

Resolve Pipeline Error

7a94fc3

kush-elastic mentioned this pull request Apr 25, 2022

[oracle_weblogic] Add Integration Package with ThreadPool Data Stream #3189

Merged

4 tasks

agithomas reviewed Apr 28, 2022

View reviewed changes

packages/oracle_weblogic/data_stream/admin_server/fields/fields.yml Outdated Show resolved Hide resolved

kush-elastic mentioned this pull request Apr 29, 2022

[oracle_weblogic] Add Integration Package with Connections Data Stream #3234

Closed

4 tasks

agithomas reviewed Apr 30, 2022

View reviewed changes

packages/oracle_weblogic/data_stream/admin_server/elasticsearch/ingest_pipeline/default.yml Show resolved Hide resolved

agithomas reviewed Apr 30, 2022

View reviewed changes

Address Review Comments

0a1fc3e

kush-elastic added enhancement New feature or request New Integration Team:Service-Integrations Label for the Service Integrations team labels May 3, 2022

kush-elastic requested a review from agithomas May 3, 2022 05:48

kush-elastic mentioned this pull request May 5, 2022

[oracle_weblogic] Add Integration Package with Console Datastream #3281

Closed

4 tasks

agithomas reviewed May 6, 2022

View reviewed changes

Minor Update to Tags Configuration

177c631

mtojek suggested changes May 6, 2022

View reviewed changes

kush-elastic requested a review from yug-rajani May 6, 2022 11:09

yug-rajani reviewed May 6, 2022

View reviewed changes

yug-rajani suggested changes May 6, 2022

View reviewed changes

kush-elastic added 2 commits May 9, 2022 17:14

Address review comments

37dfcbe

Update system test

1fce7ad

kush-elastic requested review from mtojek, yug-rajani and agithomas May 16, 2022 06:35

kush-elastic commented May 16, 2022

View reviewed changes

yug-rajani approved these changes May 16, 2022

View reviewed changes

mtojek reviewed May 16, 2022

View reviewed changes

Update system test

ffb7c77

kush-elastic requested a review from mtojek May 17, 2022 13:21

Update NOTICE.txt path

2c6f0c0

mtojek approved these changes May 18, 2022

View reviewed changes

agithomas approved these changes May 18, 2022

View reviewed changes

kush-elastic merged commit 83f3fb8 into elastic:main May 18, 2022

kush-elastic mentioned this pull request May 18, 2022

[IBM WebSphere Application Server] Add WebSphere Application Server Integration Package with JDBC data stream #2922

Merged

4 tasks

agithomas mentioned this pull request May 24, 2022

[microsoft_sqlserver] Add performance DataStream #3391

Merged

4 tasks

yug-rajani linked an issue Jun 7, 2022 that may be closed by this pull request

Oracle Weblogic Integration #2823

Closed

8 tasks

agithomas mentioned this pull request Jul 5, 2024

[O11y] Migrate infraobs packages to ecs@mappings #10171

Merged

4 tasks


		### Admin Server logs

		The `admin_server` data stream collects Admin Server logs from Adminserver.log.

		@@ -0,0 +1,37 @@
		Repository: https://github.com/oracle/docker-images

[oracle_weblogic] Add Integration Package with Admin Server Logs Data Stream #3066

[oracle_weblogic] Add Integration Package with Admin Server Logs Data Stream #3066

Conversation

kush-elastic commented Apr 11, 2022 • edited Loading

What does this PR do?

Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Apr 11, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Apr 29, 2022 • edited Loading

🌐 Coverage report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agithomas left a comment

Choose a reason for hiding this comment

mtojek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kush-elastic May 10, 2022 • edited Loading

Choose a reason for hiding this comment

mtojek May 10, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtojek May 9, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yug-rajani left a comment

Choose a reason for hiding this comment

kush-elastic commented May 6, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yug-rajani left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kush-elastic May 17, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kush-elastic commented May 18, 2022

agithomas commented May 18, 2022

agithomas left a comment

Choose a reason for hiding this comment

kush-elastic commented Apr 11, 2022 •

edited

Loading

elasticmachine commented Apr 11, 2022 •

edited

Loading

elasticmachine commented Apr 29, 2022 •

edited

Loading

kush-elastic May 10, 2022 •

edited

Loading

mtojek May 10, 2022 •

edited

Loading

mtojek May 9, 2022 •

edited

Loading

yug-rajani left a comment •

edited

Loading

kush-elastic May 17, 2022 •

edited

Loading