[IBM WebSphere Application Server] Add WebSphere Application Server Integration Package with JDBC data stream #2922

kush-elastic · 2022-03-30T12:04:49Z

Enhancement

What does this PR do?

Generated the skeleton of WebSphere Application Server integration package.
Added 1 data stream (JDBC).
Added data collection logic for the data streams.
Added the ingest pipeline for the data streams.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yaml files.
Added dashboards and visualizations.
Added pipeline test cases for the data stream.
Added system test cases for the data stream.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Clone integrations repo.
Install elastic-package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/websphere_application_server directory.
Run the following command to run tests.
elastic-package test

Related issues

Screenshots

elasticmachine · 2022-03-30T12:08:59Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-05-20T04:38:55.651+0000
Duration: 17 min 53 sec

Test stats 🧪

Test	Results
Failed	0
Passed	10
Skipped	0
Total	10

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

…k with current system test

kush-elastic · 2022-04-25T06:38:45Z

/test

elasticmachine · 2022-04-26T14:09:26Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`1/1`)	💚 3.886
Classes	100.0% (`1/1`)	💚 3.886
Methods	100.0% (`7/7`)	💚 11.201
Lines	100.0% (`137/137`)	💚 9.21
Conditionals	100.0% (`0/0`)	💚

yug-rajani

Requested some changes

..._application_server/data_stream/jdbc/_dev/test/pipeline/test-jdbc-metrics.json-expected.json

packages/websphere_application_server/data_stream/jdbc/agent/stream/stream.yml.hbs

packages/websphere_application_server/docs/README.md

packages/websphere_application_server/manifest.yml

yug-rajani

Thanks for making the adjustments, @kush-elastic. LGTM!
Please make the changes suggested in this PR (if applicable) to the other IBM WAS PRs as well.

kush-elastic · 2022-05-11T10:51:48Z

/test

ishleenk17 · 2022-05-18T11:41:56Z

...ages/websphere_application_server/data_stream/jdbc/elasticsearch/ingest_pipeline/default.yml

+  - rename:
+      field: prometheus.metrics.was_connectionpool_jdbcOperation_time_seconds_total
+      target_field: websphere_application_server.jdbc.connection.total.operations_seconds
+      ignore_missing: true


Any specific reason for having the target field as "websphere_application_server.jdbc.connection.total.operations_seconds" ?

If we have it as websphere_application_server.jdbc.was_connectionpool_jdbcOperation_time_seconds_total and follow the same notion for other fields as well, we can reduce the pipeline oto a simple rename processor as:

rename:
field: prometheus.metrics
target_field: websphere_application_server.jdbc

we can also have a looped script to do the renaming,

Please refer to the reply: #2922 (comment)

ishleenk17 · 2022-05-18T11:41:57Z

...ages/websphere_application_server/data_stream/jdbc/elasticsearch/ingest_pipeline/default.yml

+  - rename:
+      field: prometheus.metrics.was_connectionpool_jdbcOperation_time_seconds_total
+      target_field: websphere_application_server.jdbc.connection.total.operations_seconds
+      ignore_missing: true


Any specific reason for having the target field as "websphere_application_server.jdbc.connection.total.operations_seconds" ?

If we have it as websphere_application_server.jdbc.was_connectionpool_jdbcOperation_time_seconds_total and follow the same notion for other fields as well, we can reduce the pipeline oto a simple rename processor as:

rename:
field: prometheus.metrics
target_field: websphere_application_server.jdbc

Elastic has a standard field naming convention: https://www.elastic.co/guide/en/beats/devguide/current/event-conventions.html and we've been following the same standards throughout all the integration. All the fields that we are extracting in this integration also follows the same naming standards. Please refer to the shared doc and let me know if you still think we should change it.
CC: @akshay-saraswat

Thanks for the clarification.

ishleenk17 · 2022-05-18T13:24:38Z

...ages/websphere_application_server/data_stream/jdbc/elasticsearch/ingest_pipeline/default.yml

+      ignore_missing: true
+  - script:
+      description: Drops null/empty values recursively
+      lang: painless


Can we use the preexisting "Drop "Processor here ? To drop the null fields ?

I guess you should refer this for drop processor discussion.

ishleenk17 · 2022-05-18T13:30:27Z

packages/websphere_application_server/_dev/deploy/docker/docker-compose.yml

+version: '3.4'
+services:
+  websphere_application_server:
+    image: ibmcom/websphere-traditional:9.0.5.10


There is a latest Websphere image 9.0.5.11
I think it would be a good idea to use that ?

Sure, I have updated image version. Please check it out.

ishleenk17 · 2022-05-19T10:25:11Z

packages/websphere_application_server/data_stream/jdbc/fields/base-fields.yml

+  description: Data stream namespace.
+- name: '@timestamp'
+  type: date
+  description: Event timestamp.


It is always a good practice to have the event.* fields have their values populated in base-fields.yml.
Please refer to this as example:
https://github.com/elastic/integrations/blob/main/packages/prometheus/data_stream/collector/fields/base-fields.yml

We have followed this from integration developer guide.
if you still think we should change this please let us know.

This would mean that the existing fields in it will never change.
But the newer fields can be added. That's what is followed through all the integrations in Elastic.

Okay, Thanks. will update it soon.

kush-elastic · 2022-05-20T05:42:03Z

Hey @ishleenk17, thanks for the review! We have addressed all the review comments and made the required changes and re-requested review. Could you please take a look at the PR again and let us know if you think we should change anything else before we can have an approval from you over the PR?

ishleenk17 · 2022-05-20T07:20:54Z

Looks good. Approved!

Add IBM WebSphere Application Server integration skeleton

328fe8c

kush-elastic added 2 commits March 31, 2022 11:40

Add data collection, pipeline test and system test to check if CI wor…

b9e285d

…k with current system test

Add code owner entry

9eb83e3

kush-elastic self-assigned this Apr 6, 2022

kush-elastic requested a review from mtojek April 6, 2022 13:25

mtojek requested a review from a team April 6, 2022 14:08

kush-elastic added 3 commits April 7, 2022 18:49

Add datastream jdbc

8cc87a2

Update codeowner entry

089effb

Remove other data stream

df56a85

kush-elastic marked this pull request as ready for review April 8, 2022 05:30

kush-elastic requested a review from a team as a code owner April 8, 2022 05:30

kush-elastic changed the title ~~[IBM WebSphere Application Server] Add WebSphere Application Server Integration Package~~ [IBM WebSphere Application Server] Add WebSphere Application Server Integration Package with JDBC data stream Apr 8, 2022

kush-elastic added 5 commits April 11, 2022 11:32

Add SSL Configuration example

311847b

Update ingest pipeline and dashboard

a88eb2b

Update ingest pipeline

96462cc

Update ingest pipeline

a9f4930

Update pipeline test

b2dd826

masci added the Integration:WebsphereApplicationServer label Apr 13, 2022

kush-elastic mentioned this pull request Apr 13, 2022

[IBM WebSphere Application Server] Add WebSphere Application Server Integration Package with Servlet data stream #3085

Merged

4 tasks

Update codeowner and dashboard

44b6c74

kush-elastic mentioned this pull request Apr 20, 2022

[IBM WebSphere Application Server] Add WebSphere Application Server Integration Package with ThreadPool data stream #3138

Merged

4 tasks

kush-elastic added 4 commits April 21, 2022 15:47

Update documentation

f6bd642

Update readme

4da4710

Update ecs version, readme, ingest pipeline and docker.

b6fce54

Update description in readme and manifest

9c6d4de

kush-elastic requested a review from lalit-satapathy April 25, 2022 06:52

kush-elastic mentioned this pull request Apr 26, 2022

[IBM WebSphere Application Server] Add WebSphere Application Server Integration Package with Session Manager data stream #3199

Merged

4 tasks

kush-elastic added enhancement New feature or request New Integration Team:Service-Integrations Label for the Service Integrations team labels Apr 29, 2022

Update readme and manifest

9f83162

kush-elastic mentioned this pull request May 4, 2022

[IBM WebSphere Application Server] Add WebSphere Application Server Integration Package with HPEL data stream #3269

Closed

4 tasks

Update manifest

99c5ac2

kush-elastic requested review from ishleenk17 and yug-rajani May 9, 2022 05:38

yug-rajani suggested changes May 11, 2022

View reviewed changes

kush-elastic added 2 commits May 11, 2022 14:14

Update ingest pipeline, manifest, readme and dashboard.

2a50a59

Update pipeline test

2f98de9

kush-elastic requested a review from yug-rajani May 11, 2022 08:47

yug-rajani approved these changes May 11, 2022

View reviewed changes

Merge branch 'main' into package_websphere_application_server_jdbc

bc90165

kush-elastic linked an issue May 11, 2022 that may be closed by this pull request

Create IBM Websphere Application Server Integration #2847

Closed

8 tasks

ishleenk17 reviewed May 18, 2022

View reviewed changes

kush-elastic requested a review from ishleenk17 May 18, 2022 13:02

ishleenk17 reviewed May 18, 2022

View reviewed changes

kush-elastic added 2 commits May 19, 2022 00:57

update docker image version 9.0.5.10 -> 9.0.5.11

abe3e3f

Update README.md

8f91ecc

kush-elastic requested a review from ishleenk17 May 19, 2022 04:27

ishleenk17 reviewed May 19, 2022

View reviewed changes

kush-elastic added 2 commits May 19, 2022 19:04

Update ecs mapping.

dae4012

Update readme

dda8ce0

kush-elastic requested a review from ishleenk17 May 20, 2022 04:58

ishleenk17 approved these changes May 20, 2022

View reviewed changes

kush-elastic merged commit ecc2264 into elastic:main May 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[IBM WebSphere Application Server] Add WebSphere Application Server Integration Package with JDBC data stream #2922

[IBM WebSphere Application Server] Add WebSphere Application Server Integration Package with JDBC data stream #2922

kush-elastic commented Mar 30, 2022 •

edited

Loading

elasticmachine commented Mar 30, 2022 •

edited

Loading

Build stats

Test stats 🧪

kush-elastic commented Apr 25, 2022

elasticmachine commented Apr 26, 2022 •

edited

Loading

yug-rajani left a comment

yug-rajani left a comment

kush-elastic commented May 11, 2022

ishleenk17 May 18, 2022

ishleenk17 May 18, 2022

kush-elastic May 18, 2022

ishleenk17 May 18, 2022

kush-elastic May 18, 2022

ishleenk17 May 19, 2022

ishleenk17 May 18, 2022

kush-elastic May 18, 2022

ishleenk17 May 18, 2022

kush-elastic May 19, 2022 •

edited

Loading

ishleenk17 May 19, 2022

kush-elastic May 19, 2022

ishleenk17 May 19, 2022

kush-elastic May 19, 2022

kush-elastic commented May 20, 2022

ishleenk17 commented May 20, 2022

[IBM WebSphere Application Server] Add WebSphere Application Server Integration Package with JDBC data stream #2922

[IBM WebSphere Application Server] Add WebSphere Application Server Integration Package with JDBC data stream #2922

Conversation

kush-elastic commented Mar 30, 2022 • edited Loading

What does this PR do?

Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Mar 30, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

kush-elastic commented Apr 25, 2022

elasticmachine commented Apr 26, 2022 • edited Loading

🌐 Coverage report

yug-rajani left a comment

Choose a reason for hiding this comment

yug-rajani left a comment

Choose a reason for hiding this comment

kush-elastic commented May 11, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kush-elastic May 19, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kush-elastic commented May 20, 2022

ishleenk17 commented May 20, 2022

kush-elastic commented Mar 30, 2022 •

edited

Loading

elasticmachine commented Mar 30, 2022 •

edited

Loading

elasticmachine commented Apr 26, 2022 •

edited

Loading

kush-elastic May 19, 2022 •

edited

Loading