[Meta] Create Package for Apache Tomcat Integration

[SubhrataK]

The overall scope of this integration is to cover the below mentioned logs and metrics. This will be phased out in multiple releases.

• Access Logs
• Catalina Logs
• Localhost Logs
• Request Metrics
• Cache Metrics
• Thread Pool Metrics
• Memory Metrics
• Connection Pool Metrics
• Session Metrics

As part of Phase 1 of this integration, the following are covered:

Apache Tomcat

Data stream: cache

• Create data stream related to Apache Tomcat cache metrics
  • [Apache Tomcat] Add integration package with cache data stream #5424

Data stream: access

• Create data stream related to Apache Tomcat access metrics
  • [Apache Tomcat] Add integration package with access data stream #5883

Data stream: catalina

• Create data stream related to Apache Tomcat catalina logs
  • [Apache Tomcat] Add integration package with catalina data stream #5926

Data stream: localhost

• Create data stream related to Apache Tomcat localhost logs
  • [Apache Tomcat] Add integration package with localhost data stream  #5983

Data stream: request

• Create data stream related to Apache Tomcat request metrics
  • [Apache Tomcat] Add integration package with request data stream #6003

Deprecate Tomcat package

• Rename Tomcat integration to Tomcat NetWitness Logs and add deprecation note in README
  • [Tomcat] Deprecate Tomcat package #6091
• Add Add link of new “apache_tomcat” package in README.
  • [Tomcat] Update README.md #6124

Phase 2 data streams:

Data stream: ConnectionPool

• Create data stream related to Apache Tomcat ConnectionPool metrics
  • Issue: [Apache Tomcat] Add ConnectionPool Metrics datastream #6389
  • PR: [Apache Tomcat] Add integration package with connection pool data stream #6596

Data stream: ThreadPool

• Create data stream related to Apache Tomcat ThreadPool metrics
  • Issue: [Apache Tomcat] Add ThreadPool Metrics datastream #6387
  • PR: [O11y][Apache Tomcat] Add integration package with thread pool data stream #6609

Data stream: Memory

• Create data stream related to Apache Tomcat Memory metrics
  • Issue: [Apache Tomcat] Add Memory metrics datastream #6388
  • PR: [Apache Tomcat] Add integration package with memory data stream #6527

Data stream: Session

• Create data stream related to Apache Tomcat Session metrics
  • Issue: [Apache Tomcat] Add Session Metrics datastream #6391
  • PR: [Apache Tomcat] Add integration package with session data stream #6573

In-review PRs

• [O11y][Apache Tomcat] Add TSDB enablement for request and cache data streams and update ecs.yml file #6788

Related Issues

• [O11y][Apache Tomcat] TSDB Enablement #6777
• [O11y][Apache Tomcat] Add details related to Catalina and Localhost logs format in the README #7098
• [O11y][Apache Tomcat] Create Overview dashboard #6750
• [O11y][Apache Tomcat] Update processor description link #6890
• [O11y][Apache Tomcat] Filter out info logs from Catalina logs overview and Localhost logs overview saved search #6934

TSDB Issue

• [Apache Tomcat] System test is failing while TSDB is enabled in package #6099

Enhancement Requests

• [O11y][Apache Tomcat] Support Log4j log format for catalina and localhost data streams #7204
  • Support multiline parser for multiple date formats and support Log4j format for the Catalina and Localhost data streams
• [O11y] Remove forwarded tag for metrics data streams #7635
  • The forwarded tag will enable the publisher_pipeline.disable_host: true, which is the functionality of a Filebeat input setting that only applies to Filebeat-based integrations. The Apache Tomcat metrics data streams are using Metricbeat-based input. Hence need to remove the forwarded tag for metrics data streams.
• [O11y] Missing description for tag for logs data streams #7634
  • In the Filebeat-based integrations, support the description related to the forwarded tag information.
• [Tomcat] Access logs: Missing time to process request field in parsing #7584
  • A user is testing to ingest JIRA access logs via Tomcat integration. However, the log parsing fails because of the wrong log pattern. Need to support the log format that is being used.
• [O11y][Apache Tomcat] Make Access log custom format %A %X %T fields optional #7773
  • If the user hasn't %A %X %T or one of them, need to update the grok pattern by making the fields optional so that user can ingest the data without any error.
• [Apache Tomcat - Access Log] - Fix ingest pipeline tests & Update X-forwared-for IP to correct destination #7622
  • Fix ingest pipeline tests and update X-forwared-for IP field from header_forwarder to client.ip

Troubleshooting docs / issues

• update_by_query API steps to rename field for ingested documents #8365
  • Update By Query API steps that can help rename fields for ingested documents.

[niraj-elastic]

As discussed with @ishleenk17, localhost logs are collected as part of this scope since it helps users to monitor application activity. and It monitors HTTP transactions between the application server and the client.

[ishleenk17]

We will be creating a new package for Apache Tomcat (with version 0.0.1) in Technical Preview.
The older package will be slowly deprecated. We should add "To be deprecated" in the title of the Integration and point to the new integration from the older one.

[niraj-elastic]

As discussed with @ishleenk17, filestream input will be used over TCP/UDP inputs for all the logs data streams for now.

[SubhrataK]

We will be creating a new package for Apache Tomcat (with version 0.0.1) in Technical Preview. The older package will be slowly deprecated. We should add "To be deprecated" in the title of the Integration and point to the new integration from the older one.

The existing package shall be renamed as "Tomcat NetWitness Logs" and the new package shall be named as "Apache Tomcat".
The documentation of the "Tomcat NetWitness Logs" shall be updated to add "To be deprecated" and a link to the new "Apache Tomcat" integration. The new Apache Tomcat integration shall be in Beta.

[lalit-satapathy]

The existing package shall be renamed as "Tomcat NetWitness Logs" and the new package shall be named as "Apache Tomcat".

Package name in manifest file to be all lower case. Instead of renaming the existing package directory, we can change the title and/or the description?

[ishleenk17]

The existing package shall be renamed as "Tomcat NetWitness Logs" and the new package shall be named as "Apache Tomcat".

Package name in manifest file to be all lowercase. Instead of renaming the existing package directory, we can change the title and/or the description?

Clarification:

The package name in the integrations repository will be tomcat for the older one and apache_tomcat for the newer one.
The title (visible to the customer) of the packages will change: The older one will be "Apache Netwitness Logs" and the newer one will be "Apache Tomcat".

@lalit-satapathy : I hope this is what you meant.

[lalit-satapathy]

• Please link the deprecation PR in the meta description.
• Question: I see the documents are separate for each PR. Are the documentation changes of the PRs to be merged and all merge conflicts to be resolved during that process?
• Is this request data stream PR ready to be merged, since it is not reviewed internally once?

[ishleenk17]

@niraj-elastic : I assume that there are no histogram values in either of the metrics datastream since we are using Prometheus input?

[niraj-elastic]

• Please link the deprecation PR in the meta description.
• Question: I see the documents are separate for each PR. Are the documentation changes of the PRs to be merged and all merge conflicts to be resolved during that process?
• Is this request data stream PR ready to be merged, since it is not reviewed internally once?

• Sure, updated the meta with the link of package deprecation PR.
• Yes, once the PR is merged we will resolve the merge conflicts for the next PR.
• Internally Harnish is reviewing the PRs and I see he has just approved the PR.

[niraj-elastic]

@niraj-elastic : I assume that there are no histogram values in either of the metrics datastream since we are using Prometheus input?

Yes, there are no histogram values

[niraj-elastic]

While developing the integration we we found that in Catalina and Localhost logs timezone offset is not coming, hence the timezone is getting considered UTC by default by date processor. If the event is getting generated in a different timezone then the timestamp needs to get calculated accordingly otherwise by default it will consider as UTC and the timestamp will not be calculated properly. Hence we have introduced a new configuration parameter called Timezone Offset in the configuration section from where the user will be able to provide the timezone in which the events are generated so based on that timestamp calculation will be done in pipeline accordingly and events will be displayed with a correct timestamp.

[rameshelastic]

The following will be covered as part of Phase 2.

Thread Pool Metrics
Memory Metrics
Connection Pool Metrics
Session Metrics

[rameshelastic]

@milan-elastic , could you please link the Phase 2 items here? thanks!

[lalit-satapathy]

@ishleenk17, @milan-elastic,

Anything pending on apache tomcat, since it is made GA now.

[milan-elastic]

@ishleenk17, @milan-elastic,

Anything pending on apache tomcat, since it is made GA now.

Yes, Apache tomcat is GA. hence closing this issue!