Improve Lucene observability

[nscuro]

Description

Collect basic metrics:

• Total number of index operations (add, update, delete, commit), grouped by index
• Number of index documents in RAM
• Number of bytes used by the index
• Total number of documents in the index

Also, integrate Lucene's InfoStream with Dependency-Track's logging system. Lucene output will now be included when configuring LOGGING_LEVEL=DEBUG, or when the respective logger is explicitly configured in logback.xml.

Addressed Issue

Relates to #3429

Additional Details

Logging of Lucene messages can be enabled on a per-index basis:

<!-- logback.xml -->
<logger name="org.dependencytrack.search.VulnerabilityIndexer.lucene" level="DEBUG" additivity="false">
  <appender-ref ref="STDOUT"/>
</logger>

It is also possible to enable logging selectively for specific Lucene components (e.g. for IndexWriter IW):

<!-- logback.xml -->
<logger name="org.dependencytrack.search.VulnerabilityIndexer.lucene.IW" level="DEBUG" additivity="false">
  <appender-ref ref="STDOUT"/>
</logger>

Newly exposed metrics:

# HELP search_index_docs_total_objects Number of docs in this index, including docs not yet flushed (still in the RAM buffer), and including deletions
# TYPE search_index_docs_total_objects gauge
search_index_docs_total_objects{index="VULNERABILITY",} 249586.0

# HELP search_index_operations_total Total number of index operations
# TYPE search_index_operations_total counter
search_index_operations_total{index="VULNERABILITY",operation="commit",} 7.0
search_index_operations_total{index="VULNERABILITY",operation="delete",} 0.0
search_index_operations_total{index="VULNERABILITY",operation="add",} 0.0
search_index_operations_total{index="VULNERABILITY",operation="update",} 139857.0

# HELP search_index_ram_bytes_used_total_bytes Memory usage of the index in bytes
# TYPE search_index_ram_bytes_used_total_bytes gauge
search_index_ram_bytes_used_total_bytes{index="VULNERABILITY",} 7279344.0

# HELP search_index_docs_ram_total_objects Number of documents currently buffered in RAM
# TYPE search_index_docs_ram_total_objects gauge
search_index_docs_ram_total_objects{index="VULNERABILITY",} 8646.0

Checklist

• I have read and understand the contributing guidelines
• This PR fixes a defect, and I have provided tests to verify that the fix is effective
• This PR implements an enhancement, and I have provided tests to verify that it works as intended
• This PR introduces changes to the database model, and I have added corresponding update logic
• This PR introduces new or alters existing behavior, and I have updated the documentation accordingly

[codacy-production]

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.25% (target: -1.00%)	❌ 57.89% (target: 70.00%)

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (8c9a74a)	21953	16379	74.61%
Head commit (5885940)	21967 (+14)	16445 (+66)	74.86% (+0.25%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#3535)	228	132	57.89%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

_{You may notice some variations in coverage metrics with the latest Coverage engine update. For more details, visit the documentation}