Complete output format documentation (OBSDOCS-3627) - #116522
Closed
johnwilkins wants to merge 258 commits into
Closed
Complete output format documentation (OBSDOCS-3627)#116522johnwilkins wants to merge 258 commits into
johnwilkins wants to merge 258 commits into
Conversation
…-logx-about migrate logx about content, with modules and snippets
…-first-pass standalone logging docs - first pass at base 6.0 docs
…-61-initial standalone logging docs 6.1 initial migration
…-62-initial standalone logging docs - initial 6.2 commit
…ions build all the versions of standalone logging for testing purposes
…ogging-devcontainer parameterize package script and update devcontainer to use it
…ogging-devcontainer-fix update devcontainer with yq, fix errors
…ogging-devcontainer-fix-again add version dropdown for logging
…ogging-devcontainer-fix-more add sudo for yq and try to fix version drop down
…ogging-distro add main branch so that cached erb file for package command will be main
try local pageloader js file for baseurl
…main use main page loader
…-remove-temp-fix remove temp fix for local page-loader.js
…-fixes-package-script fixes for package script to run in github action
…-fixes-package-script-new fixes for package script to run in github action
OBSDOCS-3510: Add channel guidance
OBSDOCS-3110: Document HTTP output format parameter
OBSDOCS-2747: Fix syslog receiver verification to use logger instead of curl
…rding Add comprehensive documentation for forwarding logs to Google Cloud Logging using Workload Identity Federation (WIF) authentication, which provides improved security compared to service account keys. New modules: - cluster-logging-collector-log-forward-gcp-wif.adoc: Complete WIF procedure including ccoctl usage, CredentialsRequest creation, and manual token path fixes required for logging collector Updated modules: - cluster-logging-collector-log-forward-gcp.adoc: Renamed from gerund to imperative title, updated abstract, added note recommending WIF - configuring-log-forwarding.adoc: Added WIF module include Key features documented: - Environment variable setup for GCP project, pool, and provider IDs - CredentialsRequest creation with correct audience format (uses project number, not project ID) - Manual cloudTokenPath fix required after ccoctl generates secret - Complete verification steps - Both client secret and WIF authentication examples Style improvements per peer review: - Changed gerund titles to imperative verbs - Replaced "You can" constructions with imperative mood - Replaced "by using" with "using" for conciseness - Fixed dangling participial phrase (split into two sentences) - Replaced parenthetical with commas - Improved prerequisite clarity Signed-off-by: John Wilkins <jowilkin@redhat.com>
Add documentation for the new azureLogsIngestion output type, which uses
the Azure Monitor Logs Ingestion API with Data Collection Rules (DCR).
This replaces the deprecated azureMonitor output type.
Critical deadline: Microsoft will disable the old Data Collector API on
September 14, 2026. Customers using the legacy azureMonitor output must
migrate before this date.
New modules (10):
- about-azure-logs-ingestion-api.adoc: Overview of new API and advantages
over deprecated Data Collector API
- azure-logs-ingestion-dcr-requirements.adoc: DCR prerequisites intro
- azure-dcr-components.adoc: DCR component details
- azure-dcr-required-information.adoc: Required DCR information for config
- azure-dcr-authentication-permissions.adoc: DCR permissions requirements
- azure-logs-ingestion-authentication.adoc: Authentication methods intro
- azure-auth-client-secret.adoc: Client secret authentication concept
- azure-auth-workload-identity.adoc: Workload identity authentication concept
- azure-auth-permissions.adoc: Authentication permissions reference
- logging-forwarding-azure-logs-ingestion.adoc: Configuration procedure
with complete examples for client secret auth, workload identity auth,
and custom timestamp fields (ASIM support)
Updated modules (2):
- clf-outputs-reference.adoc: Added azureLogsIngestion output type
- configuring-log-forwarding.adoc: Included new Azure modules
Key features documented:
- Data Collection Rules (DCR) configuration requirements
- Two authentication methods: client secret and workload identity
- Custom timestamp field support for ASIM and other specialized schemas
- Migration guidance from deprecated azureMonitor output
- Both outputs can coexist during migration period
Documentation structure:
- Concepts introduce topics before procedures (modular docs best practice)
- No level 2 headings in modules (split into submodules with leveloffset=+3)
- No DITA-incompatible elements
- Red Hat attributes used throughout ({product-title}, {clo}, {oc-first})
- AsciiDocDITA compliant
Microsoft Entra ID rebranding:
- Created new {entra-id} attribute for Microsoft Entra ID platform
- Replaced all "Azure AD" and "Azure Active Directory" references
- Updated secret names from azure-ad-secret to azure-entra-secret
- Changed ClusterLogForwarder name from "instance" to "azure-logs-forwarder"
for clarity
Style improvements per peer review:
- Changed gerund title to imperative verb
- Replaced "You can" and "by using" constructions
- Changed prerequisites from present perfect to simple past tense
- Replaced "permissions" with "access" for user context
- Replaced "is retiring" with "will disable" for future events
- Split complex sentences for clarity
- Use "for example" instead of "such as" for specific instances
- Use "uses" instead of "follows" with technical patterns
- Use "a" before consonant sounds (entra-id)
- Added purpose to sentence fragment in reference content
- Removed "You can" from procedure steps
Related: OBSDOCS-3375, OBSDA-804
Signed-off-by: John Wilkins <jowilkin@redhat.com>
OBSDOCS-3375: Document Azure Logs Ingestion API
OBSDOCS-1639: Document GCP Workload Identity Federation authentication
…ion issues This commit consolidates multiple fixes to the installation documentation: - Add OpenStack Swift storage configuration for LokiStack - Address peer review style issues and DITA compliance warnings - Remove duplicate storage/sizing sections from configuring guide - Fix xref paths for log forwarding configuration - Address PR feedback from support team - Split Loki operator installation troubleshooting into separate module The changes improve the installation workflow by providing clearer guidance on storage configuration, reducing duplication, and addressing DITA validation issues that prevented proper portal rendering. Signed-off-by: John Wilkins <jowilkin@redhat.com> Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add documentation for configuring Splunk sourceType metadata using pod labels. This enhancement allows users to set custom sourceType values per application via kubernetes.pod_labels annotations in their Splunk output configuration. The documentation includes: - New procedure module for configuring sourceType pod labels - Updated default metadata table to include sourceType examples - Integration with existing Splunk forwarding documentation Signed-off-by: John Wilkins <jowilkin@redhat.com> Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
OBSDOCS-3551, OBSDOCS-3552, OBSDOCS-3553: Fix installation documentation issues
OBSDOCS-3446: Document Splunk sourceType configuration
Add comprehensive documentation for log filtering use cases: - Add "Understanding log filtering use cases" concept module - Document filter types: drop, prune, detectMultilineException, rate limiting - Add decision workflow for choosing appropriate filtering approach - Document filter placement and pipeline design - Add performance and cost trade-offs table - Add "Configuring drop filters for performance" procedure - Scenario 1: Drop debug logs from specific namespace - Scenario 2: Drop health check requests cluster-wide - Scenario 3: Drop logs by many criteria with complex filtering - Add verification steps with collector metrics - Add "Configuring prune filters for cost optimization" procedure - Scenario 1: Prune verbose Kubernetes metadata - Scenario 2: Prune high-cardinality fields for external forwarding - Scenario 3: Apply selective pruning per pipeline - Add payload size comparison verification - Add "Configuring multi-pipeline filtering for many teams" procedure - Basic scenario: Three teams with different requirements - Advanced scenario: Namespace-based routing with per-team filters - Add multi-destination verification steps SME feedback addressed: - Do not prune labels (preserve for Kubernetes queries and service-to-pod matching) - Remove legacy fluentd fields that no longer exist in Vector log format - Use current log format fields from cluster-logging-operator examples Peer review edits for style guide compliance: - Use "provide" instead of "offer" for technical content - Use "environment" instead of "use case" in procedure steps - Use "are true" instead of "match" for condition testing - Spell out "regular expression" before using "regex" - Break complex sentences into separate sentences - Use active voice instead of passive voice - Use "require" instead of "often need" for requirements - Use complete sentences in tables with subjects and verbs - Use "the pipeline" instead of "your pipeline" in concepts Signed-off-by: John Wilkins <jowilkin@redhat.com>
OBSDOCS-3539: Add use case guidance and examples for log filtering
…mpact Adds comprehensive documentation for understanding and managing Vector log collector metrics cardinality in OpenShift logging deployments. New Content: - Concept module explaining metrics cardinality and monitoring stack impact - Procedure for configuring metrics collection profiles (minimal vs full) - Reference for minimal metrics collected by the log collector - Best practices for multitenant logging to reduce cardinality - Troubleshooting procedure for diagnosing high cardinality issues - Assembly section organizing all cardinality-related content Key Features: - Explains how ClusterLogForwarder configuration affects component count - Documents histogram metrics that create high cardinality - Provides metrics collection profiles as primary remediation strategy - Includes multitenant configuration patterns that minimize cardinality - Offers diagnostic procedures using Prometheus tools - Contains estimates for predicting cardinality impact Technical Fixes Applied: - DITA compliance: removed callouts, fixed nested structures - Vale linting: replaced gerunds with clear subjects - S3 authentication: corrected to awsAccessKey per CRD spec - Collection profiles: removed unsupported telemetry profile - Style guide: imperative mood, complete sentences, proper articles Files Added: - modules/collector-metrics-cardinality-impact.adoc - modules/configuring-collector-metrics-profiles.adoc - modules/collector-minimal-metrics-reference.adoc - modules/best-practices-multitenant-logging.adoc - modules/troubleshooting-collector-metrics-cardinality.adoc Files Modified: - configuring/cluster-logging-collector.adoc (added cardinality section) Addresses SME and peer review feedback with technical accuracy verified against ClusterLogForwarder CRD and Cluster Monitoring Operator source. Signed-off-by: John Wilkins <jowilkin@redhat.com>
OBSDOCS-3383: Document Vector metrics cardinality and monitoring impact
Documents the useRequestsAsLimits field in LokiStack, which sets resource
limits equal to resource requests for all components. This feature is in
Technology Preview.
New module: modules/loki-resource-limits.adoc
- Explains when to use resource limits (node exhaustion, predictable
consumption, shared node environments)
- Important warning about reduced load spike handling ability
- Procedure for checking current resource usage before enabling
- Configuration example using definition list (DITA-compatible)
- Verification steps
Assembly update: configuring/configuring-the-log-store.adoc
- Added module to "Enhanced reliability and performance" section
Technical accuracy improvements (SME review):
- Corrected description of OOM killer and CPU throttling mechanisms
(kernel-level operations, not Kubernetes-level)
- Added specific technical details: Exit Code 137, CFS mechanism
- Clarified impacts: latency/timeouts/sluggishness for CPU throttling,
container restarts for OOMKilled events
- Improved multitenant scenario explanation
Style and clarity improvements (peer review):
- Applied 21 style guide fixes for conciseness, active voice, clarity
- Changed "Current usage" to "Peak usage" for technical accuracy
- Split compound sentences, broke "which" clauses
- Used restrictive clauses ("that run" vs "running")
- Removed possessives, redundant phrases, implied text
DITA compliance:
- All subsections use level 1 (=) headings per modular docs standard
- Assembly uses leveloffset=+2 for proper nesting
- Replaced callout with definition list for DITA compatibility
Vale results: 0 errors, 2 warnings (false positives for standard
.Prerequisites and .Procedure block titles), 8 suggestions (expected)
References:
- JIRA: OBSDOCS-1807
- KCS: https://access.redhat.com/solutions/7039657
- Versions: 6.6, 6.5, 6.4
Signed-off-by: John Wilkins <jowilkin@redhat.com>
OBSDOCS-1807: Document LokiStack resource limits (Technology Preview)
Complete Phase 1 simple fixes for AsciiDocDITA compliance on active OpenShift Logging modules (Steps 1-6). Changes: - Remove 37 callouts, convert to definition lists (Step 1) - Add 69 short descriptions with [role="_abstract"] (Step 2) - Add 1 missing content type attribute (Step 3) - Fix 3 block title violations (Step 4) - Move 4 concept links to Additional resources (Step 5) - Fix 2 duplicate titles by removing unused tag structure (Step 6) - Move 86 orphaned modules to modules/orphan/ - Move deprecated Elasticsearch assembly + 3 modules to orphan directories - Fix include paths in orphaned assembly - Add definite article "the" per IBM Style Guide Scope: - 223 active modules (included in assemblies) - Fixed 82 AsciiDocDITA violations - ~67 violations remain (require module restructuring) Deferred work documented in ASCIIDOCDITA_MODULES_NEEDING_RESTRUCTURE.md Related: OBSDOCS-3620 Signed-off-by: John Wilkins <jowilkin@redhat.com> Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Fix AsciiDocDITA callout violations
This commit addresses Vale errors and warnings across the OpenShift Logging documentation to achieve CQA compliance and DITA compatibility. Key changes: - Fixed Vale style violations (RedHat style guide compliance) - Resolved AsciiDocDITA errors (nested sections, unsupported titles) - Updated Google Cloud branding from "GCP" to "Google Cloud" - Fixed "using" → "by using" constructions - Added backticks to technical terms - Connected NOTE blocks to procedure steps with + continuation Deprecated Elasticsearch content: - Deleted 5 deprecated Elasticsearch troubleshooting modules - Removed all content from modules/orphan/ directory (127 files) - Updated troubleshooting-logging-alerts.adoc to remove ES references Release notes restructuring: - Split logging-release-notes-6-3-0.adoc into 5 separate modules: - logging-release-notes-6-3-0-overview.adoc (CONCEPT) - logging-release-notes-6-3-0-features.adoc (REFERENCE) - logging-release-notes-6-3-0-tech-preview.adoc (REFERENCE) - logging-release-notes-6-3-0-bug-fixes.adoc (REFERENCE) - logging-release-notes-6-3-0-known-issues.adoc (REFERENCE) - Resolved DITA nested section errors by creating flat module structure Loki resource limits: - Split loki-resource-limits.adoc into assembly + 6 modules - Created configuring-loki-resource-limits.adoc assembly - Created 6 new modules for DITA compliance Azure Entra authentication: - Created logging-loki-storage-azure-entra.adoc module - Separated Entra content from Azure storage modules - Added parameter definitions and verification section AWS STS authentication: - Created logging-loki-storage-aws-sts.adoc module - Restored content accidentally deleted during DITA fixes - Added parameter definitions and verification section Peer review feedback: - Fixed grammar: "deployments using" → "deployments that use" - Removed backticks from prose words (subquery, queriers, etc.) - Fixed "if of" → "if a...occurs" - Hardcoded GCP_SA_NAME and GCPProviderSpec in code blocks - Removed double + continuation marker - Made table formatting consistent - Changed "will delete" to "deletes" (present tense) - Added comma after introductory phrase - Removed redundant sentences - Changed "that is" → "such as" for examples - Changed "Observe" → "Review" in procedure steps - Fixed "user configured" → "user-configured" - Removed "much more" intensifier - Changed "equal to" → "equivalent to" - Added missing article "the" - Fixed "Performance degradation results:" grammar - Changed "OOMKilled" → "OOM killed" in prose - Merged duplicate description list entries - Converted .Verification to == Example condition messages - Converted NOTE to .Troubleshooting section Status: All directories now have 0 errors, 0 warnings except for ignorable TaskInclude warnings for navigation snippet includes. Signed-off-by: John Wilkins <jowilkin@redhat.com>
OBSDOCS-3628: Fix Vale and DITA compliance issues
…lies Addresses DITA compliance and page size constraints by splitting the 200+ include configuring-log-forwarding.adoc mega-assembly into: - 1 core assembly (theory, security, deployment, troubleshooting) - 5 pipeline component assemblies (inputs, filters, outputs, pipelines, advanced) - 6 destination assemblies (third-party, GCP, Splunk, Kafka, Azure, CloudWatch) Benefits: - Eliminates all DITA warnings (embedded level 2 headings removed) - Keeps pages under 10 topics / 2000 words per CCSINTL-1948 constraints - Improves navigation (12 focused pages vs 1 massive page) - Aligns with JTBD framework (each page = distinct user job) - Isolates destination changes for easier maintenance Integrates foundation for OBSDOCS-3627 output format modules. Updates topic map with 11 new entries and adds cross-references. All assemblies validated with prow build scripts. Zero DITA warnings. Signed-off-by: John Wilkins <jowilkin@redhat.com>
…g, and HTTP Completes OBSDOCS-3627 by adding detailed format reference modules for the remaining output types: - Splunk HEC: Complete event structure, metadata mappings, field transformations - Loki/LokiStack: Label extraction, stream format, LogQL query examples - Syslog: RFC 5424/3164 format, severity/facility mapping, field transformations - HTTP: JSON payload structure, batching, authentication, error handling Each module follows the established pattern: - Abstract explaining the transformation - Field mapping tables (OpenShift → destination format) - Example transformations (before/after) - Relevant additional resources Integrates modules into appropriate assemblies: - forwarding-to-splunk.adoc: Added splunk-hec-output-format - configuring-the-log-store.adoc: Added loki-output-format - forwarding-to-third-party-systems.adoc: Added http-output-format, syslog-output-format Job openshift#32 status: 8 of 8 output types now documented (100% complete) - Google Cloud Logging ✓ - Kafka ✓ - Azure Monitor ✓ - CloudWatch ✓ - Splunk ✓ (now complete with full HEC format) - Loki/LokiStack ✓ - Syslog ✓ - HTTP ✓ All modules validated with prow build scripts. Signed-off-by: John Wilkins <jowilkin@redhat.com>
Addresses all errors and critical warnings from Vale linting: Terminology fixes: - "timezone" → "time zone" (syslog-output-format) - "Kernel" → "kernel" (syslog-output-format) - "Mutual TLS" → "mutual TLS" (http-output-format) Grammar improvements: - "using" → "by using" after nouns (5 instances across modules) Clarity improvements: - "overhead" → "network traffic" / "storage requirements and query processing time" (3 instances) - More specific language for technical accuracy Remaining Vale warnings are false positives: - Block titles: Acceptable in reference modules for organizing examples - Callout warnings: Vale misinterpreting syslog priority syntax <11>, <14> as AsciiDoc callouts - Hard-wrapped lines: Single-line explanatory text, not wrapped paragraphs Vale results: 0 errors, 18 warnings (all acceptable/false positives), 24 suggestions Signed-off-by: John Wilkins <jowilkin@redhat.com>
openshift-ci
Bot
added
the
needs-rebase
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
openshift-ci
Bot
added
the
size/XXL
|
@johnwilkins: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Contributor
Author
|
Closing - incorrect base branch (main instead of standalone-logging-docs-main). Will recreate with correct target. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
17 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Complete output format documentation (OBSDOCS-3627)
Closes OBSDOCS-3627 and completes Job #32 (log forwarding output formats).
What Changed
Added comprehensive format reference modules for all 8 supported output types:
New format modules:
gcp-cloud-logging-output-format.adoc- Google Cloud LogEntry API structure, field mappings, severity mappingkafka-message-structure.adoc- Kafka message protocol, configuration field mappingazure-logs-ingestion-output-format.adoc- Azure Monitor DCR structure, request format, KQL transformationscloudwatch-output-format.adoc- CloudWatch PutLogEvents API structure, field transformationssplunk-hec-output-format.adoc- Complete Splunk HEC event structure, metadata mappingsloki-output-format.adoc- Loki streams, label extraction, LogQL query examplessyslog-output-format.adoc- RFC 5424/3164 formats, severity/facility mappinghttp-output-format.adoc- HTTP JSON payload structure, batching, authenticationAssembly restructuring (prerequisite for format integration):
configuring-log-forwarding.adocmega-assembly (200+ includes) into 12 focused assembliesIntegration:
Coverage
Before: 1 of 8 formats partially documented (Splunk metadata only)
After: 8 of 8 formats fully documented (100% complete)
User Value
Users can now:
Testing
scripts/check-asciidoctor-build.sh)Related Work
Assembly restructuring addresses CCSINTL-1948 Phase 0 requirements (DITA compliance, page size constraints, JTBD framework alignment) and creates the infrastructure needed for format module integration.
Commits
Files changed: 27 total (2 modified assemblies, 11 new assemblies, 7 new format modules, 7 style-fixed modules)