Skip to content

Add security violations filter processor, and plug it in to the default OTLP logs pipeline when TcplogReceiver is configured#1576

Merged
dhurley merged 3 commits intonginx:mainfrom
kamalchaturvedi:security_violations_filter_processor
Apr 7, 2026
Merged

Add security violations filter processor, and plug it in to the default OTLP logs pipeline when TcplogReceiver is configured#1576
dhurley merged 3 commits intonginx:mainfrom
kamalchaturvedi:security_violations_filter_processor

Conversation

@kamalchaturvedi
Copy link
Copy Markdown
Contributor

@kamalchaturvedi kamalchaturvedi commented Mar 25, 2026

Proposed changes

COMMIT 1:

Add a new security violations filter processor that validates the first message as a one-time gate — on the very first log record, it checks:
- The body is a string (not int, bytes, etc.).
- The body has exactly 28 pipe-separated fields matching the secops-dashboard-log profile format.

If either check fails, the gate closes permanently — all subsequent messages are dropped with zero overhead (early return before any iteration) until the OTel collector is restarted. If it passes, the gate opens permanently and all future string-bodied records flow through.

COMMIT 2:

Add support for NAP V5 (containerized mode) by verifying syslog IP configured against docker0 interface IP as well (along with 127.0.0.1)

COMMIT 3:

Plug security violation filter processor into the default logs pipeline, replacing the:
- logsgzipprocessor: This processor was introduced for NGINX One for a deprecated project, which is not in scope anymore.
- securityviolationsprocessor: This processor was converting the sylog security violation into a JSON body log record format, and then forwarding it. This deterministix JSON log body conversion is an overhead for the agent to process for every single log record.

Verification

  • Default OTEL pipeline now, once TcplogReceiver is configured by the agent (when app_protect_security_log directive has syslog:server=127.0.0.1:1514 configured)
  pipelines:
    metrics/default:
      receivers:
        - hostmetrics
      processors:
        - batch/default_metrics
        - resource/default
      exporters:
        - otlp/default
    logs/default:
      receivers:
        - tcplog/nginx_app_protect
      processors:
        - securityviolationsfilter/default
        - batch/default_logs
        - resource/default
      exporters:
        - otlp/default
  • In the first log the processor encounters, if valid number of CSV separated fields are sent as per the log-profile configured, the violation is forwarded to the management plane (as per the batch settings). Any subsequent violation is also forwarded.

  • In the first log the processor encounters, if the expected CSV separated fields are not sent as per the log-profile configured , the violation is filtered out. Any subsequent violation is also filtered out, and not forwarded by agent.

Here is the error log that can be verified in the /var/log/nginx-agent/opentelemetry-collector-agent.log file for that error case:

2026-03-25T21:44:42.233Z	error	securityviolationsfilterprocessor/processor.go:102	Security violation log does not appear to be CSV format. Ensure the NAP logging profile uses the secops-dashboard-log format. All security violation logs will be dropped until the collector is restarted.	{"resource": {"service.instance.id": "3ff23ae7-7713-4745-bc28-cfc2198f8eb4", "service.name": "otel-nginx-agent", "service.version": "v3.8.0"}, "otelcol.component.id": "securityviolationsfilter/default", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "logs/default", "otelcol.signal": "logs", "expected_fields": 27, "actual_fields": 1}
github.com/nginx/agent/v3/internal/collector/securityviolationsfilterprocessor.(*securityViolationsFilterProcessor).ConsumeLogs.(*securityViolationsFilterProcessor).ConsumeLogs.ResourceLogsSlice.All.func2.(*securityViolationsFilterProcessor).ConsumeLogs-range1.(*securityViolationsFilterProcessor).ConsumeLogs.(*securityViolationsFilterProcessor).ConsumeLogs.ResourceLogsSlice.All.func2.(*securityViolationsFilterProcessor).ConsumeLogs-range1.ScopeLogsSlice.All.func5.(*securityViolationsFilterProcessor).ConsumeLogs.(*securityViolationsFilterProcessor).ConsumeLogs.ResourceLogsSlice.All.func2.(*securityViolationsFilterProcessor).ConsumeLogs-range1-range4.func6.1

Checklist

Before creating a PR, run through this checklist and mark each as complete.

  • I have read the CONTRIBUTING document
  • I have run make install-tools and have attached any dependency changes to this pull request
  • If applicable, I have added tests that prove my fix is effective or that my feature works
  • If applicable, I have checked that any relevant tests pass after adding my changes
  • If applicable, I have updated any relevant documentation (README.md)
  • If applicable, I have tested my cross-platform changes on Ubuntu 22, Redhat 8, SUSE 15 and FreeBSD 13

@kamalchaturvedi kamalchaturvedi requested a review from a team as a code owner March 25, 2026 03:13
@kamalchaturvedi kamalchaturvedi force-pushed the security_violations_filter_processor branch from 9ce64b2 to 482dc34 Compare March 26, 2026 05:47
@github-actions github-actions Bot added chore Pull requests for routine tasks dependencies labels Mar 26, 2026
@kamalchaturvedi kamalchaturvedi changed the title Draft: Add security violations filter processor, and plug it in to the default OTLP logs pipeline when TcplogReceiver is configured Add security violations filter processor, and plug it in to the default OTLP logs pipeline when TcplogReceiver is configured Mar 26, 2026
Comment thread internal/config/config.go Outdated
Comment thread internal/collector/securityviolationsfilterprocessor/factory.go
Comment thread internal/datasource/config/nginx_config_parser.go
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 1, 2026

Codecov Report

❌ Patch coverage is 88.23529% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.94%. Comparing base (4a95e53) to head (1f88bd7).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
internal/datasource/config/nginx_config_parser.go 71.42% 5 Missing and 3 partials ⚠️
...ector/securityviolationsfilterprocessor/factory.go 42.85% 3 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1576      +/-   ##
==========================================
- Coverage   85.02%   84.94%   -0.08%     
==========================================
  Files         103      105       +2     
  Lines       13589    13680      +91     
==========================================
+ Hits        11554    11621      +67     
- Misses       1518     1546      +28     
+ Partials      517      513       -4     
Files with missing lines Coverage Δ
internal/collector/factories.go 100.00% <100.00%> (ø)
...tor/securityviolationsfilterprocessor/processor.go 100.00% <100.00%> (ø)
internal/config/config.go 88.37% <100.00%> (+0.65%) ⬆️
internal/config/types.go 87.09% <ø> (ø)
...ector/securityviolationsfilterprocessor/factory.go 42.85% <42.85%> (ø)
internal/datasource/config/nginx_config_parser.go 79.25% <71.42%> (-0.22%) ⬇️

... and 7 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4a95e53...1f88bd7. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…st message as a one-time gate — on the very first log record, it checks:

- The body is a string (not int, bytes, etc.).
- The body has exactly 27 pipe-separated fields matching the secops_dashboard-log profile format.

If either check fails, the gate closes permanently — all subsequent messages are dropped with zero overhead (early return before any iteration) until the OTel collector is restarted. If it passes, the gate opens permanently and all future string-bodied records flow through.
…nfigured against docker0 interface IP as well (along with 127.0.0.1)
@kamalchaturvedi kamalchaturvedi force-pushed the security_violations_filter_processor branch from 8f02abb to a944b3e Compare April 2, 2026 02:20
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Apr 2, 2026
…ne, replacing the:

- logsgzipprocessor: This processor was introduced for NGINX One for a deprecated project, which is not in scope anymore.
- securityviolationsprocessor: This processor was converting the sylog security violation into a JSON body log record format, and then forwarding it. This deterministed JSON log body parsing is an overhead for the agent to process for every single log record.
@kamalchaturvedi kamalchaturvedi force-pushed the security_violations_filter_processor branch from a944b3e to 1f88bd7 Compare April 3, 2026 19:37
@dhurley dhurley merged commit 3d4cf59 into nginx:main Apr 7, 2026
27 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore Pull requests for routine tasks dependencies documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants