Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: adds auditlog plugins API #787

Merged
merged 15 commits into from
May 16, 2023
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
72 changes: 33 additions & 39 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ SecDatasets are added as replacement for .data files. WASM support is an essenti

Two new SecLang operators are added which can be used to query datasets. `pmFromDataset` [#361](https://github.com/corazawaf/coraza/pull/361) and `ipMatchFromDataset` [#75e8217](https://github.com/corazawaf/coraza/commit/75e821700de9fbfafde6c763f474c7add8dab319) which can be used instead of their file based equivalents for those environments which can't access the filesystem.


```apache
SecDataset restricted-files-1 `
.my.cnf
Expand All @@ -33,15 +32,15 @@ SecRule REQUEST_FILENAME "@pmFromDataset restricted-files-1" \
"...msg:'Match sample_dataset'"
```

* **FEATURE: Restpath Operator** - [#282](https://github.com/corazawaf/coraza/pull/282) - `@restpath` takes a path pattern as a parameter eg. `/path/to/{id}/{name}` and aids evaluation of application urls. The path will be transformed to a regex and assigned to variables in `ARGS_PATH`, `ARGS_NAMES`, and `ARGS`.
* **FEATURE: Restpath Operator** - [#282](https://github.com/corazawaf/coraza/pull/282) - `@restpath` takes a path pattern as a parameter eg. `/path/to/{id}/{name}` and aids evaluation of application urls. The path will be transformed to a regex and assigned to variables in `ARGS_PATH`, `ARGS_NAMES`, and `ARGS`.

```apache
SecRule REQUEST_URI "@restpath /some/random/url/{id}/{name}" "…..chain"
SecRule ARGS_PATH:id "!@eq %{user:session_id}" "deny"
```

* **FEATURE: Redirect Operator** - [#290](https://github.com/corazawaf/coraza/pull/290) -
Redirect takes a status code and url as parameter and based on this information returns an http redirect to the client.
Redirect takes a status code and url as parameter and based on this information returns an http redirect to the client.

```apache
SecRule REQUEST_URI "/redirect" "phase:1,id:1,status:302,redirect:http://www.example.com
Expand All @@ -62,67 +61,62 @@ SecRule MULTIPART_PART_HEADERS "Content-Disposition" "id:300, phase:2, log"

Similar performance to modsecurity is archived, we are faster or slower, depending on the payload. Usually, big payloads work better in Coraza. Coraza v3 is ~50% faster than Coraza v2.0.1, more than 200% faster than Coraza v2.0.0.

- Replace the Aho Corasick string matching implementation used internally with Petar Dambovaliev’s implementation (60% less memory consumption and 233% faster execution time) [#302](https://github.com/corazawaf/coraza/pull/302)
- Optimize validateNID operator [#30e5b56](https://github.com/corazawaf/coraza/commit/30e5b564d4d7c6688fb819c97b0891e097570a2e) [#348](https://github.com/corazawaf/coraza/pull/348)
- Use io.Discard instead of /dev/null to save a syscall when debug log output should be discarded [#354](https://github.com/corazawaf/coraza/pull/354)
- Optimize Body Buffering [#505](https://github.com/corazawaf/coraza/pull/505)
- Remove unused mutex in RuleGroup [#381](https://github.com/corazawaf/coraza/pull/381)
- Speed up random string generation by switching to a pseudorandom generator [#403](https://github.com/corazawaf/coraza/pull/403)
- Improve SecLang parser performance [#412](https://github.com/corazawaf/coraza/pull/412)
- Use strings.Builder to avoid copy string input to bytes in `urlencode` [#320]((https://github.com/corazawaf/coraza/pull/320) and `base64decode` [#319](https://github.com/corazawaf/coraza/pull/319)
- Use lookup table for byte range validation in `validateByteRange` [#490](https://github.com/corazawaf/coraza/pull/490)
* Replace the Aho Corasick string matching implementation used internally with Petar Dambovaliev’s implementation (60% less memory consumption and 233% faster execution time) [#302](https://github.com/corazawaf/coraza/pull/302)
* Optimize validateNID operator [#30e5b56](https://github.com/corazawaf/coraza/commit/30e5b564d4d7c6688fb819c97b0891e097570a2e) [#348](https://github.com/corazawaf/coraza/pull/348)
* Use io.Discard instead of /dev/null to save a syscall when debug log output should be discarded [#354](https://github.com/corazawaf/coraza/pull/354)
* Optimize Body Buffering [#505](https://github.com/corazawaf/coraza/pull/505)
* Remove unused mutex in RuleGroup [#381](https://github.com/corazawaf/coraza/pull/381)
* Speed up random string generation by switching to a pseudorandom generator [#403](https://github.com/corazawaf/coraza/pull/403)
* Improve SecLang parser performance [#412](https://github.com/corazawaf/coraza/pull/412)
* Use strings.Builder to avoid copy string input to bytes in `urlencode` [#320]((<https://github.com/corazawaf/coraza/pull/320>) and `base64decode` [#319](https://github.com/corazawaf/coraza/pull/319)
* Use lookup table for byte range validation in `validateByteRange` [#490](https://github.com/corazawaf/coraza/pull/490)

### API Changes

A lot of effort was added to optimize and clean up the Coraza API which resulted in a couple of breaking API changes.

- **New Variables Engine** - [#277](https://github.com/corazawaf/coraza/pull/277) - Implements a new Variables Engine similar to modsecurity. Variables have two pointers `tx.Collections[]` and `tx.variables.*` which either allow programmatic access using the proper collection mechanism or using dynamic variable names.
* **New Variables Engine** - [#277](https://github.com/corazawaf/coraza/pull/277) - Implements a new Variables Engine similar to modsecurity. Variables have two pointers `tx.Collections[]` and `tx.variables.*` which either allow programmatic access using the proper collection mechanism or using dynamic variable names.

There are multiple variable types (Simple, Map, Proxy, Translation) with different helpers and generic helpers. Each type has its own variable (string, map, proxy, etc.) https://github.com/corazawaf/coraza/tree/v3/dev/collection
There are multiple variable types (Simple, Map, Proxy, Translation) with different helpers and generic helpers. Each type has its own variable (string, map, proxy, etc.) <https://github.com/corazawaf/coraza/tree/v3/dev/collection>

**BREAKING**: Raw data can only be accessed through a RequestBodyProcessor

- **BREAKING**: Library entry points are converted to immutable interfaces [#397](https://github.com/corazawaf/coraza/pull/397)
* **BREAKING**: Library entry points are converted to immutable interfaces [#397](https://github.com/corazawaf/coraza/pull/397)

This simplifies the interface and provides a safe mechanism to invoke Coraza and handle transactions. It further allows for major changes without updating the public API and maintains compatibility. Any calls to seclang.NewParser can be removed.

The approach for doing this is to first move existing interfaces into internal/corazawaf, internal/seclang and start creating public API from scratch delegating to these. After completing migration, there is probably cleanup that could be done which may result in the removal of delegation, but that could happen after locking in a public API.

* **BREAKING**: Rename Waf to WAF to follow Go type naming conventions of acronyms. `coraza.NewWaf` must be accessed as `coraza.NewWAF` and `tx.Waf` as `tx.WAF`. [#373](https://github.com/corazawaf/coraza/pull/373)

* **BREAKING**: Rename methods of *Transaction* [#518](https://github.com/corazawaf/coraza/pull/518)
* Interrupted &rarr; IsInterrupted
* ResponseBodyAccessible &rarr; IsResponseBodyAccessible
* IsProcessableResponseBody &rarr; IsResponseBodyProcessable
* RequestBodyAccessible &rarr; IsRequestBodyAccessible

- **BREAKING**: Rename Waf to WAF to follow Go type naming conventions of acronyms. `coraza.NewWaf` must be accessed as `coraza.NewWAF` and `tx.Waf` as `tx.WAF`. [#373](https://github.com/corazawaf/coraza/pull/373)

- **BREAKING**: Rename methods of *Transaction* [#518](https://github.com/corazawaf/coraza/pull/518)
- Interrupted &rarr; IsInterrupted
- ResponseBodyAccessible &rarr; IsResponseBodyAccessible
- IsProcessableResponseBody &rarr; IsResponseBodyProcessable
- RequestBodyAccessible &rarr; IsRequestBodyAccessible

- **BREAKING: Remove ZAP** - [#682b59](https://github.com/corazawaf/coraza/commit/6828b59811f5a1b0b86213533a71ec9aaea229c8) [#b64ede7](https://github.com/corazawaf/coraza/commit/b64ede757c7409d7ab9e441bbdfcf6157a3aa6b0) - Debug logging is now an interface and Zap is removed . Support to allow `SecDebugLog` to log to `/dev/stderr` or `/dev/stdout` is added in [#449](https://github.com/corazawaf/coraza/pull/#449)
* **BREAKING: Remove ZAP** - [#682b59](https://github.com/corazawaf/coraza/commit/6828b59811f5a1b0b86213533a71ec9aaea229c8) [#b64ede7](https://github.com/corazawaf/coraza/commit/b64ede757c7409d7ab9e441bbdfcf6157a3aa6b0) - Debug logging is now an interface and Zap is removed . Support to allow `SecDebugLog` to log to `/dev/stderr` or `/dev/stdout` is added in [#449](https://github.com/corazawaf/coraza/pull/#449)

- **BREAKING: WAFConfig Type**: A new immutable WAFConfig type is added to initialize Coraza, which replaces seclang.NewParser.
Each WithXXX function (WithRules, WithDirectives, WithDirectivesFromFile, WithAuditLog, WithContentInjection, WithRequestBodyAccess, WithResponseBodyAccess, WithDebugLogger, WithErrorLogger, WithRootFS) of this type returns a new instance including the corresponding change. [#bffb435/config.go#L1
* **BREAKING: WAFConfig Type**: A new immutable WAFConfig type is added to initialize Coraza, which replaces seclang.NewParser.
Each WithXXX function (WithRules, WithDirectives, WithDirectivesFromFile WithContentInjection, WithRequestBodyAccess, WithResponseBodyAccess, WithDebugLogger, WithErrorLogger, WithRootFS) of this type returns a new instance including the corresponding change. [#bffb435/config.go#L1

- **Rules, Action & Transaction Interfaces** - New immutable interfaces are added
* **Rules, Action & Transaction Interfaces** - New immutable interfaces are added

- **BREAKING**: Convert rulematch types to interfaces [#478](https://github.com/corazawaf/coraza/pull/478)
* **BREAKING**: Convert rulematch types to interfaces [#478](https://github.com/corazawaf/coraza/pull/478)

- **BREAKING**: https://github.com/corazawaf/coraza/pull/503
* **BREAKING**: <https://github.com/corazawaf/coraza/pull/503>

- **BREAKING**: Add RequestBodyAccessible() and ResponseBodyAccessible() to types.transaction.

- **BREAKING**: Export Request/Response BodyAccess values [#499](https://github.com/corazawaf/coraza/pull/)
* **BREAKING**: Add RequestBodyAccessible() and ResponseBodyAccessible() to types.transaction.

* **BREAKING**: Export Request/Response BodyAccess values [#499](https://github.com/corazawaf/coraza/pull/)

### Testing

- Introduce new CRS testing suite for Coraza v3 based on Go HTTPServer and go-ftw. Remove Caddy to avoid circular project dependency [#457](https://github.com/corazawaf/coraza/pull/457)
- Automatically perform Benchmarks to detect performance regressions [#301](https://github.com/corazawaf/coraza/pull/301)
- Enhance the test engine to perform tests of returned interruptions (output.interruption).
- Add testing for disruptive actions.
- Switch to mage instead of pre-commit [#315](https://github.com/corazawaf/coraza/pull/315) [#355](https://github.com/corazawaf/coraza/pull/355) [#356](https://github.com/corazawaf/coraza/pull/356)


* Introduce new CRS testing suite for Coraza v3 based on Go HTTPServer and go-ftw. Remove Caddy to avoid circular project dependency [#457](https://github.com/corazawaf/coraza/pull/457)
* Automatically perform Benchmarks to detect performance regressions [#301](https://github.com/corazawaf/coraza/pull/301)
* Enhance the test engine to perform tests of returned interruptions (output.interruption).
* Add testing for disruptive actions.
* Switch to mage instead of pre-commit [#315](https://github.com/corazawaf/coraza/pull/315) [#355](https://github.com/corazawaf/coraza/pull/355) [#356](https://github.com/corazawaf/coraza/pull/356)

--------------------------------------------------------------

Expand Down
174 changes: 0 additions & 174 deletions auditlog/auditlog.go

This file was deleted.

91 changes: 0 additions & 91 deletions auditlog/formats_json.go

This file was deleted.