feat: HTTP Processing & Secrets Retrieval

[jshlbrd]

Description

• Adds an HTTP processor that fulfills a variety of GET and POST request use cases
  • Adds two HTTP processing patterns for interacting with GreyNoise
• Adds functionality to retrieve and interpolate secrets from environment variables or AWS Secrets Manager
• Adds capability to use a KV Store outside of the internal package
• Refactors the HTTP and Sumo Logic sinks

Motivation and Context

Similar to the DNS processor, the HTTP processor lets users enrich data by making HTTP requests (GET or POST). This one is more complex than DNS for two reasons:

• HTTP requests to external services usually require authentication
• HTTP can host more diverse content than DNS can

We're addressing authentication through a feature referred to as secrets retrieval. Components across the system (conditions, processors, sinks) can opt in to secrets retrieval and interpolate secrets with configuration files during runtime. Here's a summary of how this works:

• It relies on the templated variable ${SECRETS ...} (the same design is used in other regexp-based libraries) to identify secrets inside of configurations
• If the templated variable is found, then it is parsed into a secrets location and secrets name
• The location (either env var or Secrets Manager) is queried to retrieve the secret associated with the name
• The secret is interpolated with the configuration using string replacement and stored in memory for up to 15 minutes

Secrets retrieval is designed similar to internal/file, so new secrets locations can be added as needed.

We're addressing the diversity of content and use cases by allowing more flexibility in the processor's data and object handling patterns compared to other processors. These are the known use cases it supports, though there may be more:

• Integration with any external enrichment service that supports GET and POST requests (intended use case)
• Download (GET) text files and insert their value into objects
• Download (GET) non-text files (e.g., binary) and overwrite the contents of a capsule
• Send (POST) data from objects to an HTTP server without modifying the capsule

How Has This Been Tested?

The changes were integration tested using the two new examples files. The HTTP processor use cases was tested with these external services:

• GreyNoise
• VirusTotal
• ipinfo.io
• Cloudflare

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.

[shellcromancer]

Super cool feature add, unlocks the whole web for enrichment and processing. 🕸️ 🌐

Left some comments around the secrets package but otherwise looks super.

[shellcromancer]

Looks great, thanks for adding the tests!