Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: source port validation #2434

Merged
merged 4 commits into from
May 27, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions docs/troubleshooting/troubleshoot_SC4S_server.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,19 @@ just _one_ bad index will "taint" the entire batch (in this case, 1000 events) a
imperative that the container logs be free of these kinds of errors in production._ You can use the alternate HEC debug destination (below)
to help debug this condition by sending direct "curl" commands to the HEC endpoint outside of the SC4S setting.

## Invalid SC4S listen ports

[SC4S exclusively grant a port to a device when `SC4S_LISTEN_{vendor}_{product}_{TCP/UDP/TLS}_PORT={port}`](https://splunk.github.io/splunk-connect-for-syslog/main/sources/#unique-listening-ports).

During startup, SC4S validates that listening ports are configured correctly, and in case of misconfiguration, you will be able see any issues in container logs.

You will receive an error message similar to the following if listening ports for `MERAKI SWITCHES` are configured incorrectly:
```
SC4S_LISTEN_MERAKI_SWITCHES_TCP_PORT: Wrong port number, don't use default port like (514,614,6514)
SC4S_LISTEN_MERAKI_SWITCHES_UDP_PORT: 7000 is not unique and has already been used for another source
SC4S_LISTEN_MERAKI_SWITCHES_TLS_PORT: 999999999999 must be integer within the range (0, 10000)
```

## SC4S Local Disk Resource Considerations
* Check the HEC connection to Splunk. If the connection is down for a long period of time, the local disk buffer used for backup will exhaust local
disk resources. The size of the local disk buffer is configured in the env_file: [Disk buffer configuration](https://splunk-connect-for-syslog.readthedocs.io/en/latest/configuration/#disk-buffer-variables)
Expand Down
1 change: 1 addition & 0 deletions package/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ COPY package/etc/local_config /etc/syslog-ng/local_config
COPY package/etc/local_config /etc/syslog-ng/local_config
COPY package/sbin/entrypoint.sh /
COPY package/sbin/healthcheck.sh /
COPY package/sbin/source_ports_validator.py /

ENV SC4S_CONTAINER_OPTS=--no-caps
ARG VERSION=unknown
Expand Down
1 change: 1 addition & 0 deletions package/Dockerfile.lite
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ COPY package/lite/etc/addons /etc/syslog-ng/addons

COPY package/sbin/entrypoint.sh /
COPY package/sbin/healthcheck.sh /
COPY package/sbin/source_ports_validator.py /


RUN chmod -R 755 /etc/syslog-ng/
Expand Down
2 changes: 2 additions & 0 deletions package/sbin/entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,8 @@ fi
export SOURCE_SIMPLE_SET=$(printenv | grep '^SC4S_LISTEN_SIMPLE_.*_PORT=.' | sed 's/^SC4S_LISTEN_SIMPLE_//;s/_..._PORT\=.*//;s/_[^_]*_PORT\=.*//' | sort | uniq | xargs echo | sed 's/ /,/g' | tr '[:upper:]' '[:lower:]' )
export SOURCE_ALL_SET=$(printenv | grep '^SC4S_LISTEN_.*_PORT=.' | grep -v "disabled" | sed 's/^SC4S_LISTEN_//;s/_..._PORT\=.*//;s/_[^_]*_PORT\=.*//' | sort | uniq | xargs echo | sed 's/ /,/g' | tr '[:lower:]' '[:upper:]' )

python3 /source_ports_validator.py

syslog-ng --no-caps --preprocess-into=- | grep vendor_product | grep set | grep -v 'set(.\$' | sed 's/^ *//' | grep 'value("fields.sc4s_vendor_product"' | grep -v "\`vendor_product\`" | sed s/^set\(// | cut -d',' -f1 | sed 's/\"//g' >/tmp/keys
syslog-ng --no-caps --preprocess-into=- | grep 'meta_key(.' | sed 's/^ *meta_key(.//' | sed "s/')//" >>/tmp/keys
rm -f $SC4S_ETC/conf.d/local/context/splunk_metadata.csv.example >/dev/null || true
Expand Down
41 changes: 41 additions & 0 deletions package/sbin/source_ports_validator.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
import collections
import os
import logging


logger = logging.getLogger(__name__)


def is_valid_port(raw_port: str) -> bool:
return raw_port.isdigit() and 0 < int(raw_port) < 65565


def validate_source_ports(sources: list[str]) -> None:
source_ports = {}
for source in sources:
for proto in ["TCP", "UDP", "TLS", "RFC5426", "RFC6587", "RFC5425"]:
source_ports[(source, proto)] = os.getenv(f"SC4S_LISTEN_{source}_{proto}_PORT", "disabled").split(",")


busy_ports_for_proto = collections.defaultdict(set)
for source, proto in source_ports.keys():
for port in source_ports[(source, proto)]:
if not port or port == "disabled":
continue

elif not is_valid_port(port):
logger.error(f"SC4S_LISTEN_{source}_{proto}_PORT: {port} must be integer within the range (0, 65565)")

elif source != "DEFAULT" and port in source_ports[("DEFAULT", proto)]:
logger.error(f"SC4S_LISTEN_{source}_{proto}_PORT: Wrong {port} number, don't use default port like {port}")

elif port in busy_ports_for_proto[proto]:
logger.error(f"SC4S_LISTEN_{source}_{proto}_PORT: {port} is not unique and has already been used for another source")

else:
busy_ports_for_proto[proto].add(port)


if __name__ == "__main__":
sources = os.getenv("SOURCE_ALL_SET").split(",")
validate_source_ports(sources)
Loading