SG-for-Pods / AWS VPC CNI Diagnostics Toolkit (Linux)

This toolkit collects comprehensive diagnostics for AWS EKS pods using Security Groups for Pods (SGFP). It gathers pod, node, and AWS ENI data into a single bundle, optionally pulls CloudTrail ENI API activity (for throttles/errors), and generates a markdown report plus a post-analyzer summary.

Features

• Comprehensive Pod Diagnostics: Collects pod annotations, conditions, network namespace routes/rules, interface statistics, socket statistics, and reachability tests
• Security Group Validation: Automatically validates actual SGs on pod ENI against expected SGs from pod, deployment, replicaset, or namespace annotations
• SecurityGroupPolicy CRD Compliance: Validates SecurityGroupPolicy CRD resources for compliance with the 5 security groups per policy limit, detects violations, and reports when multiple policies are used
• SG Details: Shows Security Group IDs, names, and descriptions for easy identification
• Node Diagnostics: Collects conntrack usage, interface error statistics, socket overruns, and AWS VPC CNI logs (automatically via temporary debug pod)
• CNI Log Analysis: Automatically collects and analyzes CNI logs from /var/log/aws-routed-eni/ including ipamd, plugin, network-policy-agent, and eBPF SDK logs
• iptables Rule Analysis: Collects and analyzes iptables rules (filter and NAT tables), including pod-specific rule detection and kube-proxy mode validation (iptables vs IPVS)
• kube-proxy Analysis: Validates kube-proxy iptables chains (KUBE-SERVICES, KUBE-NODEPORTS, KUBE-MARK-MASQ), checks masquerade rules, and verifies service rules for pods
• MTU Configuration Analysis: Detects MTU mismatches between pod and node interfaces, identifies fragmentation issues, and checks for multiple MTU values on non-loopback interfaces
• Reverse Path Filtering (rp_filter): Validates rp_filter settings for pod ENI scenarios, recommends loose mode (rp_filter=2) for asymmetric routing support, and flags security risks
• ENI/Instance Limits: Checks instance type ENI and IP limits, compares current usage vs limits, validates trunk ENI branch ENI count, and estimates max pods capacity
• Route Table Drift Analysis: Detects missing or incorrect routes (default route, subnet routes, metadata service route), validates route consistency, and identifies routing issues
• Health Probe Analysis: Analyzes health probe configurations (liveness, readiness, startup), verifies probe ports are listening, checks for probe failures, and validates network access from kubelet
• NetworkPolicy Analysis: Analyzes NetworkPolicies to detect if they block DNS, health probes, metrics, or service traffic, validates podSelector matches, and checks for missing egress rules
• DNS / CoreDNS / NodeLocal DNSCache Analysis: Analyzes DNS infrastructure, CoreDNS pod status and scaling, NodeLocal DNSCache presence and configuration, DNS service endpoints, and DNS resolution tests
• Custom Networking / ENIConfig Analysis: Validates ENIConfig resources (subnet CIDR → AZ mapping), checks node ENIConfig assignments, verifies ENIConfig subnet existence in VPC, and flags configuration mismatches
• NAT Gateway SNAT Port Exhaustion Analysis: Monitors NAT gateway connection counts via CloudWatch, detects approaching/exceeded connection limits (~55,000 per gateway), and provides recommendations for mitigation
• AMI / CNI / Kernel Drift Analysis: Detects version mismatches between Kubernetes, kube-proxy, aws-node, OS image (AMI), and kernel versions, flags non-EKS-optimized AMIs, and identifies outdated components
• Metrics Comparison: Automatically compares baseline metrics (healthy state) with incident metrics to identify changes indicating network problems (DNS errors, ENI/IP allocation failures, pending pods, etc.)
• Connectivity Analysis: Advanced analysis for pod connectivity issues after large churns, including ENI attachment timing, subnet IP availability, CNI log errors, and SYN_SENT connection detection (identifies pods trying to connect but waiting for ACK)
• AWS ENI State: Captures trunk and branch ENI information, subnet IP availability, and instance type details
• API Diagnostics: Analyzes CloudTrail events for ENI-related throttles and errors (with dry-run detection)
• Quick Status Check: Fast validation script to check pod ENI status without full collection (includes optional network connections display with SYN_SENT detection)
• Log Files Summary: Report includes concise summary of all log files with error counts and file paths
• View Related Logs Helper: Helper script to easily view pod-specific log lines from collected bundles
• Node Debug Pod: Helper script to create debug pods on nodes (supports pod name or node name)
• Baseline Metrics Capture: Standalone script to capture cluster metrics snapshot (kubectl stats, Prometheus metrics from CoreDNS, aws-node, kube-proxy, kubelet) for comparison during incidents
• Baseline Comparison: Compare baseline metrics with incident state to identify changes
• All-in-One Doctor Script: Single command to collect, analyze, and report
• Consistent Output Format: Uses [PREFIX] format for clear, parseable output
• Security Group Rules Analysis: Analyzes security group ingress/egress rules to verify cross-node traffic is allowed
• Source Pod Security Group Validation: Checks security groups of source pods attempting to connect to identify mismatches
• Packet Capture Analysis: Analyzes tcpdump/pcap files with pod IP mapping to identify communicating pods and local vs remote traffic
• Network Traffic Capture: Automated script to capture network traffic from a pod's network namespace using tcpdump
• Cluster Pod Snapshot: Comprehensive cluster-wide pod snapshot (IPs, ENIs, metadata) captured at diagnostic time for later analysis (packet captures, historical reference)
• Healthy Pod Baselines: Save healthy pod diagnostic bundles as baselines for comparison with unhealthy pods, with normalized JSON format for automated analysis
• Investigation Tracking: Structured investigation documents to track facts, unknowns, theories, and conclusions for pod connectivity issues

Documentation

Detailed documentation for each diagnostic check is available in the doc/ directory. See the documentation index for a complete list of all checks.

Key checks:

• MTU Configuration Analysis
• kube-proxy iptables Analysis
• Reverse Path Filtering (rp_filter)
• ENI/Instance Limits
• Route Table Drift
• Health Probes
• Network Policies
• DNS / CoreDNS / NodeLocal DNSCache
• Custom Networking / ENIConfig
• NAT Gateway SNAT Port Exhaustion
• Metrics Comparison
• Security Group Validation
• SecurityGroupPolicy CRD Compliance
• IPAMD State Analysis
• Pod Events Analysis
• CloudTrail API Diagnostics
• AWS VPC CNI ConfigMap Settings
• Security Group Rules Analysis
• Source Pod Security Group Validation
• Packet Capture Analysis
• Network Traffic Capture
• Network Namespace Leaks
• DNS Resolution
• SYN_SENT Detection
• CNI Logs Analysis
• Subnet IP Availability
• And more... (see doc/README.md for full list)

Each document explains what we check, why it matters, how we check it, and recommended actions when issues are found.

Requirements

• Linux
• kubectl, jq, awk, grep
• aws CLI configured (and AWS_REGION set, e.g., export AWS_REGION=us-west-2)
• Permissions: ec2:DescribeNetworkInterfaces, optionally CloudTrail lookup-events, optionally CloudWatch GetMetricStatistics (for NAT gateway SNAT port exhaustion detection)

Directory Structure

All diagnostic data and reports are organized by kubectl context (cluster name) for easy identification:

data/
  <cluster-context>/
    sgfp_bundle_<cluster-context>_<pod-name>_YYYYMMDD_HHMMSS/
      pod_<pod-name>/
      node_<node-name>/
      aws_<node-name>/
      report.md
    sgfp_diag_YYYYMMDD_HHMMSS/
    sgfp_api_diag_YYYYMMDD_HHMMSS/
    sgfp_baseline_<label>_YYYYMMDD_HHMMSS/
    .sgfp_baseline_latest

reports/
  <cluster-context>/
    sgfp_bundle_<cluster-context>_<pod-name>_YYYYMMDD_HHMMSS.md

Benefits:

• Cluster identification: All data is tagged with the kubectl context, making it easy to identify which cluster the diagnostics belong to
• Organized structure: Data files are in data/, reports are in reports/
• Context in output: Console output and reports include the cluster context name

The cluster context is automatically detected from kubectl config current-context and sanitized for use in directory names.

Quick Start

Option 1: All-in-One (Recommended)

export AWS_REGION=us-west-2

# Run everything: collect, API diag, report, analyze, and display
# Output will be organized by kubectl context
./sgfp_doctor.sh <pod-name> -n default --minutes 60

# Mark a healthy pod and save as baseline
./sgfp_doctor.sh <pod-name> -n default --mark-healthy

# Mark an unhealthy pod for comparison
./sgfp_doctor.sh <pod-name> -n default --mark-unhealthy

Option 2: Step-by-Step

export AWS_REGION=us-west-2

# 1) Collect a bundle for a pod (namespace default)
# Output: data/<cluster-context>/sgfp_bundle_<cluster-context>_<pod-name>_YYYYMMDD_HHMMSS/
./sgfp_collect.sh -n default <pod-name>

# 2) (Optional) CloudTrail ENI API diagnostics for last 60 minutes
# Output: data/<cluster-context>/sgfp_api_diag_YYYYMMDD_HHMMSS/
WINDOW_MINUTES=60 ./sgfp_api_diag.sh

# 3) Generate a report for the bundle
# Output: data/<cluster-context>/sgfp_bundle_.../report.md
#        reports/<cluster-context>/sgfp_bundle_..._YYYYMMDD_HHMMSS.md
B=$(ls -dt data/*/sgfp_bundle_* 2>/dev/null | head -1)
./sgfp_report.sh "$B"

# 4) Post analyze the bundle
./sgfp_post_analyze.sh "$B"

Using Make Targets

# All-in-one
make doctor POD=<pod> NS=default

# Or step-by-step
make collect POD=<pod> NS=default
make api WINDOW_MINUTES=60
make report BUNDLE=<dir>
make analyze BUNDLE=<dir>

# Clean up all diagnostic output directories
make clean

What gets collected

Bundle structure (example, organized by cluster context):

data/<cluster-context>/sgfp_bundle_<cluster-context>_<pod>_<timestamp>/
  pod_<pod>/
    pod_annotations.json
    pod_conditions.json
    pod_ip.txt
    node_name.txt
    pod_netns_routes_rules.txt
    pod_reachability.txt
    pod_veth_interface.txt                 # veth interface name
    pod_interface_stats.txt                # Interface statistics with errors
    pod_sockstat.txt                       # Pod socket statistics
    pod_sockstat6.txt                      # Pod IPv6 socket statistics
    pod_snmp.txt                           # Pod socket overruns
    pod_timing.txt                         # Pod creation/start timestamps
    pod_events.txt                         # Pod events
    pod_full.json                          # Full pod JSON
    pod_container_statuses.json            # Container statuses
    pod_connections.txt                     # Pod network connections (listening ports and established)
    pod_conntrack_connections.txt          # Conntrack connections filtered by pod IP
    ipamd_introspection.json               # IPAMD introspection data
    ipamd_pool.json                        # IPAMD pool state
    ipamd_networkutils.json                # IPAMD network utils config
    aws_node_errors.log                    # Filtered aws-node errors
    pod_branch_eni_id.txt
    pod_branch_eni_describe.json
    pod_branch_eni_sgs.txt              # Actual SGs on pod ENI
    pod_branch_eni_sgs_details.json     # SG IDs, names, descriptions
    pod_branch_eni_sgs_rules.json      # Full SG rules (including IpPermissions for ingress/egress)
    pod_parent_trunk_eni.txt
    pod_expected_sgs.txt                 # Expected SGs from pod annotation
    deployment_expected_sgs.txt          # Expected SGs from deployment annotation
    replicaset_expected_sgs.txt          # Expected SGs from replicaset annotation
    namespace_expected_sgs.txt           # Expected SGs from namespace annotation
    pod_securitygrouppolicies.json       # SecurityGroupPolicy CRDs in pod's namespace
    pod_securitygrouppolicies_all.json  # All SecurityGroupPolicy CRDs (cluster-wide)
    deployment_annotations.json
    replicaset_annotations.json
    namespace_annotations.json
    aws_node_full.log
  node_<node>/
    node_conntrack_mtu.txt
    node_conntrack_table.txt              # Full conntrack table (for connection analysis)
    node_pod_ips.txt                      # All pod IPs on this node (for same-node identification)
    aws_node_full.log
    node_interface_dev_stats.txt          # Interface error statistics
    node_interface_ip_stats.txt            # ip -s link statistics
    node_sockstat.txt                      # Socket statistics
    node_sockstat6.txt                     # IPv6 socket statistics
    node_snmp.txt                          # Socket overruns
    node_netns_list.txt                    # List of network namespaces
    node_netns_details.json                # Network namespace details (interfaces, IPs, process count, timing)
    node_pod_ip_map.txt                    # Map of active pod IPv4 addresses to namespace/name
    node_pod_ipv6_map.txt                  # Map of active pod IPv6 addresses to namespace/name
    node_interfaces_state.txt              # All interface states
    node_all_ips.txt                       # All IP addresses on node
    node_duplicate_ips.txt                 # Duplicate IP addresses (if any)
    node_dns_tests.txt                     # DNS resolution tests
    node_coredns_pods.json                 # CoreDNS pod status
    node_nodelocal_dns_pods.json           # NodeLocal DNSCache pod status
    node_dns_service.json                  # DNS service configuration
    node_dns_endpoints.json                # DNS service endpoints
    node_nodelocal_dns_service.json        # NodeLocal DNSCache service (if enabled)
    node_coredns_config.json               # CoreDNS configuration
    node_pending_pods.json                 # Pods in Pending state (for IP exhaustion analysis)
    node_info.json                         # Node information (Kubernetes version, OS image, kernel)
    node_k8s_version.txt                   # Kubernetes version
    node_os_image.txt                      # OS image (AMI)
    node_kernel_version.txt                 # Kernel version
    node_container_runtime_version.txt     # Container runtime version
    node_aws_node_daemonset.json           # aws-node DaemonSet configuration
    node_aws_node_image.txt                # aws-node image
    node_aws_node_version.txt              # aws-node version
    node_kube_proxy_daemonset.json         # kube-proxy DaemonSet configuration
    node_kube_proxy_image.txt              # kube-proxy image
    node_kube_proxy_version.txt            # kube-proxy version
    node_labels.json                       # Node labels
    node_file_descriptors.txt              # File descriptor usage
    node_memory_info.txt                   # Memory information
    node_k8s_networkpolicies.json          # Kubernetes NetworkPolicies
    node_eniconfigs.json                   # ENIConfig resources (custom networking)
    node_annotations.json                  # Node annotations (may contain ENIConfig references)
    node_calico_networkpolicies.yaml       # Calico network policies (if Calico)
    node_bpf_programs.txt                  # eBPF programs (if Cilium)
    node_dmesg_network.txt                 # Network-related kernel messages
    node_arp_table.txt                     # ARP table
    node_iptables_filter.txt               # iptables filter rules
    node_iptables_nat.txt                  # iptables NAT rules
    node_routes_all.txt                    # Route table (all tables)
    node_veth_interfaces.txt               # veth interfaces
    node_syslog_network.txt                # Network-related syslog entries
    node_rp_filter.txt                     # Reverse path filtering (rp_filter) settings per interface
    cni_logs/                              # AWS VPC CNI logs
      ipamd.log
      plugin.log
      network-policy-agent.log
      ebpf-sdk.log
      egress-v6-plugin.log
      ipamd-latest-rotated.log
      *.errors                              # Error summaries for each log
  aws_<node>/
    vpc_id.txt
    node_instance_id.txt
    node_instance_type.txt                # Instance type (e.g., m8gd.2xlarge)
    trunk_eni_id.txt
    trunk_eni.json
    all_instance_enis.json
    _all_branch_enis_in_vpc.json         # best-effort, may be empty
    subnets.json                          # Subnet IP availability
    nat_gateways.json                     # NAT gateway information
    nat_<nat-id>_metrics.json            # CloudWatch metrics for each NAT gateway
  metrics_comparison.txt                  # Metrics comparison results (if baseline comparison enabled)
  incident_baseline/                      # Incident baseline metrics snapshot (if baseline comparison enabled)
    timestamp.txt
    kubectl_top_pods.txt
    kubectl_top_nodes.txt
    coredns_<pod>_metrics.txt
    aws_node_<pod>_metrics.txt
    kube_proxy_<pod>_metrics.txt
    kubelet_<node>_metrics.txt
    cluster_pods.json
    cluster_nodes.json
    cluster_services.json
    pod_status_summary.json
    pods_with_pod_eni.txt
    pending_pods_count.txt
  report.md                              # generated by sgfp_report.sh (includes metrics comparison if available)

CloudTrail API diag folder:

sgfp_api_diag_<timestamp>/
  events_eni.json
  flat_events.json
  eni_errors.tsv                        # Real errors/throttles (excludes dry-runs)
  eni_dryruns.tsv                        # Dry-run operations (informational)
  eni_all_issues.tsv                     # All events with error codes
  error_codes_summary.txt                # Summary of all error codes
  throttle_by_action.txt
  throttle_by_caller.txt
  calls_by_user.txt                       # API calls by user/caller ARN
  top_api_calls.txt

Scripts

sgfp_doctor.sh - All-in-One Orchestrator

Runs all diagnostics in sequence: collect → API diag → report → analyze → display report.

Baseline Comparison: If SGFP_BASELINE_DIR is set, the doctor script automatically captures an incident baseline snapshot and compares metrics with the baseline.

Health Status Marking: Use --mark-healthy to automatically save the collection as a healthy baseline, or --mark-unhealthy to tag an unhealthy pod for later comparison.

# With baseline comparison
export SGFP_BASELINE_DIR=sgfp_baseline_morning_20251109_080000
./sgfp_doctor.sh <pod> -n <namespace> [--minutes N] [--days D] [--region R] [--skip-api] [--api-dir DIR]

# Mark healthy pod and save as baseline
./sgfp_doctor.sh <pod> -n <namespace> --mark-healthy

# Mark unhealthy pod for comparison
./sgfp_doctor.sh <pod> -n <namespace> --mark-unhealthy

# Without baseline comparison
./sgfp_doctor.sh <pod> -n <namespace> [--minutes N] [--days D] [--region R] [--skip-api] [--api-dir DIR]

sgfp_collect.sh - Collection Orchestrator

Collects pod, node, and AWS diagnostics into a bundle.

Health Status Marking: Use --mark-healthy to automatically save as a healthy baseline, or --mark-unhealthy to tag the collection.

./sgfp_collect.sh -n <namespace> <pod-name>

# Mark healthy and save as baseline
./sgfp_collect.sh -n <namespace> --mark-healthy <pod-name>

# Mark unhealthy for comparison
./sgfp_collect.sh -n <namespace> --mark-unhealthy <pod-name>

Features:

• Automatically detects available shell in pod (sh, /bin/sh, /bin/bash, bash)
• Gracefully handles missing network tools (ip, ping)
• Collects Security Groups from pod ENI via AWS API
• Collects expected SGs from pod, deployment, replicaset, and namespace annotations
• Traverses Kubernetes owner references (Pod → ReplicaSet → Deployment)

sgfp_pod_diag.sh - Pod Diagnostics

Collects pod-specific information including annotations, conditions, network namespace routes/rules.

sgfp_node_diag.sh - Node Diagnostics

Collects node-level diagnostics: conntrack usage, interface error statistics, socket overruns, and AWS VPC CNI logs.

Features:

• Automatic CNI Log Collection: Collects CNI logs from /var/log/aws-routed-eni/ via temporary debug pod when not running on node (pod is automatically cleaned up)
• Conntrack Collection: Collects full conntrack table via temporary pod if needed (for connection analysis)
• Pod IP Collection: Collects all pod IPs on the node (for same-node vs cross-node connection identification)
• Collects interface error statistics from /proc/net/dev and ip -s link
• Collects socket statistics including overruns from /proc/net/sockstat and /proc/net/snmp
• Creates error summaries for each CNI log file
• Enhanced orphaned namespace detection: Uses IP-based matching to accurately identify truly orphaned network namespaces by:
  • Collecting actual IP addresses (IPv4 and IPv6) from each network namespace
  • Creating a map of all active pod IPs to namespace/name identifiers
  • Matching namespace IPs against active pod IPs (only flags as orphaned if no matching pod found, no processes, and older than 1 hour)
• Detects IP address conflicts (duplicate IPs on node)
• Tests DNS resolution (Kubernetes DNS, metadata service)
• Collects CoreDNS pod status and configuration
• Collects NodeLocal DNSCache pod status (if enabled)
• Collects DNS service endpoints and configuration
• Checks for resource exhaustion (file descriptors, memory pressure)
• Collects network policy rules (Kubernetes and CNI-specific: Calico, Cilium)
• Checks network interface states (interfaces in unexpected DOWN state)
• Collects kernel logs (dmesg), ARP table, iptables rules (filter and NAT), and route tables (all tables)

sgfp_aws_diag.sh - AWS ENI Diagnostics

Collects AWS ENI information: instance ID, VPC ID, trunk ENI, branch ENIs.

sgfp_api_diag.sh - CloudTrail API Diagnostics

Analyzes CloudTrail events for ENI-related API calls, throttles, and errors.

WINDOW_MINUTES=60 ./sgfp_api_diag.sh

Features:

• Distinguishes real errors/throttles from dry-run operations
• Categorizes events: eni_errors.tsv (real issues), eni_dryruns.tsv (informational)
• Provides summaries by action, caller, and error codes

sgfp_report.sh - Report Generator

Generates a markdown report from the collected bundle.

Features:

• Shows Security Group IDs, names, and descriptions
• Validates actual SGs against expected SGs (from annotations)
• Network Connections:
  • Pod network connections (listening ports and established connections from pod's perspective)
  • SYN_SENT Detection: Automatically detects and reports connections in SYN_SENT state (pod sending SYN but waiting for ACK - indicates connectivity issues)
  • Shows destination IPs and ports that the pod is trying to connect to but cannot complete
  • Conntrack connections with direction labels (INBOUND/OUTBOUND)
  • Same-node vs cross-node vs external connection identification
  • Connection states (ESTABLISHED, CLOSE, TIME_WAIT, etc.)
• iptables Rules:
  • Shows iptables filter and NAT table summaries (chain and rule counts)
  • kube-proxy Mode Detection: Identifies kube-proxy mode (iptables vs IPVS) and validates kube-proxy chains
  • KUBE-SERVICES Chain Validation: Checks if kube-proxy chains are active and processing traffic
  • Masquerade Rules: Validates masquerade rules are present (required for service traffic)
  • Pod-Specific Rule Detection: Automatically searches for iptables rules matching the pod's IP address and veth interface
  • Displays example rules when found (e.g., Kubernetes service DNAT rules)
  • Notes when no pod-specific rules are found (may be normal if no network policies apply)
• MTU Configuration:
  • Extracts MTU values from node interfaces (excluding loopback)
  • Compares pod interface MTU with node interface MTU
  • Warns about MTU mismatches that may cause fragmentation
  • Checks kernel logs for fragmentation-related messages
  • Reports standard (1500) or jumbo (9001) frame configuration
• Reverse Path Filtering (rp_filter):
  • Collects rp_filter settings for all interfaces
  • For pod ENI scenarios, recommends rp_filter=2 (loose mode) to allow asymmetric routing
  • Flags rp_filter=1 (strict mode) as a potential issue for pod ENI
  • Warns about rp_filter=0 (disabled) as a security risk
• Route Table Drift:
  • Validates default route (0.0.0.0/0) presence
  • Checks local subnet routes
  • Verifies metadata service route (169.254.169.254)
  • Compares VPC subnet routes (if available)
  • Counts total routes and flags anomalies
• Health Probes:
  • Extracts liveness, readiness, and startup probe configurations
  • Verifies probe ports are listening
  • Checks pod Ready and ContainersReady conditions
  • Searches pod events for probe failures
  • Validates NetworkPolicies and node Security Groups allow probe traffic
• NetworkPolicies:
  • Finds NetworkPolicies that apply to the pod (namespace and podSelector matching)
  • Checks if ingress rules allow health probes from nodes
  • Validates DNS egress rules (port 53)
  • Checks for metrics egress rules
  • Warns about restrictive policies that may block required traffic
• ENI/Instance Limits:
  • Shows instance type and ENI/IP limits
  • Compares current ENI usage vs instance limits
  • Validates trunk ENI branch ENI count (approaching 50 limit)
  • Estimates max pods capacity (without trunking)
  • Warns when approaching or at limits
• Log Files Summary: Concise list of all log files with error counts and file paths
• Node CNI Logs: Shows CNI log errors with recent examples in the Node State section
• View Related Logs: Provides helper script commands to view pod-specific log lines
• Uses consistent [OK], [ISSUE], [INFO] format

sgfp_post_analyze.sh - Quick Analysis

Provides a quick summary of potential issues found in the bundle.

Features:

• Validates Security Groups (actual vs expected)
• Checks pod status, readiness gates, routing tables
• Uses consistent [OK], [ISSUE], [INFO] format

sgfp_analyze_connectivity.sh - Connectivity Analysis

Advanced analysis for diagnosing pod connectivity issues, especially after large pod churns.

Features:

• Analyzes ENI attachment state and timing
• Route Table Drift Analysis: Validates default route, subnet routes, metadata service route, and route consistency
• Health Probe Analysis: Analyzes probe configurations, verifies probe ports are listening, checks for probe failures, and validates network access
• NetworkPolicy Analysis: Analyzes NetworkPolicies to detect if they block DNS, health probes, metrics, or service traffic
• DNS / CoreDNS / NodeLocal DNSCache Analysis: Analyzes CoreDNS pod status and scaling, NodeLocal DNSCache presence, DNS service endpoints, and DNS resolution
• AMI / CNI / Kernel Drift Analysis: Detects version mismatches between Kubernetes, kube-proxy, aws-node, OS image (AMI), and kernel versions, flags non-EKS-optimized AMIs, and identifies outdated components
• ENI/Instance Limits Analysis: Checks instance type ENI/IP limits, current usage, trunk ENI branch ENI count, and estimates max pods
• Checks IPAMD state and branch ENI limits
• Validates subnet IP availability
• Analyzes pod events for network-related issues
• Analyzes CNI logs (both aws-node and node-level CNI logs)
• Checks readiness gate timing
• Detects truly orphaned network namespaces using IP-based matching (matches namespace IPs against active pod IPs)
• Detects IP address conflicts
• Tests DNS resolution
• Checks for resource exhaustion (file descriptors, memory pressure)
• Validates network interface states
• MTU Configuration Analysis: Detects MTU mismatches, fragmentation issues, and validates MTU consistency
• kube-proxy iptables Analysis: Validates kube-proxy mode, chain activity, and masquerade rules
• Reverse Path Filtering (rp_filter) Analysis: Validates rp_filter settings for pod ENI scenarios and flags security risks

sgfp_node_debug.sh - Node Debug Pod

Creates a debug pod on a node for interactive troubleshooting.

# Debug node where a pod is running
./sgfp_node_debug.sh <pod-name> -n <namespace>

# Debug node directly
./sgfp_node_debug.sh <node-name>

# With custom image
./sgfp_node_debug.sh <pod-name> -n <namespace> <image>

Features:

• Automatically detects if argument is a pod name or node name
• Uses kubectl debug node/ for proper node debugging
• Defaults to ubuntu image

sgfp_view_logs.sh - View Related Logs

Helper script to view pod-specific log lines from a diagnostic bundle.

# View all pod-related log lines
./sgfp_view_logs.sh <bundle-dir>

# View only errors/warnings
./sgfp_view_logs.sh <bundle-dir> --errors-only

# View all log lines (not filtered)
./sgfp_view_logs.sh <bundle-dir> --all-logs

Features:

• Automatically extracts pod identifiers (pod name, container ID, ENI ID, IP, UID) from bundle
• Searches all log files (aws-node logs, CNI logs) for pod-related lines
• Three modes: default (pod-related), errors-only, or all-logs
• Shows which search patterns are being used

sgfp_quick_check.sh - Quick Status Check

Quick validation script for pod ENI status without full diagnostic collection.

# Check pod in default namespace
./sgfp_quick_check.sh <pod-name>

# Check pod in specific namespace
./sgfp_quick_check.sh -n <namespace> <pod-name>

# Show network connections (pod-level and node-level conntrack)
./sgfp_quick_check.sh -c <pod-name>
./sgfp_quick_check.sh --connections -n <namespace> <pod-name>

Features:

• Validates pod ENI annotation is present and correct
• Compares pod IP with CNI status IP
• Shows branch ENI ID and trunk association ID
• Displays requested vs actual Security Groups with names and descriptions
• Network Connections (with --connections or -c flag):
  • Pod network connections (listening ports and established connections from pod's perspective)
  • SYN_SENT Detection: Automatically detects and warns about connections in SYN_SENT state (pod sending SYN but waiting for ACK - indicates connectivity issues)
  • Shows destination IPs and ports that the pod is trying to connect to but cannot complete
  • Conntrack connections (node-level, filtered by pod IP) showing both inbound TO pod and outbound FROM pod
  • Inbound/outbound connection counts
  • Same-node vs cross-node vs external connection identification
  • Connection states (ESTABLISHED, CLOSE, TIME_WAIT, etc.)
  • Automatically creates temporary privileged pod for conntrack access if needed
• Provides quick PASS/WARN result
• No diagnostic collection required (fast check)

sgfp_baseline_capture.sh - Baseline Metrics Capture

Captures a snapshot of cluster metrics for comparison during incidents. Run this periodically (e.g., each morning) or before deployments to establish a baseline.

# Capture baseline with optional label
./sgfp_baseline_capture.sh [--label <label>]

# Examples:
./sgfp_baseline_capture.sh
./sgfp_baseline_capture.sh --label morning
./sgfp_baseline_capture.sh --label pre-deploy-v1.2.3

What it collects:

• kubectl top pods - CPU and memory usage for all pods
• kubectl top nodes - CPU and memory usage for all nodes
• CoreDNS metrics - Prometheus metrics from CoreDNS pods (port 9153)
• aws-node metrics - VPC CNI metrics from aws-node pods (port 61678)
• kube-proxy metrics - kube-proxy metrics (port 10249)
• kubelet metrics - Node metrics from kubelet (port 10250/10255, via proxy or direct)
• Cluster state summary - Pod counts, node counts, service counts, pod status summary
• Network metrics summary - Pods with pod ENI, pending pods count

Output:

sgfp_baseline_<label>_<timestamp>/
  timestamp.txt
  kubectl_top_pods.txt
  kubectl_top_nodes.txt
  coredns_<pod>_metrics.txt
  aws_node_<pod>_metrics.txt
  kube_proxy_<pod>_metrics.txt
  kubelet_<node>_metrics.txt
  cluster_pods.json
  cluster_nodes.json
  cluster_services.json
  pod_status_summary.json
  pods_with_pod_eni.txt
  pending_pods_count.txt

Usage workflow:

1. Capture baseline (e.g., each morning or before deployment):

   make baseline LABEL=morning
   # or
   ./sgfp_baseline_capture.sh --label pre-deploy

   The script automatically exports SGFP_BASELINE_DIR and saves it to .sgfp_baseline_latest for easy reference.

2. During incident, collect diagnostics with automatic metrics comparison:

   # Set baseline directory (or use saved .sgfp_baseline_latest)
   export SGFP_BASELINE_DIR=sgfp_baseline_morning_20251109_080000
   
   # Run doctor - it will automatically:
   # - Capture incident baseline snapshot
   # - Compare metrics with baseline
   # - Include comparison in report
   make doctor POD=<pod> NS=default

   The doctor script automatically:

   • Captures an incident baseline snapshot (sgfp_baseline_incident_<timestamp>/)
   • Runs metrics comparison analysis
   • Regenerates the report to include metrics comparison section
   • Saves comparison results to metrics_comparison.txt in the bundle
   • Copies incident baseline metrics to incident_baseline/ in the bundle

3. Manual comparison (if needed separately):

   # Option 1: Use saved baseline (from .sgfp_baseline_latest)
   make compare-baseline INCIDENT=sgfp_bundle_<pod>_<timestamp>
   
   # Option 2: Use environment variable
   export SGFP_BASELINE_DIR=sgfp_baseline_morning_20251109_080000
   make compare-baseline INCIDENT=sgfp_bundle_<pod>_<timestamp>
   
   # Option 3: Specify baseline explicitly
   make compare-baseline BASELINE=sgfp_baseline_morning_20251109_080000 INCIDENT=sgfp_bundle_<pod>_<timestamp>

Note: Metrics collection uses kubectl exec when possible, with fallback to kubectl port-forward for pods that don't have wget/curl available. Some metrics may not be accessible depending on cluster configuration (e.g., kubelet metrics require kubectl proxy or direct node access).

sgfp_compare_baseline.sh - Baseline Comparison

Compares baseline metrics with incident/diagnostic bundle to identify changes.

# Option 1: Use saved baseline (from .sgfp_baseline_latest)
./sgfp_compare_baseline.sh <incident-bundle-dir>

# Option 2: Use environment variable
export SGFP_BASELINE_DIR=<baseline-dir>
./sgfp_compare_baseline.sh <incident-bundle-dir>

# Option 3: Specify baseline explicitly
./sgfp_compare_baseline.sh <baseline-dir> <incident-bundle-dir>

The script automatically uses the baseline directory in this priority order:

1. Command argument (if provided)
2. SGFP_BASELINE_DIR environment variable
3. .sgfp_baseline_latest file (created by baseline capture script)

What it compares:

• Pod and node resource usage (CPU/memory)
• Total pod counts
• Pending pod counts (flags increases as IP exhaustion indicator)
• Pods with pod ENI counts
• CoreDNS metrics: DNS request rate, SERVFAIL errors, cache hit rate
• aws-node metrics: ENI/IP/Branch ENI allocation failures
• kube-proxy metrics: iptables rule count, sync duration
• Detailed metrics files (for manual comparison)

Note: The doctor script automatically performs this comparison when SGFP_BASELINE_DIR is set, and includes results in the generated report under "Metrics Comparison (Baseline vs Incident)" section.

Security Group Validation

The toolkit automatically validates Security Groups by:

1. Collecting actual SGs from the pod's ENI via AWS API
2. Collecting expected SGs from Kubernetes annotations (in priority order):
   • Pod annotation: vpc.amazonaws.com/security-groups
   • Deployment annotation: vpc.amazonaws.com/security-groups
   • ReplicaSet annotation: vpc.amazonaws.com/security-groups
   • Namespace annotation: vpc.amazonaws.com/security-groups
3. Collecting SecurityGroupPolicy CRDs that match the pod (via podSelector)
4. Validating SecurityGroupPolicy compliance (max 5 SGs per policy, up to 10 total across multiple policies)
5. Comparing actual vs expected and reporting mismatches

The report shows:

• Actual SGs with names and descriptions
• Expected SGs (if specified)
• Validation status: Match, Mismatch, or No expected SGs specified

Security Group Rules Analysis

The toolkit includes scripts to analyze security group rules for cross-node traffic:

sgfp_check_sg_rules_for_cross_node.sh - Check Security Group Rules for Cross-Node Traffic

Analyzes security group ingress rules to verify if cross-node source IPs are allowed for a specific port.

./sgfp_check_sg_rules_for_cross_node.sh <bundle-dir>

What it does:

• Extracts allowed security groups for a specific port (e.g., port 6000) from the pod's ENI security group rules
• Identifies source IPs attempting to connect (from conntrack data)
• Checks if source IPs are allowed by CIDR ranges or security group references
• Reports which source IPs are blocked and which are allowed

Output:

• Lists allowed security groups for the port
• Shows source IPs and whether they're allowed or blocked
• Provides recommendations for fixing blocked traffic

sgfp_check_source_pod_sgs.sh - Check Source Pod Security Groups

Checks security groups of source pods attempting to connect to identify mismatches with allowed security groups.

./sgfp_check_source_pod_sgs.sh <bundle-dir>

What it does:

• Extracts allowed security groups for the target port from security group rules
• Identifies source pods attempting to connect (from conntrack data)
• Retrieves security groups from source pods (pod ENI or node security groups)
• Compares source pod security groups against allowed list
• Reports which source pods have matching security groups and which don't

Output:

• Lists allowed security groups for the port
• Shows each source pod and its security groups
• Indicates if source pod security groups match the allowed list
• Provides recommendations for fixing mismatches

Note: If source pods have been deleted/recreated since diagnostic collection, the script may not be able to retrieve their security groups. Run diagnostics while source pods are active for accurate results.

Cluster Pod Snapshot

Each diagnostic bundle now includes a comprehensive cluster-wide pod snapshot (cluster_pod_snapshot.json) captured at the time of collection. This snapshot includes:

• Pod IPs: IPv4, IPv6, and all IPs for each pod
• Pod Metadata: Namespace, name, UID, node, phase, labels
• Pod ENI Information: ENI ID, private IP, security groups
• Pod Status: Ready condition, container IDs, timestamps
• Owner References: Deployment/ReplicaSet information

This snapshot is useful for:

• Packet Capture Analysis: Map IPs in packet captures to pods even after pods have changed
• Historical Reference: See what pods existed at the time of the diagnostic
• Cross-Referencing: Link IPs, ENIs, and other identifiers across different diagnostic data

Querying the Pod Snapshot

Use the sgfp_query_pod_snapshot.sh helper script to query the snapshot:

# Show snapshot metadata
./sgfp_query_pod_snapshot.sh <bundle-dir>

# Find pod by IP address
./sgfp_query_pod_snapshot.sh <bundle-dir> ip 10.4.243.90

# Find pods by name pattern
./sgfp_query_pod_snapshot.sh <bundle-dir> name be-innkeeper

# Find pod by ENI ID
./sgfp_query_pod_snapshot.sh <bundle-dir> eni eni-081829d313cc36576

# List all pods
./sgfp_query_pod_snapshot.sh <bundle-dir> all

The snapshot file is located at: <bundle-dir>/cluster_pod_snapshot.json

Investigation Tracking

Structured investigation documents help track facts, unknowns, theories, and conclusions for pod connectivity issues. These documents live in the investigations/ directory and serve as a knowledge base for diagnosing problems.

Investigation Document Structure

Each investigation document tracks:

• Known Facts: Things we know are working/correct vs. broken/incorrect
• Unknowns: Things we need to check or verify
• Theories & Hypotheses: Possible explanations with supporting/contradicting evidence
• Observations: Data points from diagnostic reports
• Conclusions: What we've determined from analysis
• Data Sources: References to diagnostic bundles and reports

Creating/Updating Investigations

# Create a new investigation from template
cp investigations/template.md investigations/my-investigation.md

# Update investigation with new diagnostic data
./sgfp_update_investigation.sh investigations/my-investigation.md <bundle-dir> --status unhealthy

Main Investigation Document

• investigations/pod-connectivity-issues.md - Main investigation for pod connectivity problems
  • Tracks the "restart fixes it" issue
  • Documents theories about incomplete network namespace setup
  • References diagnostic bundles and analysis documents

Investigation Template

See investigations/template.md for a structured template you can copy for new investigations.

Checking Known Issues

Use sgfp_check_known_issues.sh to check diagnostic bundles against known issues from GitHub, changelogs, and security advisories:

# Check a diagnostic bundle against known issues
./sgfp_check_known_issues.sh <bundle-dir>

What it checks:

• Component versions (Kubernetes, OS, kernel, aws-node, kube-proxy, CoreDNS)
• Known GitHub issues (amazon-vpc-cni-k8s, amazon-linux-2023, kubernetes, coredns)
• Security advisories (ALAS)
• Version compatibility (e.g., kube-proxy must match Kubernetes version)
• Pattern matching against known issue symptoms
• Version-specific known problems

Output includes:

• Detected component versions
• Relevant known issues with references
• Pattern matches in diagnostic data
• Recommendations for next steps

Packet Capture Analysis

The toolkit includes scripts to analyze packet capture files and capture network traffic:

sgfp_analyze_pcap.sh - Analyze Packet Capture Files

Analyzes tcpdump/pcap files with pod IP mapping to identify communicating pods and traffic patterns.

./sgfp_analyze_pcap.sh <capture-file> [bundle-dir]

What it does:

• Analyzes packet capture files (text format from tcpdump or binary .pcap files)
• Maps IP addresses to pod names using diagnostic bundle data
• Identifies source and destination pods
• Analyzes traffic patterns, connection states, and errors
• Provides pod-specific analysis for the target pod

Output:

• Protocol breakdown
• Top source and destination IPs (with pod mapping)
• Connection state analysis
• Error analysis (SYN, RST, etc.)
• Pod-specific traffic analysis

sgfp_analyze_pcap_with_pod_mapping.sh - Analyze Packet Capture with Pod Mapping

Enhanced packet capture analysis with detailed pod IP mapping and local vs remote traffic breakdown.

./sgfp_analyze_pcap_with_pod_mapping.sh <capture-file> <bundle-dir>

What it does:

• Extracts all IP addresses from packet capture
• Maps IPs to pod names using diagnostic bundle
• Identifies local (same node) vs remote (cross-node) traffic
• Provides summary of communicating pods
• Analyzes traffic patterns by pod

Output:

• Summary of communicating pods
• Local vs remote traffic breakdown
• Pod-to-pod communication matrix

sgfp_pod_tcpdump.sh - Capture Network Traffic from Pod

Automated script to capture network traffic from a pod's network namespace using tcpdump.

./sgfp_pod_tcpdump.sh <pod-name> [namespace] [tcpdump-args]

What it does:

• Finds the node where the pod is running
• Creates a debug pod on the same node with sysadmin profile
• Extracts network namespace name from diagnostic reports (if available)
• Provides instructions for running tcpdump in the pod's network namespace
• Automatically installs tcpdump if needed

Usage:

# Basic capture (port 6000)
./sgfp_pod_tcpdump.sh be-conductor default

# Custom tcpdump arguments
./sgfp_pod_tcpdump.sh be-conductor default "-i any -n -v port 6000"

# Capture all traffic
./sgfp_pod_tcpdump.sh be-conductor default "-i any -n -v"

Output:

• Creates a debug pod on the target node
• Provides commands to enter the pod's network namespace
• Instructions for installing tcpdump if needed
• Commands to capture traffic from within the pod's network namespace or on the veth interface

Note: The sgfp_doctor.sh script automatically outputs the tcpdump command for the pod at the end of its run.

Output Format

All scripts use a consistent [PREFIX] output format:

• [OK] - Successful check
• [ISSUE] - Problem detected
• [INFO] - Informational message
• [WARN] - Warning
• [ERROR] - Error condition

Script-specific prefixes:

• [DOCTOR] - Doctor script
• [NODE] - Node diagnostics
• [AWS] - AWS diagnostics
• [API] - API diagnostics
• [ANALYZE] - Post-analyze script
• [REPORT] - Report generator

Make Targets

make collect POD=<pod> NS=default      # Collect diagnostics
make api WINDOW_MINUTES=60             # API diagnostics
make report BUNDLE=<dir>               # Generate report
make analyze BUNDLE=<dir>              # Post-analyze
make analyze-connectivity BUNDLE=<dir> # Connectivity analysis
make doctor POD=<pod> NS=default       # All-in-one
make node-debug TARGET=<pod|node>      # Create debug pod on node
make view-logs BUNDLE=<dir>            # View pod-related log lines
make quick-check POD=<pod> NS=default  # Quick pod ENI status check
make baseline [LABEL=<label>]         # Capture baseline metrics snapshot
make compare-baseline INCIDENT=<dir> [BASELINE=<dir>]  # Compare baseline vs incident (uses saved baseline if not specified)
make check-sg-rules BUNDLE=<dir>      # Check security group rules for cross-node traffic
make check-source-sgs BUNDLE=<dir>    # Check source pod security groups
make analyze-pcap FILE=<capture> [BUNDLE=<dir>]  # Analyze packet capture file
make pod-tcpdump POD=<pod> [NS=default] [ARGS="..."]  # Capture network traffic from pod
make clean                             # Remove all diagnostic output directories
make clean-debug-pods NS=<namespace>   # Clean up debug pods interactively

Notes

• The collectors are best-effort. Missing permissions or components are handled gracefully; files still get created (possibly empty) so later steps won't crash.
• Shell Detection: Scripts automatically detect available shells in pods (sh, /bin/sh, /bin/bash, bash) for better compatibility.
• Network Tools: Missing network tools (ip, ping) in pods are handled gracefully with informative messages.
• ICMP Reachability: ICMP may be blocked; pod_reachability.txt is informational only.
• Security Groups: SG names and descriptions require ec2:DescribeSecurityGroups permission. The toolkit automatically fetches this information when available.
• Dry-Run Operations: API diagnostics distinguish between real errors/throttles and successful dry-run validations.
• CNI Log Collection: When node diagnostics are run, the toolkit automatically creates a temporary privileged pod on the node to collect CNI logs from /var/log/aws-routed-eni/ and conntrack data. The pod is automatically cleaned up after collection.
• Connection Analysis:
  • Pod connections show listening ports and established connections from the pod's perspective
  • Conntrack connections show both directions (INBOUND TO pod and OUTBOUND FROM pod) with connection states
  • Connections are identified as same-node (pod on same node), cross-node (pod on different node in VPC), or external (outside VPC)
  • This helps diagnose if connectivity issues are local to the node or cross-node networking problems
• Log Files Summary: The report includes a concise summary of all log files with error counts and file paths, making it easy to identify which logs need attention
• View Related Logs Helper: The sgfp_view_logs.sh script automatically extracts pod identifiers and searches all log files for pod-related lines, with options to view only errors or all logs
• Enhanced Orphaned Namespace Detection: Uses IP-based matching to accurately identify truly orphaned network namespaces by collecting actual IP addresses from namespaces and matching them against active pod IPs (eliminates false positives from interface-count-only checks)
• Process Counting: Counts processes within each network namespace for additional context when determining if a namespace is truly orphaned
• Node Debug Pod: The sgfp_node_debug.sh script can accept either a pod name (will find the node) or a node name directly.
• Output Directories: All diagnostic output directories (data/, reports/, and legacy sgfp_bundle_*, sgfp_diag_*, sgfp_api_diag_*) are automatically ignored by git (see .gitignore).

Requirements

• Linux
• kubectl, jq, awk, grep
• aws CLI configured (and AWS_REGION set, e.g., export AWS_REGION=us-west-2)
• Permissions:
  • ec2:DescribeNetworkInterfaces (required)
  • ec2:DescribeSecurityGroups (for SG names/descriptions)
  • CloudTrail lookup-events (optional, for API diagnostics)