`collectlogs` script may capture empty ORC controller pod log

[mandre]

Problem description

Description

In the flamingo E2E run for PR #835, the hack/collectlogs script produced a 0-byte orc-pod.log file. The ORC controller pod was running, healthy (Ready, 0 restarts), and had been operating successfully for ~12 minutes at zap-log-level=5 running 73 tests. There should have been substantial log output.

Evidence

The kubectl logs command completed in ~60ms — impossibly fast for a controller that ran 12 minutes at debug verbosity. No error was emitted on stderr:

14:42:57.7049368Z + kubectl logs -n orc-system -l control-plane=controller-manager --tail=-1
14:42:57.7644628Z + kubectl get -n orc-system all -o yaml

The pod was confirmed running with the correct label (control-plane=controller-manager), the disk had space, and memory was available. Despite this, the resulting orc-pod.log file is 0 bytes.

Cause

Unknown. Possible explanations include:

• A kubelet or containerd log retrieval issue specific to Kind with Kubernetes v1.35.0
• Container log rotation dropping all accessible logs (the default containerLogMaxSize is 10Mi, which may be exceeded with zap-log-level=5 and 73 concurrent tests)
• A transient kubectl/API server issue fetching logs from the Kind node

Impact

Without controller logs, the CI failure investigation had to be reconstructed entirely from Neutron-side logs and source code analysis. Having the ORC controller logs would have immediately revealed the terminal error on the Router resource.

Suggested Improvements

Make log collection more robust:

# Capture pod name explicitly
ORC_POD=$(kubectl get pods -n orc-system -l control-plane=controller-manager -o jsonpath='{.items[0].metadata.name}')

# Capture logs with error output, and add a fallback
kubectl logs -n orc-system "$ORC_POD" --tail=-1 > "$LOG_DIR/orc-pod.log" 2>&1

# Warn if the log file is unexpectedly empty
if [ ! -s "$LOG_DIR/orc-pod.log" ]; then
    echo "WARNING: orc-pod.log is empty, trying --previous" >&2
    kubectl logs -n orc-system "$ORC_POD" --previous > "$LOG_DIR/orc-pod-previous.log" 2>&1 || true
fi

This adds:

1. Explicit pod name instead of label selector (avoids any selector-related issues)
2. Captures stderr from kubectl logs into the file so any error is preserved
3. Detects empty log files and tries --previous as a fallback
4. Produces a warning so the empty log is visible in CI output

ORC version

main

Additional information

No response

Relevant log output