Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improving port forwarding error handling #1839

Merged
merged 2 commits into from
Mar 7, 2023
Merged

Improving port forwarding error handling #1839

merged 2 commits into from
Mar 7, 2023

Conversation

ramiro-gamarra
Copy link
Contributor

Reason for Change:

Port forwarding to a Kubernetes pod can be brittle but can be restarted if the appropriate signals are handled. This PR improves the API of the internal port forwarding wrapper PortForwarder to allow clients to maintain port forwarding sessions alive.

Example usage:

pfOpts := PortForwardingOpts{
    Namespace:     "default",
    LabelSelector: "type=goldpinger-pod",
    LocalPort:     9090,
    DestPort:      8080,
}

pf, _ := NewPortForwarder(restConfig, logger, pfOpts)

if err := pf.Forward(ctx); err != nil {
    // errors if initial connection fails
}

go pf.KeepAlive(ctx) // handles connection failures and re-attempts to port forward any pod matching the opts to the same local address

defer pf.Stop()

Notes:

}

// KeepAlive can be used to restart the port forwarding session in the background.
func (p *PortForwarder) KeepAlive(ctx context.Context) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reconsidering if we even need to export this method? Seems like callers would just want to forward and stop, not have to also enable keep alive explicitly?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agree

@timraymond
Copy link
Member

What is the problem that motivated these changes? They're good, but I just don't understand the wider context.

@ramiro-gamarra
Copy link
Contributor Author

What is the problem that motivated these changes? They're good, but I just don't understand the wider context.

@timraymond We leverage port forwarding in integration tests that deploy goldpinger to a kubernetes cluster, since the tests need to make assertions against responses from the API that goldpinger pods expose, and these endpoints are private to the cluster (tests are executed on a VM on a different network from the k8s cluster). Currently, if the port forwarding session dies, that error is not communicated to the ongoing test suite, and tests will eventually fail due to timeouts, leading to PR checks needing to be retried, possibly multiple times.

Copy link
Contributor

@rbtr rbtr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is worth taking as-is to immediately improve the CI reliability. Tweaks like maybe removing KeepAlive can come later 🙂

@ramiro-gamarra ramiro-gamarra merged commit 6303016 into Azure:master Mar 7, 2023
@ramiro-gamarra ramiro-gamarra deleted the port-forward-resiliency branch March 7, 2023 21:42
rjdenney pushed a commit that referenced this pull request Mar 13, 2023
* adding error handling hook to portforwarder. documenting exported symbols. removing unnecessary build constraints

* improving the port forward wrapper api
rbtr pushed a commit that referenced this pull request Sep 8, 2023
* adding error handling hook to portforwarder. documenting exported symbols. removing unnecessary build constraints

* improving the port forward wrapper api
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants