Repository navigation
Unbreaking Kpoof
It occurred to me while attending kubecon earlier this week that my last commit broke kpoof for a portion of the pods it would attempt to forward. While on a break from the conference, I used kpoof to diagnose DNS collisions for our MVP of farmotive.io. The last release for kpoof included a code refactor that enabled all exposed ports (rather than one) to be accessed locally; unfortunately, this introduced a breaking change in which a single exposed port for a pod defeated the remote enumeration when said port is less than 1001. While at Kubecon, in a rush to move past the block and identify our issue, I simply reverted to core, and rather than running kpoof, I used the default kubectl command for port forwarding: kubectl port-forward <pod-name> --namespace farmotive 8080:80. Our root cause for DNS collisions stemmed from having DNS entries in both domains.google and GKE's cloud-dns name service, which we'll address in a separate blog post.
This morning, before dawn, I confirmed my fear. In case I was running a local version of kpoof and had not rebased with master, some git fetch --all love and git rebase upstream/master master in my lypht/kpoof local fork confirmed that I wasn't in some odd branch that did not conform to the latest stable release. Validation with which kpoof also told me I wasn't aliased or symlinked to a broken build, as the result correctly pointed to /usr/local/bin/kpoof.
Sometimes not being in front of a computer can help with the debugging process. The image above is representative of the two fish I caught once the sun came up. The act of fishing allowed me to step out of the code on the screen and think logically about what could have broken the default behavior. For the interested, both fish came on a watermelon red fluke in 1-3 feet of water, steam broiling off of an otherwise still pond surface. The only true variance was complete shadow versus direct sunlight, although I missed another, bigger fish which was also in sun.
"What could have caused the break between version 1 and latest?" The impetus of the refactor was an issue of some unwanted concatenation when a pod's deployment object definition exposed multiple ports at once. Although the latest version addressed this to forward all exposed ports, I had not thought that it would in turn introduce a bug when a pod only exposed a single port.
When a user runs kpoof a prompt for a list of the cluster's accessible namespaces is returned. Once a namespace is selected, a subsequent list of pods within that namespace is returned. Once a pod is selected, all ports are attached locally. If any local ports are under 1001, kpoof is written to transform this to n+50000, where n is the port in question (as an example, local port 80, which is denied by default for port-forward, is transformed to 50080). This is in the code's comments and in the README.md, but for posterity, here's a good summary of the issue that necessitates this transform.
Using kpoof against our static web front-end written in hugo, yet served by an unadulterated nginx:alpine image, results in forwarding from 127.0.0.1:50080 -> 50080. As the local port is 80, this fails for obvious reasons.
Kpoof allows us to pass a specific port to forward. We can test the latest release by forcing a port, like so:
$ kpoof -p
Namespace? (default farmotive):
1 365bass
2 basekamp
3 default
4 farmotive
5 kuard
6 kube-public
7 kube-system
8 lypht
9 m1gr8
10 paw
11 r6by
12 rhin
13 whereapy
Pod number? (default 1):
1 affiliates-drupal-5579ff68cb-dm88v
2 affiliates-mariadb-989465b77-rzn6k
3 farm-ddog-datadog-6r6kt
4 farm-ddog-datadog-gg426
5 farm-ddog-datadog-wppjm
6 farm-ddog-kube-state-metrics-8598f99cf6-7g6jb
7 farmdb-mysql-481243574-st27x
8 site-farm-chart-55b64c76b-k7zng
9 site-farm-chart-55b64c76b-kxqkq
10 site-farm-chart-55b64c76b-tm9hg
9
Remote port number? (default 1):
1 80
Local port number? (defaults to 80):
8080
Forwarding from 127.0.0.1:8080 -> 80
It works with -p. So there must have been some new code added in the latest version that does not respect the transform snippet. Looking at the latest release, I found this:
LOCAL_CHECK=$(port_check ${LOCAL_PORT:-$(port_sniff)})
kubectl port-forward \
$(ns_param) $(po_select) $LOCAL_CHECK:${REMOTE_PORT:-$(port_sniff)}
}
po_pf_all() {
kubectl port-forward $(ns_param) $(po_select) $(port_x)
}
You can dig in to the code to see what's happening beneath the hood, but we're evaluating the local check (our unix port-range denial default) in po_pf which is now only evaluated if -p or --port=true, but I failed to implement a similar local port operator when refactoring to forward all exposed ports. But wait a second... If this is failing for a single port, it would stand to reason that it would fail for multiple exposed ports. So in my original (incorrect) assumption, the transform worked as expected for most use cases (multiple exposed ports from one pod) but only failed when a single port was exposed. An easy test proved the problem was greater than I had imagined: Running kpoof against a pod with multiple ports exposed below our denied range also failed to respect the remote ports:
Namespace? (default farmotive):
1 365bass
2 basekamp
3 default
4 farmotive
5 kuard
6 kube-public
7 kube-system
8 lypht
9 m1gr8
10 paw
11 r6by
12 rhin
13 whereapy
Pod number? (default 1):
1 affiliates-drupal-5579ff68cb-dm88v
2 affiliates-mariadb-989465b77-rzn6k
3 farm-ddog-datadog-6r6kt
4 farm-ddog-datadog-gg426
5 farm-ddog-datadog-wppjm
6 farm-ddog-kube-state-metrics-8598f99cf6-7g6jb
7 farmdb-mysql-481243574-st27x
8 site-farm-chart-55b64c76b-k7zng
9 site-farm-chart-55b64c76b-kxqkq
10 site-farm-chart-55b64c76b-tm9hg
1
Forwarding from 127.0.0.1:50443 -> 50443
Forwarding from 127.0.0.1:50080 -> 50080
I spent far more time coming up with the solution than I care to admit. I fished again this morning, with no luck. I'll try again before dark if I get a chance. But warmly back under blankets and resolving our issue, I was finally able to get my syntax correct to enable both single and multi port forwards with the following edit:
port_x() {
for i in $(port_list); do
echo $(port_check $i):${i}
done
}
Success! Testing both pods with a single exposed port and pods with multiple exposed ports, we see the desired behavior:
Forwarding from 127.0.0.1:50080 -> 80
and
Forwarding from 127.0.0.1:50443 -> 443
Forwarding from 127.0.0.1:50080 -> 80
respectively.
Testing is invaluable in understanding the issue, and perseverance with getting one's syntax correct when the fix is identified, although demoralizing with many iterations, ultimately results in the ecstasy of seeing one's code work as expected. I'll cut this minor release yet this morning, hopefully before the Chiefs beat the Raiders.