controller: Fix an issue wrt cleanup of stale patch port. #757

Job	Run time
dpdk gcc	9s
osx clang --disable-ssl	1m 58s
linux rpm fedora	6m 50s
Prepare container	1m 55s
linux gcc --disable-ssl	2m 26s
linux clang --disable-ssl	2m 26s
linux gcc test -500	14m 12s
linux gcc test 501-1000	10m 8s
linux gcc test 1001-	5m 52s
linux clang test sanitizers -300	11m 19s
linux clang test sanitizers 301-600	14m 42s
linux clang test sanitizers 601-900	9m 2s
linux clang test sanitizers 901-1200	7m 28s
linux clang test sanitizers 1201-	7m 4s
linux gcc test -ljemalloc -500	14m 53s
linux gcc test -ljemalloc 501-1000	11m 8s
linux gcc test -ljemalloc 1001-	5m 53s
linux gcc system-test-dpdk dpdk -100	11m 0s
linux gcc system-test-dpdk dpdk 101-200	17m 37s
linux gcc system-test-dpdk dpdk 201-	18m 47s
linux gcc system-test-userspace -100	9m 46s
linux gcc system-test-userspace 101-200	16m 1s
linux gcc system-test-userspace 201-	17m 44s
linux gcc system-test -100	12m 52s
linux gcc system-test 101-200	18m 39s
linux gcc system-test 201-	24m 16s
linux clang system-test sanitizers -100	14m 30s
linux clang system-test sanitizers 101-200	20m 3s
linux clang system-test sanitizers 201-	26m 40s
linux x86 gcc --disable-ssl	2m 34s
	5h 37m 54s

Provide feedback