controller: Fix an issue wrt cleanup of stale patch port. #760

Job	Run time
osx clang --disable-ssl	1m 33s
linux gcc --disable-ssl	2m 14s
linux clang --disable-ssl	2m 39s
linux gcc test -500	2m 37s
linux gcc test 501-1000	2m 26s
linux gcc test 1001-	3m 9s
linux clang test sanitizers -300	3m 19s
linux clang test sanitizers 301-600	3m 4s
linux clang test sanitizers 601-900	3m 47s
linux clang test sanitizers 901-1200	3m 36s
linux clang test sanitizers 1201-	3m 28s
linux gcc test -ljemalloc -500	2m 29s
linux gcc test -ljemalloc 501-1000	2m 29s
linux gcc test -ljemalloc 1001-	2m 18s
linux clang test -ljemalloc -500	3m 3s
linux clang test -ljemalloc 501-1000	2m 34s
linux clang test -ljemalloc 1001-	2m 30s
linux gcc system-test -100	2m 15s
linux gcc system-test 101-200	2m 12s
linux gcc system-test 201-	2m 36s
linux clang system-test sanitizers -100	3m 7s
linux clang system-test sanitizers 101-200	3m 18s
linux clang system-test sanitizers 201-	2m 52s
linux gcc m32 --disable-ssl	2m 21s
	1h 5m 56s

Provide feedback