Why does _pick_unused_port_without_server() try random ports instead of letting the OS choose?

[dragonsinth]

Upgrading to the latest Go AppEngine runtime (which has an embedded copy of this code) put us into a code path that uses port_picker to select an open port number, which is then passed off to the Go subprocess to bind. Since the upgrade, we've seen a large number of both CI failures and local dev failures stemming from the Go process being unable to bind the port that the python code is asking it to bind.

Digging into the portpicker code, I'm curious why it tries random ports in the 15000-25000 range rather than just letting the OS pick a free port? My understanding is that letting the OS assign a free port would interoperate better with other (non-Python) processes random binding port 0, since the OS will try to ensure that successive port 0 binds don't conflict with each other.

[gpshead]

I don't have a good explanation for the ordering of those two range(10) loops. The code this originated from internally did it that way 14 years ago with little comment. (cl/4870845 internally for curious Googlers)

If you swap the order of those loops is it more reliable in your test automation?
I'm open to making that change if it works more reliably for someone not using a portserver.

As you already know, if you pick a port and have something else bind to it later, there is always a race condition no matter how you have picked the port. If an OS attempts to avoid that with rapid bind 0 calls, nice (it's in a better position to know), but it isn't completely gone.

Regardless, I recommend you use a portserver on each host. That is the best way to avoid this race condition. That is the code path that we use most often within Google, most of our automated test environments include a running port server.

[dragonsinth]

If you swap the order of those loops is it more reliable in your test automation?
I'm open to making that change if it works more reliably for someone not using a portserver.

Locally, we just removed the random loop and random set entirely (I couldn't figure out what value it added-- when would the OS not be able to satisfy a bind(0) call?) I'll need to watch it for a few days and see what happens to our flake rate.

As you already know, if you pick a port and have something else bind to it later, there is always a race condition no matter how you have picked the port. If an OS attempts to avoid that with rapid bind 0 calls, nice (it's in a better position to know), but it isn't completely gone.

Yep! Just hoping it drives that particular flake down from common to very rare.

Regardless, I recommend you use a portserver on each host. That is the best way to avoid this race condition. That is the code path that we use most often within Google, most of our automated test environments include a running port server.

That explanation makes a lot of sense. In our stack, dev_appserver is literally the only thing using portpicker, everything else that uses random ports just does a bind(0).

[gpshead]

anything to report on your flake rate?

FWIW, in the cpython test suite we just bind to port 0 for the stdlib's test.support.socket_helper.find_unused_port() function used by the cpython stdlib test suite. that call is best avoided due to the race conditions in the face of parallel test execution, but overall it has worked reasonably well for our buildbot fleet.

That's a much more limited scenario though as those tend to be dedicated test machines with little contention beyond -j test parallelization.

[dragonsinth]

Nah, we gave up and abandoned App Engine.

Mostly kidding-- we were moving off App Engine for other reasons anyway. :)

[gpshead]

LOL

[dragonsinth]

FWIW: editing portpicker to just use system bind(0) did actually remove the flakes for us :)

[gpshead]

i'll keep this open to remind me to consider doing that. our dominant use case at work (massive CI system) always has its own portserver running so we aren't exercising that codepath much in areas of the company I regularly see the fallout from.