It turns out this was not due (only) to the schema validation errors, though there were a few. The error was actually coming from the fact that Swagger by default uses a hosted schema validator which has to remotely access the swagger definition file in order to validate it. This means that if your Swagger definition restricted access, then the hosted instance would not be able to access/validate it, resulting in this error.
I haven't tracked down the exact login in play here, but it seems that this behavior is disabled when running locally (perhaps for all IPs) but enabled when running with what appears to be a public URL.
For now I've just disabled the validation since I, for one, found it surprising that this remote service would be involved in scanning your APIs. I believe this repo is what you'd want to run if you were to host this validator internally, but it's Java based and I'm not going to touch that.