Support for multiple hosts in postgresql connection string

[wingc-china]

PostgresSQL supports connecting with multiple hosts in connection string.
https://www.postgresql.org/docs/current/static/libpq-connect.html#libpq-multiple-hosts

The multiple hosts in connection string is like
postgresql+psycopg2://user:password@host1:port1,user:password@host2:port2/dbname
Unfortunately current SQLAlchemy behavior is parsing password@host1:port1,user:password out as password to try connecting.

In fact both libpq and psycopg2(psycopg/psycopg2#602) support multi-host already. We might want to add more code to the engine/url.py to add this support in SQLAlchemy.

[zzzeek]

OK, we can do this, but that URL format would be impossible to integrate because it is not RFC-1738 (unless it is, and I'm just ignorant of that variety, though the doc you point out says "URIs generally follow RFC 3986, except that multi-host connection strings are allowed"). we have a homegrown URL parser which I'd prefer not to mess with at this point.

Also the doc you refer to says, " the host, hostaddr, and port options accept a comma-separated list of values. " so user:pass is not part of it.

Propose postgresql+psycopg2://user:pass@/dbname?host=host1:port1&host=host2&host=..., since we do support repeated query parameters.

Also note if you have an application that needs to do this right now, you can use creator, so this capability is present now just not nicely within the URL.

[wingc-china]

Thanks! I didn't know I could use the creator. Will give it a try.
In fact I did some hack to make code in url.py try not parsing port to integer, but pass the port parameter as string to psycopg2 layer. And I tried a connection string as "postgresql+psycopg2://[username]:[password]@Host1,host2:port1,port2/[db name]". It seems working well :)
Let me know if anything else I can help, like testing or something.

Connection string sample was mis-quoted . Just corrected. :)

[zzzeek]

well anyone can volunteer to implement on this one it's pretty easy...

[wronglink]

Hi there. I'm here for the same issue. Actually one another workaround format for multi-host connection string is: postgresql+psycopg2://user:pass@/dbname?host=host1,host2,host3&port=111,222,333. But I think that the postgres docs format is way more readable: postgresql+psycopg2://user:pass@host1:111,host2:222,host3:333/dbname as it is easy to mix host parameters with some additional parameters like ssl or writable replica requirement.

And we should also take into account that it's really common way to connect sqlalchemy via URL inside some libraries and frameworks, like flask-sqlalchemy or celery. It would be way more convenient if public API url.make_url would support the example from the postgres docs.

[zzzeek]

@wronglink this goes beyond "easy to read". We are talking about the URL object which intends to provide a "good enough" URL not just for postgresql using libpq, but for any other database or database driver as well. Consider that this object presents a unified view of the common elements we use to connect to a database, including that there is a scalar ".host" and ".port" attribute. Turning these into lists to suit a specific feature of a single driver on a single database platform is not appropriate, it would add confusion and ambiguity to the URL object in terms of all other platforms.

An obvious workaround to turning ".hosts" and ".port" into lists is that we could instead keep .host and .port and then add new collections like .additional_hosts and .additional_ports, which is more awkward but doesn't break backwards compatibility for the world. Still, complexities arise, such as, what if the length of the .additional_hosts and the .additional_ports list are not the same ? This is also an issue with the approach you suggest of "?host=a&host=b?port=a&port=b". OK, perhaps this should be a list of tuples, .additional_host_ports or something like that, so if a port were missing from one of the host specifications, it's just blank.

So other than the parsing and generation having to be rewritten as well as all the new tests that would need to be written, and that there's this awkward ".additional_hosts_ports" attribute that seems to be pretty hardcoded to one particular driver, that's how it would look. All of this effort is unnecessary with my proposal that we just keep this simple and localized to the psycopg2 module and use the existing query string functionality. As I am already months if not years behind in the workload of issues that need to be attended towards, I'm not in a position to add these efforts to said workload when a much simpler approach is easily available.

After all that effort and rewriting, we have still bolted straight into URL a whole way of working that is entirely arbitrary and specific to one driver. Don't other drivers support multiple hosts? Well sure. The main example that comes to mind is odbc for SQL Server. The pyodbc client driver doesn't support any of that, so the issue is kind of moot, but if it did, note that in SQL Server's case, the multiple hosts are actually defining a more complex concept of primary and failover hosts. Which means if we in theory wanted our URL to work with those, all the work we did to support multiple hosts is still useless, because the different hosts need to be qualified as to what their role is. To be more general, if lots of DBAPIs supported multiple host/port combinations, this would be much easier to do. But they don't, so the reality of how such a feature should be generalized for other drivers is unknown. When we have an unknown as to how an API should be organized, the best answer to what we should do is to avoid guessing, which I'm sensitive towards because I'm typically the person that has to rip out bad decisions. My proposal avoids this guessing by keeping the change entirely localized to the psycopg2 dialect and nothing else.

In the general sense, a driver that can connect to multiple hosts may need to define any number of qualities for each host, like weighting, failover, failback, etc. It would look pretty amateurish that SQLAlchemy's URL bolted on a multiple-host/port concept that doesn't even work in the general sense.

the fact is, if you truly want to use postgresql's native URL format exactly as written, that is not a problem, that's why create_engine accepts the creator argument. Take the URL and pass it right through to psycopg2.

And we should also take into account that it's really common way to connect sqlalchemy via URL inside some libraries and frameworks, like flask-sqlalchemy or celery

this proposal does not impact that ability in any way, you'd be able to pass ?host&port along on the URL as you normally would. There are even ways to plug in the URL parsing to create_engine() right now, using engine strategies, so you could intercept the custom PG style URL up front and do what you want with it. I would sooner make these URL interception strategies more robust and documented so that custom URL parsing schemes can be plugged in before I would want to hardcode onto URL itself.

[jvanasco]

I agree with @zzzeek's implementation idea, but wanted to suggest an alternative that may take the same amount of work - but offer wider benefits.

What if the creator argument accepted a "dotted notation" string which specifies a callable? (SqlAlchemy does this on relationships, Pyramid does this all over). That would handle the concern for flask-sqlalchemy, celery and other configuration-file based apps to enjoy the benefit of this pyscopg2 feature, WHILE also have general control over the creator within configs as well (yay!).

That could be accompanied by an example in the PostgreSQL dialect docs on how to use the creator to handle the multiple host arguments.

[zzzeek]

I agree with @zzzeek's implementation idea, but wanted to suggest an alternative that may take the same amount of work - but offer wider benefits.

What if the creator argument accepted a "dotted notation" string which specifies a callable? (SqlAlchemy does this on relationships, Pyramid does this all over). That would handle the concern for flask-sqlalchemy, celery and other configuration-file based apps to enjoy the benefit of this pyscopg2 feature, WHILE also have general control over the creator within configs as well (yay!).

you mean if "creator" were perhaps a name to another entrypoint-linked module; we have a lot of entrypoint stuff going on already. let me publicize that there is already a plugin system for engines that allows any module or package to be called up based on an entrypoint name where it can then affect the process of building up the engine in any way it needs. I would have suggested using this immediately here, except that at the moment, the plugin system still works after the URL has been parsed. So a system by which the CreateEnginePlugin can also be called up before the URL is parsed would be needed. I think a way this can work is if in create_engine we do some kind of partial URL parsing, e.g. we look only at the "dbname+driver://" part first, or perhaps we only look at the query string part first, then we locate plugins based on that side of things first and give them the chance to give us a URL object of some kind, and only then when we didn't get a URL object do we fall back to our normal parsing. that way drivers and third party apps can make for more arbitrary formats of URL to come in. this would just involve a little bit more tinkering at the above code.

but again, query params are pretty open ended already so I still think a feature like this should be done the simple way first.

That could be accompanied by an example in the PostgreSQL dialect docs on how to use the creator to handle the multiple host arguments.

[shreyashag]

Is this still open? I would like to take this up!