infra/specify postgres version to match when dumping and restoring #913
Conversation
Signed-off-by: Nicolas Höning <nicolas@seita.nl>
Signed-off-by: Nicolas Höning <nicolas@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
| More info (e.g. on the format) at https://manpages.ubuntu.com/manpages/trusty/man1/pg_wrapper.1.html | ||
| """ | ||
| if pg_version: | ||
| return f"--cluster {pg_version}/main" |
There was a problem hiding this comment.
This didn't work for me. I got an Error: Invalid version 15 specified with --cluster. However, changing this line to return f"--version {pg_version}" made the dump work. I'm running PostgreSQL 13.5. I'm guessing you are running version 14, so perhaps you can test whether my remedy would suit you, too?
There was a problem hiding this comment.
I just implemented this change in my last commit for your testing convenience. If it works for you, too, please consider refactoring (e.g. renaming) pg_cluster_arg_from_version.
flexmeasures/cli/db_ops.py
Outdated
| More info (e.g. on the format) at https://manpages.ubuntu.com/manpages/trusty/man1/pg_wrapper.1.html | ||
| """ | ||
| if pg_version: | ||
| return f"--version {pg_version}" |
There was a problem hiding this comment.
@Flix6x Both pg_dump --help and pg_restore --help agree that --version leads to its very own outcome:
-V, --version output version information, then exit
And indeed, when I test flexmeasures db-ops dump --pg-version 15, the command says it created a dump file, but it actually did not. (as it exited, without an error, so our code assumed it did what we want).
You said "the cluster argument isn't working nicely with my PostgreSQL version (13)", but without further explanation. Right now, I don't know what that means. It worked with my Postgres version 14.
There was a problem hiding this comment.
Indeed, I failed to properly validate my solution. The error I get with return f"--cluster {pg_version}/main" is:
Exception happened during dump: Error: Invalid version 15.5 specified with --cluster
I'll revert my last commit. Then, we can try to work the issue I encounter with v13 (any suggestions?), or we could document that it only works for v14 or higher.
There was a problem hiding this comment.
Actually, according to the PostgreSQL docs (v13):
Because pg_dump is used to transfer data to newer versions of PostgreSQL, the output of pg_dump can be expected to load into PostgreSQL server versions newer than pg_dump's version. pg_dump can also dump from PostgreSQL servers older than its own version. (Currently, servers back to version 8.0 are supported.) However, pg_dump cannot dump from PostgreSQL servers newer than its own major version; it will refuse to even try, rather than risk making an invalid dump. Also, it is not guaranteed that pg_dump's output can be loaded into a server of an older major version — not even if the dump was taken from a server of that version.
Docs of newer versions have the same text, so I wonder how this could be working for you?
There was a problem hiding this comment.
Maybe we'd need to create a new script for this that spins up the required docker containers, creates the dump, moves the file and destroys itself, similar to what we have in ci/update-packages.sh?
There was a problem hiding this comment.
Might be interesting. Maybe using Docker pg containers for development makes more sense in general.
Do you propose this both for restore and dump?
I want to restore into my local database.
There was a problem hiding this comment.
Do you propose this both for
restoreanddump?
Yes, but only in the context of dumping from and restoring to a (newer) remote database server. For restoring to an older local database server, I'm not sure whether going with the docker-route would help with anything.
…ant cluster" This reverts commit ba4859a. Signed-off-by: F.N. Claessen <felix@seita.nl>
Description
When dumping or restoring a database, FlexMeasures devs and hosters would have a problem, when the target database has a different version than the default local database. One needs to install the version of the target database locally, but
pg_dumpandpg_restoremight still choose the default local version by default and then report a version mismatch as error.Look & Feel
This PR adds a new CLI option
--pg-versionto the two commandsflexmeasures db-ops dumpandflexmeasures db-ops restore.How to test
Dump the simulation database to file (it has PG 15, you might have PG 14 locally as default - which you can test with
pg_dump --version.Restore that dump to your local test database (set the db connection string in your ~/.flexmeasures.cfg first). I'm not sure this would run into the version mismatch, but you can see if restoring still works.