Use copy with binary.

[VaibhaveS]

• The binary format is a little faster than the text format (https://www.postgresql.org/docs/current/sql-copy.html).
• It's also useful in cases where the data stored in the BYTEA column, wherein the size in text format may exceed the 1GB field limit size of PostgreSQL.

pgsql.c:3159              [SOURCE 836084] [XX000] ERROR:  invalid memory alloc request size 1119193891
pgsql.c:3174              [SOURCE 836084] Context: Failed to fetch data from source
pgsql.c:3159              [TARGET 5216] [57014] ERROR:  COPY from stdin failed: Failed to get data from source

[dimitri]

While I like the idea, the main difficulty with it is how to make sure that the source and target Postgres servers are binary compatible? It's very hard to say because it is not well documented; and Postgres server compile time options will have an impact in that area (e.g. timestamp internal representation can be either bigint or float8).

Is is possible to use the binary protocol for selected columns, as in the libpq API? Then we could use it only for bytea columns, or a list of data type we know won't be a problem.

Otherwise we can add a new command line option such as --use-copy-binary to drive the decision, leaving the compatibility check to the users.

[VaibhaveS]

Is is possible to use the binary protocol for selected columns, as in the libpq API? Then we could use it only for bytea columns, or a list of data type we know won't be a problem.

I'll check this out.

EDIT: There isn't.

[VaibhaveS]

The actions are failing due to a weird reason.

[dimitri]

The actions are failing due to a weird reason.

We are having a look with @hanefi ; the current suspicion is that the old spelling docker-compose is not available anymore, the new spelling docker compose might need to be used now. I remember trying and failing to implement that transition some time ago, where it was the opposite problem. If that hypotheses explains it, then we should have a fix shortly (enough).

[dimitri]

If you rebase to the current main branch, it should be all good again now.

[hanefi]

One minor comment about whitespaces, otherwise looks good.

[dimitri]

Was about to merge then realized we are missing a long description of the new option in the docs at that place: https://pgcopydb.readthedocs.io/en/latest/ref/pgcopydb_clone.html#options -- @VaibhaveS can you add that?

[dimitri]

Missing option coverage in docs.